JP2005102192A - Content receiving apparatus, video/audio output timing control method, and content providing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable surely adjusting the lip sync between video and audio at a decoder side without making a viewer have a sense of incongruity. <P>SOLUTION: A plurality of video frames VF1 and a plurality of audio frames AF1 are acquired and accumulated as a result of receiving and decoding a plurality of encoded video frames to which a video time stamp VTS is attached, and a plurality of encoded audio frames to which an audio time stamp ATS is attached. A time difference generated by deviation between the reference clock of the encoder side and the system time clock stc of the decoder side is calculated by renderers 37, 67. According to the time difference and using to the audio frame output timing when successively outputting a plurality of audio frames AF1 in frame unit as a reference, the video frame output timing of the plurality of video frames VF1 is adjusted per frame. Thus, lip sync can be performed while maintaining the audio continuity. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a content receiving apparatus, a video / audio output timing control method, and a content providing system, and is suitable for application to, for example, eliminating the lip sync between video and audio on the decoder side receiving content. .

2. Description of the Related Art Conventionally, in a content receiving apparatus, when content is received from an encoder server and decoded, it is separated into a video packet and an audio packet constituting the content, and a video time stamp attached to the video packet after decoding, respectively, By outputting a video frame and an audio frame based on the audio time stamp attached to the audio packet, the output timing of the video and the audio is matched (that is, lip synced) (for example, Patent Document 1). , See Patent Document 2).
Japanese Patent Laid-Open No. 8-280008 Japanese Patent Laid-Open No. 2004-15553

  By the way, in the content receiving apparatus having such a configuration, the system time clock on the decoder side and the reference clock on the encoder side are not always synchronized with each other, and the encoder is based on the clock jitter in the system time clock on the decoder side. There may be a slight shift in the clock frequency with the reference clock on the side.

  In addition, since the data length of the content receiving device is different between the video frame and the audio frame, when the system time clock on the decoder side and the reference clock on the encoder side are not completely synchronized, the video time stamp and the audio time stamp Even if the video frame and the audio frame are output based on the above, there is a problem that the output timing of the video and the audio does not match and the lip sync is shifted.

  The present invention has been made in consideration of the above points, and a content receiving apparatus, video audio and the like that can reliably adjust the lip sync between video and audio on the decoder side without causing the viewer user to feel uncomfortable. An output timing control method and a content providing system are proposed.

  In order to solve such a problem, in the present invention, a plurality of encoded video frames sequentially attached with video time stamps based on a reference clock on the encoder side, and a plurality of encodings sequentially attached with audio time stamps based on the reference clock. Decoding means for receiving and decoding an audio frame from a content providing apparatus on the encoder side, and storing a plurality of video frames and a plurality of audio frames obtained as a result of decoding the encoded video frame and the encoded audio frame by the decoding means Storage means, calculation means for calculating a time difference caused by a difference between the clock frequency of the reference clock on the encoder side and the clock frequency of the system time clock on the decoder side, and sequentially outputting a plurality of audio frames in units of frames according to the time difference When To be provided with a timing adjusting means for adjusting a video frame output timing when sequentially outputting a plurality of video frames in a frame unit based on the audio frame output timing.

  Multiple video frames based on the audio frame output timing when multiple audio frames are output sequentially in units of frames according to the time difference caused by the clock frequency difference between the reference clock on the encoder side and the system time clock on the decoder side By adjusting the video frame output timing when sequentially outputting video in frame units, the difference between the clock frequency on the encoder side and the decoder side can be absorbed, and the audio frame output timing can be synchronized with the video frame output timing to lip sync it can.

  In the present invention, a plurality of encoded video frames sequentially attached with video time stamps based on the reference clock on the encoder side and a plurality of codes sequentially attached with audio time stamps based on the reference clock are provided to the decoding means. A decoding step for receiving and decoding the encoded audio frame from the content providing apparatus on the encoder side, and a plurality of video frames obtained as a result of decoding the encoded video frame and the encoded audio frame in the decoding step with respect to the storage unit; A storage step for accumulating a plurality of audio frames, a difference calculating step for causing the calculation means to calculate a time difference caused by a difference between a clock frequency of a reference clock on the encoder side and a clock frequency of a system time clock on the decoder side, and timing Adjusting hand On the other hand, the timing adjustment that adjusts the video frame output timing when sequentially outputting multiple video frames in units of frames based on the audio frame output timing when sequentially outputting multiple audio frames in units of frames according to the time difference Steps.

  Multiple video frames based on the audio frame output timing when multiple audio frames are output sequentially in units of frames according to the time difference caused by the clock frequency difference between the reference clock on the encoder side and the system time clock on the decoder side By adjusting the video frame output timing when sequentially outputting video in frame units, the difference between the clock frequency on the encoder side and the decoder side can be absorbed, and the audio frame output timing can be synchronized with the video frame output timing to lip sync it can.

  The present invention further provides a content providing system having a content providing apparatus and a content receiving apparatus, wherein the content providing apparatus includes a plurality of encoded video frames with video time stamps based on a reference clock on the encoder side, and a reference clock. Encoding means for generating a plurality of encoded audio frames with audio time stamps based on, and a transmission means for sequentially transmitting a plurality of encoded video frames and a plurality of encoded audio frames to a content receiving device, The content receiving device receives a plurality of encoded video frames sequentially attached with video time stamps and a plurality of encoded audio frames sequentially attached with audio time stamps from a content providing device on the encoder side and decodes them Means and decoding means Accordingly, storage means for storing a plurality of video frames and a plurality of audio frames obtained as a result of decoding the encoded video frame and the encoded audio frame, the clock frequency of the reference clock on the encoder side, and the clock frequency of the system time clock on the decoder side Calculating means for calculating a time difference caused by the difference between the two and a plurality of video frames sequentially output in units of frames with reference to an audio frame output timing when a plurality of audio frames are sequentially output in units of frames according to the time difference. Timing adjusting means for adjusting the video frame output timing is provided.

  Multiple video frames based on the audio frame output timing when multiple audio frames are output sequentially in units of frames according to the time difference caused by the clock frequency difference between the reference clock on the encoder side and the system time clock on the decoder side By adjusting the video frame output timing when sequentially outputting video in frame units, the difference between the clock frequency on the encoder side and the decoder side can be absorbed, and the audio frame output timing can be synchronized with the video frame output timing to lip sync it can.

  As described above, according to the present invention, audio is output when a plurality of audio frames are sequentially output in units of frames in accordance with a time difference caused by a difference in clock frequency between a reference clock on the encoder side and a system time clock on the decoder side. By adjusting the video frame output timing when sequentially outputting multiple video frames in units of frames based on the frame output timing, the difference in clock frequency between the encoder side and the decoder side is absorbed, and the video frame is used as the audio frame output timing. A content receiving apparatus and video audio which can lip-sync in accordance with the output timing, and thus can reliably adjust the lip-sync between video and audio on the decoder side without making the viewer user feel uncomfortable It is possible to realize a force timing control method and the content providing system.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) Overall Configuration of Content Providing System In FIG. 1, reference numeral 1 denotes the content providing system of the present invention as a whole, which is roughly divided into a content providing apparatus 2 on the content distribution side and a first content reception on the content receiving side The apparatus 3 and the second content receiving apparatus 4 are configured.

  In the content providing system 1, the content providing device 2, the Web server 14, and the first content receiving device 3 are connected to each other via the Internet 5, and a content distribution destination acquired from the Web server 14 via the Internet 5. The URL (Uniform Resource Locator) and metadata related to the content are analyzed by the Web browser 15 in the first content receiving device 3, and the metadata and URL are supplied to the streaming decoder 9.

  The streaming decoder 9 accesses the streaming server 8 of the content providing apparatus 2 based on the URL analyzed by the Web browser 15 and makes a distribution request for the content desired by the user.

  In the content providing apparatus 2, content data corresponding to user-desired content is encoded in advance by the encoder 7, the resulting elementary stream is packetized by the streaming server 8, and the first content is received via the Internet 5. Distribution to the device 3 is performed.

  As a result, the content providing system 1 can realize pre-encoded streaming such as video on demand (VOD) in which content desired by the user in response to a request from the first content receiving device 3 is distributed from the content providing device 2. It is made like that.

  The first content receiving device 3 restores the original video and audio by decoding the elementary stream by the streaming decoder 9 and outputs the original video and audio from the monitor 10.

  In the content providing system 1, the first content receiving device 3 and the second content receiving device 4 are connected by a wireless LAN 6 compliant with a standard such as IEEE (Institute of Electrical and Electronics Engineers) 802.11a / b / g. The first content receiving device 3 is supplied from the outside, such as terrestrial digital, BS (Broadcast Satellite) / CS (Communication Satellite) digital or terrestrial analog broadcast content or DVD (Digital Versatile Disc), Content from a general video camera such as VideoCD can be wirelessly transmitted to the second content receiving device 4 in a form of being relayed after being encoded in real time by the real-time streaming encoder 11.

  Incidentally, the 1st content receiver 3 and the 2nd content receiver 4 do not necessarily need to be connected by wireless LAN 6, and may be connected by wired LAN.

  The second content receiving device 4 performs storm playback by decoding the content received from the first content receiving device 3 by the real-time streaming decoder 12 and outputs the playback result to the monitor 13.

  Thus, between the first content receiving device 3 and the second content receiving device 4, the content supplied from the outside is encoded in real time by the real-time streaming encoder 11 in the first content receiving device 3, and the second content. Live streaming can be realized by transmitting to the receiving device 4 and performing streaming playback on the second content receiving device 4.

(2) Configuration of Content Providing Device As shown in FIG. 2, the content providing device 2 is composed of an encoder 7 and a streaming server 8, and digitally converts a video signal VS1 taken from outside via a video input unit 21. Later, it is sent to the video encoder 22 as video data VD1.

  The video encoder 22 compresses and encodes the video data VD1 by a predetermined compression encoding method or various compression encoding methods compliant with, for example, the MPEG1 / 2/4 (Moving Picture Experts Group) standard, and the resulting video device 22 The mental stream VES1 is sent to the video ES storage unit 23 formed of a ring buffer.

  The video ES accumulation unit 23 temporarily accumulates the video elementary stream VES1, and then sends the video elementary stream VES1 to the packet generation unit 27 and the video frame counter 28 of the streaming server 8.

  The video frame counter 28 counts the video elementary stream VES1 in frame frequency units (29.97 [Hz], 30 [Hz], 59.94 [Hz] or 60 [Hz]), and uses the counted up value as a reference. The value is converted into a value in units of 90 [KHz] based on the clock, and is sent to the packet generation unit 27 as a video time stamp VTS (VTS1, VTS2, VTS3,...) For each video frame in 32-bit representation.

  Further, the content providing apparatus 2 digitally converts the audio signal AS1 captured from the outside via the audio input unit 24 of the encoder 7, and then sends the audio signal AS1 to the audio encoder 25 as audio data AD1.

  The audio encoder 25 compresses and encodes the audio data AD1 by a predetermined compression encoding method or various compression encoding methods compliant with the MPEG1 / 2/4 audio standard, and the resulting audio elementary stream AES1 is a ring buffer. To the audio ES storage unit 26.

  The audio ES accumulation unit 26 once accumulates the audio elementary stream AES1, and then sends the audio elementary stream AES1 to the packet generation unit 27 and the audio frame counter 29 of the streaming server 8.

  Similarly to the video frame counter 28, the audio frame counter 29 converts the count-up value of the audio frame into a value in units of 90 [KHz] based on the reference clock common to the video, and the audio time stamp ATS (ATS1, ATS2) for each audio frame. , ATS3,...) Is expressed as 32 bits and sent to the packet generator 27.

  The packet generation unit 27 divides the video elementary stream VES1 into packets of a predetermined data size, generates video packets by adding video header information to each packet, and also converts the audio elementary stream AES1 to a predetermined data size. An audio packet is generated by dividing the packet and adding audio header information to each packet.

  Here, as shown in FIG. 3, an audio packet and a video packet include an IP (Internet Protocol) header for communication between hosts in the Internet layer, a TCP (Transmission Control Protocol) header for transmission control in the transport layer, and real-time data transfer. The audio time stamp ATS and video time stamp VTS described above are written in a 4-byte time stamp area in the RTP header, which consists of a control RTP (Real Time Transport Protocol) header and an RTP payload.

  The packet generation unit 27 (FIG. 2) generates video packet data having a predetermined number of bytes based on the video packet and the video time stamp VTS, and audio having a predetermined number of bytes based on the audio packet and the audio time stamp ATS. Packet data is generated and multiplexed data MXD1 is generated by multiplexing the packet data, and then transmitted to the packet data storage unit 30.

  When the packet data storage unit 30 stores a predetermined amount of the multiplexed data MXD1, the packet data storage unit 30 transmits the multiplexed data MXD1 to the first content receiving device 3 via the Internet 5 by RTP / TCP (Real Time Transport Protocol / Transmission Control Protocol). It is made to do.

(3) Module Configuration of Streaming Decoder in First Content Receiving Device As shown in FIG. 4, the streaming decoder 9 of the first content receiving device 3 is multiplexed data MXD1 transmitted from the content providing device 2 by RTP / TCP. Is temporarily stored in the input packet storage unit 31 and then transmitted to the packet division unit 32.

  Here, the input packet storage unit 31 is configured to send the multiplexed data MXD1 to the packet dividing unit 32 when the multiplexed data MXD1 transmitted via the Internet 5 is stored for a predetermined amount of packets. Thus, the processing of the multiplexed data MXD1 can be executed continuously without interruption in the packet division unit 32 at the subsequent stage.

  The packet division unit 32 divides the multiplexed data MXD1 into video packet data VP1 and audio packet data AP1, and sends the audio packet data AP1 to the audio decoder 35 in units of audio frames via the input audio buffer 33 that is a ring buffer. At the same time, the video packet data VP1 is sent to the video decoder 36 in units of frames via the input video buffer 34 which is a ring buffer.

  Here, in the input audio buffer 33 and the input video buffer 34, the audio packet data AP1 for one audio frame and the video packet data VP1 for one video frame can be successively decoded by the audio decoder 35 and the video decoder 36 in the subsequent stage. Therefore, it has a capacity for instantaneously supplying data for at least one audio frame and one video frame to the audio decoder 35 and the video decoder 36 at any time.

  The packet division unit 32 can recognize the video time stamp VTS and the audio time stamp ATS by analyzing the video header information of the video packet data VP1 and the audio header information of the audio packet data AP1. The time stamp VTS and the audio time stamp ATS are sent to the timing control circuit 37 A of the renderer 37.

  The audio decoder 35 restores the audio frame AF1 before compression encoding by decoding the audio packet data AP1 in units of audio frames, and sequentially sends them to the renderer 37.

  The video decoder 36 restores the video frame VF1 before compression encoding by decoding the video packet data VP1 in units of video frames, and sequentially sends them to the renderer 37.

  By the way, in the streaming decoder 9, content metadata MD is supplied from the Web browser 15 to the system controller 50, and the content type of the content is determined based on the metadata MD in the system controller 50 as the content discriminating means. It is determined whether it is composed of audio and video, only video, or only audio, and the content type determination result CH is sent to the renderer 37.

  The renderer 37 temporarily stores the audio frame AF1 in the output audio buffer 38 that is a ring buffer, and similarly temporarily stores the video frame VF1 in the output video buffer 39 that is a ring buffer.

  The renderer 37 then causes the content type determination result CH from the system controller 50 and the audio time stamp ATS to lip-sync the video of the video frame VF1 to be output to the monitor 10 by the timing control circuit 37A and the audio of the audio frame AF1. The final output timing is adjusted based on the video time stamp VTS and the video frame VF1 and the audio frame AF1 are sequentially output from the output video buffer 39 and the output audio buffer 38 at the output timing.

(4) Lip sync adjustment processing at decoder side in pre-encoded streaming (4-1) Video frame and audio frame output timing adjustment method in pre-encoded streaming As shown in FIG. The video time stamp VTS (VTS1, VTS2, VTS3,..., VTSn) and audio time stamp ATS (ATS1, ATS2, ATS3,..., ATSn) sent from the packet division unit 32 are temporarily stored in the buffers 42 and 43, respectively. After being stored, the data is sent to the comparator circuit 46.

  The timing control circuit 37A also sends only the first video time stamp VTS1 and audio time stamp ATS1 in the content to the subtractor circuits 44 and 45, respectively.

  The subtractor circuits 44 and 45 pull back the values of the first video time stamp VTS1 and the audio time stamp ATS1 by a predetermined time, and send them to the STC circuit 41 as the preset video time stamp VTSp and the preset audio time stamp ATSp. To do.

  The STC circuit 41 presets the value of the system time clock stc in accordance with a preset sequence determined in the order of the preset video time stamp VTSp and the preset audio time stamp ATSp, that is, the value of the system time clock stc is set to the preset video time. Adjustment (replacement) is made in the order of the stamp VTSp and the preset audio time stamp ATSp.

  Here, the STC circuit 41 uses the preset video time stamp VTSp and the preset audio time stamp ATSp obtained by pulling back the values of the first video time stamp VTS1 and the audio time stamp ATS1 by a predetermined time, and the value of the system time clock stc. When the first video time stamp VTS1 and audio time stamp ATS1 reach the comparator circuit 46 via the buffers 42 and 43, the preset system time clock supplied from the STC circuit 41 to the comparator circuit 46 is set. The value of stc indicates a time before the video time stamp VTS1 and the audio time stamp ATS1.

  As a result, in the comparator circuit 46 of the timing control circuit 37A, the value of the system time clock stc after the preset is not already passed with respect to the first video time stamp VTS1 and the audio time stamp ATS1, so that the first time The video frame Vf1 and the audio frame Af1 corresponding to the video time stamp VTS1 and the audio time stamp ATS1 can also be reliably output.

  In practice, as shown in FIGS. 6A and 6B, when the content type is composed of audio and video, the value of the system time clock stc is set to the preset video time stamp VTSp, and the preset audio time stamp. If presetting is performed according to a preset sequence determined in the order of ATSp, it means that the preset value is always updated with the audio time stamp ATSp for presetting after the value of the system time clock stc is preset with the video time stamp VTSp for presetting. .

  At this time, the comparator circuit 46 compares the video time stamp VTS with the system time clock stc after the preset value is updated with the preset audio time stamp ATSp, thereby comparing the value of the system time clock stc after the preset. The time difference from the video time stamp VTS attached by the content providing apparatus 2 on the encoder side is calculated.

  On the other hand, if the content type is composed only of audio, the preset video time stamp VTSp is not sent to the timing control circuit 37A, so the value of the system time clock stc is set as the preset video time stamp VTSp. According to the preset sequence determined in the order of the preset audio time stamp ATSp, it naturally means that the value of the system time clock stc is preset with the preset audio time stamp ATSp.

  Similarly, when the content type is composed only of video, the preset audio time stamp ATSp is not sent to the timing control circuit 37A, so the value of the system time clock stc is set as the preset video time stamp. This means that if the preset sequence determined in the order of VTSp and preset audio time stamp ATSp is followed, the value of the system time clock stc is preset with the preset video time stamp VTSp.

  This is only when the content type is only audio, or only video, and there is no need to adjust the lip sync of video and audio. Therefore, after the preset with the preset audio time stamp ATSp The audio frame AF1 may be output when the value of the system time clock stc matches the audio time stamp ATS, and the value of the system time clock stc after being preset by the preset video time stamp VTSp and the video time stamp VTS. The video frame VF1 may be output when the two match.

  Actually, in the timing control circuit 37A of the renderer 37, for example, when the content type is composed of audio and video, as shown in FIG. 7, the audio frame AF1 (Af1, Af2,. At the time points Ta1, Ta2, Ta3,... That sequentially output Af3,...) To the monitor 10, the value of the system time clock stc supplied through the crystal oscillator circuit 40 (FIG. 4) and the STC circuit 41. Are preset in the order of the preset video time stamp VTSp and the preset audio time stamp ATSp, so that the value of the system time clock stc is finally set to the preset audio time stamp ATSp1, ATSp2, ATSp3, ... and to match.

  This is very noticeable to the user when the sound is interrupted or skipped during reproduction. Therefore, the timing control circuit 37A of the renderer 37 processes the audio frame AF1 (Af1, Af2, Af3,...) With the lip sync adjustment process. This is because it is necessary to adjust the output timing of the video frame VF1 (Vf1, Vf2, Vf3,...) In accordance with the output of the audio frame AF1 (Af1, Af2, Af3,...).

  When the output timing (time points Ta1, Ta2, Ta3,...) Of the audio frame AF1 (Af1, Af2, Af3,...) Is determined, the timing control circuit 37A of the renderer 37 determines the video frame VF1 (Vf1, Vf2, Vf3). ,... Are output at a frame frequency of 30 [Hz] based on the system time clock stc, at any time point Tv1, Tv2, Tv3,..., The count value of the system time clock stc after the preset and the video frame VF1 ( The video time stamps VTS (VTS1, VTS2, VTS3,...) Attached to Vf1, Vf2, Vf3,.

  In the comparator circuit 46, the video frame VF1 (Vf1, Vf2,...) Is output from the output video buffer 39 when the count value of the system time clock stc after the preset and the video time stamp VTS (VTS1, VTS2, VTS3,...) Match. Vf3,...) Is output to the monitor 10.

  By the way, the comparator circuit 46 compares the count value of the system time clock stc after presetting with the video time stamp VTS (VTS1, VTS2, VTS3,...) Sent from the buffer 42, and as a result, the system time clock stc after presetting. If the difference value D1 (time difference) between the count value of the video and the video time stamp VTS (VTS1, VTS2, VTS3,...) Is equal to or less than a threshold value TH representing a predetermined time, the video and audio match for the user. Since the timing control circuit 37A matches the count value of the preset system time clock stc and the video time stamp VTS (VTS1, VTS2, VTS3,...), The video frame VF1 ( Vf1, Vf2, V 3, ...) may be directly output to the monitor 10.

  In other cases, for example, at the timing of time Tv2, the difference value D1 between the count value of the system time clock stc after the preset and the video time stamp VTS2 is larger than the predetermined threshold value TH, and the video is delayed from the sound. In this case, since the video does not catch up with the audio due to the difference between the clock frequency on the encoder side and the clock frequency on the decoder side, the timing control circuit 37A of the renderer 37 constitutes a GOP (Group Of Picture). For example, a video frame Vf3 (not shown) corresponding to a B picture is skipped without being decoded and the next video frame Vf4 is output.

  In this case, the renderer 37 does not skip the “P” picture stored in the output video buffer 39 because it is a reference frame when the video decoder 36 decodes the next picture. By skipping the “B” picture, which is a non-reference frame that does not become the reference frame, lip sync is performed while preventing image quality deterioration.

  By the way, in the renderer 37, if the “B” picture to be skipped does not exist in the output video buffer 39 and there are only “I” picture and “P” picture, the “B” picture cannot be skipped. Because it is not possible, it will not be able to catch up with the video.

  Therefore, in the renderer 37, when the “B” picture to be skipped does not exist in the output video buffer 39, the monitor output timing of the monitor 10 is, for example, 60 [Hz] as shown in FIG. By utilizing the fact that the picture refresh timing of the power video frame VF1 is 30 [Hz], the picture refresh timing is shortened.

  Specifically, the renderer 37 determines that when the difference value D1 between the count value of the system time clock stc and the video time stamp VTS after presetting with the preset audio time stamp ATSp exceeds 16.666. When the monitor output timing is delayed by one frame or more with respect to the output timing, the picture refresh timing is changed from 30 [Hz] to 60 [Hz] instead of skipping the video frame VF1 for one frame. The (N + 1) th picture is output.

  In other words, the renderer 37 shortens the picture refresh interval from 1/30 seconds to 1/60 seconds for the “I” picture and “P” picture that are affected by the image quality degradation due to the skip, so that the “I” picture and “ The video can be caught up with the sound without causing the image quality deterioration due to the skipping of the “P” picture.

  On the other hand, the timing control circuit 37A of the renderer 37 has a difference value D1 between the preset value of the system time clock stc and, for example, the video time stamp VTS2 at the timing of the time point Tv2 larger than a predetermined threshold value TH. If the audio is behind the video, the audio is not catching up with the video due to the difference between the clock frequency on the encoder side and the clock frequency on the decoder side, so the video frame Vf2 currently being output is repeated. The output is repeated.

  On the other hand, in the timing control circuit 37A of the renderer 37, for example, when the content type is composed only of video, the video frame VF1 (Vf1, Vf2, Vf3,...) Decoded by the video decoder 36 is sent to the monitor 10. At the timing of the time points Tv1, Tv2, Tv3,..., Which are sequentially output, the video frame VF1 (Vf1, Vf1,. Vf2, Vf3, ...) may be output to the monitor 10.

  Similarly, the timing control circuit 37A of the renderer 37 monitors the audio frame AF1 (Af1, Af2, Af3,...) That has been decoded by the audio decoder 35, for example, when the content type is only audio. At the time points Ta1, Ta2, Ta3,... That are sequentially output to the audio frame AF1 (Af1) at the timing when the count value of the system time clock stc preset by the preset audio time stamp ATSp matches the audio time stamp ATS. , Af2, Af3,...) May be output from the speaker of the monitor 10.

(4-2) Lip sync adjustment processing procedure in pre-encoded streaming As described above, the timing control circuit 37A of the renderer 37 in the streaming decoder 9 uses the audio frame AF1 (Af1, Af2, Af3,...) As a reference to the video frame. The output timing adjustment method for lip-syncing video and audio by adjusting the output timing of VF1 (Vf1, Vf2, Vf3,...) Is summarized as shown in the flowchart of FIG. The timing control circuit 37A enters from the start step of the routine RT1 and proceeds to the next step SP1.

  In step SP1, the renderer 37 presets the value of the system time clock stc in accordance with a preset sequence determined in the order of the preset video time stamp VTSp and the preset audio time stamp ATSp, and proceeds to the next step SP2.

  Here, the renderer 37 always updates the preset value with the preset audio time stamp ATSp after presetting the value of the system time clock stc with the preset video time stamp VTSp when the content type is audio and video. Then, the process proceeds to the next step SP2.

  In this case, at the time points Ta1, Ta2, Ta3,... When the audio frame AF1 (Af1, Af2, Af3,...) Is output to the monitor 10 (FIG. 7), the value of the system time clock stc and the preset audio time. The stamp ATSp (ATSp1, ATSp2, ATSp3,...) Matches.

  The renderer 37 does not have a preset audio time stamp ATSp when the content type is composed only of video. Therefore, the renderer 37 presets the value of the system time clock stc with the preset video time stamp VTSp for a predetermined time. When it has elapsed, the process proceeds to the next step SP2.

  Furthermore, when the content type is composed only of audio, the renderer 37 does not have the preset video time stamp VTSp, so that the preset audio time stamp ATSp arrives without waiting for the preset video time stamp VTSp. After presetting the value of the system time clock stc at the time, the process proceeds to the next step SP2.

  In step SP2, the renderer 37 determines whether or not the content is only a video based on the content type determination result CH supplied from the system controller 50. If a positive result is obtained, the renderer 37 proceeds to the next step SP3.

  In step SP3, the renderer 37 makes the video frame VF1 (Vf1, Vf2) when the count value of the system time clock stc preset by the preset video time stamp VTSp matches the video time stamp VTS because the content is only video. , Vf3,...) Are output to the monitor 10, and the process proceeds to the next step SP12 to end the process.

  On the other hand, if a negative result is obtained in step SP2, this means that the type of content is not only video, but is either audio and video or only audio. At this time, the renderer 37 proceeds to the next step SP4.

  In step SP4, the renderer 37 determines whether or not the content is only audio based on the content type determination result CH. If a positive result is obtained, the renderer 37 proceeds to the next step SP3.

  In step SP3, the renderer 37 makes the audio frame AF1 (Af1, Af2) when the count value of the system time clock stc preset by the preset audio time stamp ATSp matches the audio time stamp ATS because the content is only audio. , Af3,.

  On the other hand, if a negative result is obtained in step SP4, this indicates that the content type is composed of audio and video. At this time, the renderer 37 proceeds to the next step SP5.

  In step SP5, the renderer 37 has a content type consisting of audio and video, so that the count value of the system time clock stc preset by the preset audio time stamp ATSp and the time points Tv1, Tv2, Tv3, A difference value D1 (= stc−VTS) from the time stamp VTS (VTS1, VTS2, VTS3,...) Of the video frame VF1 (Vf1, Vf2, Vf3,...) To be output at the timing of. The process moves to the next step SP6.

  In step SP6, the renderer 37 determines whether or not the difference value D1 (absolute value) calculated in step SP7 is larger than a predetermined threshold value TH. If a negative result is obtained here, this means that the difference value D1 cannot be determined by the user who has seen and heard the video and the audio as being shifted between the video and the audio. This represents that the time is 100 (msec) or less. At this time, the renderer 37 moves to the next step SP3.

  In step SP3, the renderer 37 does not have enough time to determine that the video and the audio are deviated from each other. In this case, the video frame VF1 is output to the monitor 10 as it is, and the audio frame AF1 is also in principle. The data is output to the monitor 10 as it is, and the process proceeds to the next step SP12 to end the process.

  On the other hand, if an affirmative result is obtained in step SP6, this means that the difference value D1 is larger than the predetermined threshold value TH, that is, the difference between the video and the audio for the user who has seen and listened to the video and the audio. In this case, the renderer 37 proceeds to the next step SP7.

  In step SP7, the renderer 37 determines whether the video is behind the audio based on the audio time stamp ATS and the video time stamp VTS. If a negative result is obtained, the renderer 37 proceeds to the next step SP8.

  In step SP8, the renderer 37 repeats and repeatedly outputs the video frame VF1 constituting the picture currently being output so that the audio catches up with the video because the video is ahead of the audio, and then the next step. The process proceeds to SP12 and the process is terminated.

  On the other hand, if an affirmative result is obtained in step SP7, this indicates that the video is delayed from the audio. At this time, the renderer 37 moves to the next step SP9 and skips to the output video buffer 39. It is determined whether or not a “B” picture exists, and if a positive result is obtained, the process proceeds to the next step SP10.

  In step SP10, the renderer 37 skips the B picture (in this case, the video frame Vf3) without decoding and outputs it in order to recover the delay of the video with respect to the audio, thereby recovering the delay of the video with respect to the audio and lip-sync. Then, the process proceeds to the next step SP12 to end the process.

  On the other hand, if a negative result is obtained in step SP9, this indicates that the “B” picture to be skipped does not exist in the output video buffer 39, and the “B” picture cannot be skipped. The renderer 37 proceeds to the next step SP11.

  In step SP11, the renderer 37 uses the fact that the monitor output timing of the monitor 10 is 60 [Hz] while the picture refresh timing of the video frame VF1 is 30 [Hz] as shown in FIG. By shortening the picture refresh timing in accordance with the monitor output timing of the monitor 10, the video is caught up with the sound without causing picture quality deterioration due to skipping the picture, and the process is moved to the next step SP12 and the process is terminated. To do.

(5) Circuit Configuration of Real-Time Streaming Encoder in First Content Receiving Device First content receiving device 3 (FIG. 1) is an externally supplied content such as terrestrial digital, BS / CS digital or terrestrial analog broadcast Alternatively, content from a general video camera such as a DVD, VideoCD, or the like is encoded in real time by the real-time streaming encoder 11 and then wirelessly transmitted to the second content receiving device 4 so that it can also be a content providing side. Has been made.

  The circuit configuration of the real-time streaming encoder 11 in the first content receiving device 3 will be described with reference to FIG. The real-time streaming encoder 11 digitally converts the video signal VS2 and the audio signal AS2 constituting the content supplied from the outside through the video input unit 51 and the audio input unit 53, and converts this into video data VD2 and audio data AD2. The data is sent to the encoder 52 and the audio encoder 54.

  The video encoder 52 compresses and encodes the video data VD2 using a predetermined compression encoding method or various compression encoding methods compliant with, for example, the MPEG1 / 2/4 standard, and generates the resulting video elementary stream VES2 as a packet. The data is sent to the unit 56 and the video frame counter 57.

  In the video frame counter 57, the video elementary stream VES2 is counted in frame frequency units (29.97 [Hz], 30 [Hz], 59.94 [Hz] or 60 [Hz]), and the counted up value is used as a reference. The value is converted into a value in units of 90 [KHz] based on the clock, and is sent to the packet generator 56 as a video time stamp VTS (VTS1, VTS2, VTS3,...) For each video frame in 32-bit representation.

  The audio encoder 54 compresses and encodes the audio data AD2 by a predetermined compression encoding method or various compression encoding methods compliant with the MPEG1 / 2/4 audio standard, and generates a packet of the resulting audio elementary stream AES2 The data is sent to the unit 56 and the audio frame counter 58.

  As with the video frame counter 57, the audio frame counter 58 converts the audio frame count value into a value in units of 90 [KHz] based on the common reference clock and the audio time stamp ATS (ATS1, ATS2, ATS3,...). As 32 bits and sent to the packet generator 56.

  The packet generation unit 56 divides the video elementary stream VES2 into packets of a predetermined data size, generates video packets by adding video header information to each packet, and generates the audio elementary stream AES2 of a predetermined data size. An audio packet is generated by dividing the packet and adding audio header information to each packet.

  Here, as shown in FIG. 11, the control packet added to the front of the RTCP (Real Time Control Protocol) packet is an IP (Internet Protocol) header for host-to-host communication in the Internet layer, for user datagram data transfer. 4 bytes RTP timestamp area in the sender information in the RTCP packet sender port, which consists of the UDP (User Datagram Protocol) header, the RTCP packet sender report for the real time data transfer control, and the RTCP packet. The snapshot information of the system time clock value on the encoder side is written as a PCR (Program Clock Reference) value, and is sent from the PCR circuit 61 for clock recovery on the decoder side.

  The packet generator 56 generates video packet data having a predetermined number of bytes based on the video packet and the video time stamp VTS, and generates audio packet data having a predetermined number of bytes based on the audio packet and the audio time stamp ATS. Then, multiplexed data MXD2 is generated by multiplexing these, and then transmitted to the packet data storage unit 59.

  When the packet data storage unit 59 stores a predetermined amount of the multiplexed data MXD2, the packet data storage unit 59 transmits the multiplexed data MXD2 to the second content receiving device 4 via the wireless LAN 6 by RTP / TCP.

  Incidentally, the real-time streaming encoder 11 also supplies the video data VD2 digitally converted by the video input unit 51 to a PLL (Phase-Locked Loop) circuit 55. The PLL circuit 55 synchronizes the STC circuit 60 with the clock frequency of the video data VD2 based on the video data VD2, and also synchronizes the video encoder 52, the audio input unit 53, and the audio encoder 54 with the clock frequency of the video data VD2. It is made like that.

  As a result, the real-time streaming encoder 11 can execute the compression encoding process for the video data VD2 and the compression encoding process for the audio data AD2 via the PLL circuit 55 at timing synchronized with the clock frequency of the video data VD2. A clock reference pcr synchronized with the clock frequency of the video data VD2 can be transmitted to the real-time streaming decoder 12 in the second content receiver 4 via the (Program Clock Reference) circuit 61.

  At this time, the PCR circuit 61 is configured to transmit the clock reference pcr to the real-time streaming decoder 12 of the second content receiving device 4 by using UDP (User Datagram Protocol) which is located in the lower layer of the RTP protocol and requires real-time property. As a result, high-speed performance can be ensured, and live streaming that requires real-time performance can be supported.

(6) Circuit Configuration of Real-Time Streaming Decoder in Second Content Receiving Device As shown in FIG. 12, the real-time streaming decoder 12 in the second content receiving device 4 transmits from the real-time streaming encoder 11 of the first content receiving device 3. The multiplexed data MXD2 is temporarily stored in the input packet storage unit 71 and then sent to the packet division unit 72.

  The packet division unit 72 divides the multiplexed data MXD2 into video packet data VP2 and audio packet data AP2, and sends the audio packet data AP2 to the audio decoder 74 in units of audio frames via the input audio buffer 73 formed of a ring buffer. At the same time, the video packet data VP2 is sent to the video decoder 76 in units of frames via the input video buffer 75 which is a ring buffer.

  Here, also in the input audio buffer 73 and the input video buffer 75, the audio packet data AP2 and the video packet data VP2 for one audio frame and one video frame can be successively decoded by the audio decoder 74 and the video decoder 76 in the subsequent stage. Therefore, it is sufficient to have a data capacity for at least one audio frame and one video frame.

  The packet division unit 72 can recognize the audio time stamp ATS and the video time stamp VTS by analyzing the video header information of the video packet data VP2 and the audio header information of the audio packet data AP2. The time stamp ATS and the video time stamp VTS are sent to the renderer 77.

  The audio decoder 74 restores the audio frame AF2 before compression encoding by decoding the audio packet data AP2 in units of audio frames, and sequentially sends them to the renderer 77.

  The video decoder 76 restores the video frame VF2 before compression encoding by decoding the video packet data VP2 in units of video frames, and sequentially sends them to the renderer 77.

  The renderer 77 temporarily stores the audio frame AF2 in the output audio buffer 78 that is a ring buffer, and similarly temporarily stores the video frame VF2 in the output video buffer 79 that is a ring buffer.

  Then, the renderer 77 adjusts the final output timing based on the audio time stamp ATS and the video time stamp VTS so as to lip-sync the video of the video frame VF2 output to the monitor 13 and the audio of the audio frame AF2. The audio frame AF2 and the video frame VF2 are sequentially output from the output audio buffer 78 and the output video buffer 79 to the monitor 13 at the output timing.

  Incidentally, the real-time streaming decoder 12 receives the clock reference pcr transmitted by the UDP from the PCR circuit 61 of the real-time streaming encoder 11 in the first content receiving device 3 and inputs it to the subtraction circuit 81.

  The subtraction circuit 81 calculates a difference between the clock reference pcr and the system time clock stc supplied from the STC circuit 84, and sequentially subtracts this through the filter 82, the voltage controlled crystal oscillator circuit 83, and the STC circuit 84. To form a PLL (Phase Locked Loop) and gradually converge to the clock reference pcr of the real-time streaming encoder 11, and finally render the system time clock stc synchronized with the real-time streaming encoder 11 by the clock reference pcr. 77.

  As a result, the renderer 77 compresses and encodes the video data VD2 and the audio data AD2 by the real-time streaming encoder 11 in the first content receiving apparatus 3, and synchronizes with the clock frequency when the video time stamp VTS and the audio time stamp ATS are counted. The output timing of the video frame VF2 and the audio frame AF2 can be adjusted on the basis of the system time clock stc.

  In practice, the renderer 77 temporarily stores the audio frame AF2 in the output audio buffer 78 that is a ring buffer, and temporarily stores the video frame VF2 in the output video buffer 79 that is a ring buffer. Are output in a lip sync state based on the system time clock stc, audio time stamp ATS, and video time stamp VTS synchronized with the encoder side by the clock reference pcr supplied from the PCR circuit 61 of the real-time streaming encoder 11. The output timing is adjusted.

(7) Lip sync adjustment processing on the decoder side in live streaming (7-1) Output timing adjustment method of video frame and audio frame in live streaming As shown in FIG. The clock time of the system time clock stc is locked by the PLL to the value of the clock reference pcr supplied from the PCR circuit 61 at a predetermined cycle, and then the audio time is passed through the monitor 13 synchronized based on the system time clock stc. The output of the audio frame AF2 and the video frame VF2 is controlled according to the stamp ATS and the video time stamp VTS.

  That is, the renderer 77 synchronizes the clock frequency of the system time clock stc with the value of the clock reference pcr, and in accordance with the system time clock stc and the audio time stamp ATS (ATS1, ATS2, ATS3,...), The audio frame AF2 (Af1,. Af2, Af3,...) Are sequentially output to the monitor 13.

  Here, since the value of the clock reference pcr and the clock frequency of the system time clock stc maintain the synchronous relationship as described above, the count value of the system time clock stc and the video time stamp VTS (VTS1, VTS2, VTS3, )), For example, the difference value D2V between the count value of the system time clock stc and the video time stamp VTS1 does not occur at the time point Tv1.

  However, since the clock reference pcr supplied from the PCR circuit 61 of the real-time streaming encoder 11 is transmitted by UDP that requires real-time performance, the clock reference pcr is not subjected to retransmission control so that high-speed performance is important. 2 may not reach the real-time streaming decoder 12 of the content receiver 4 or may include error data.

  In such a case, the synchronization between the value of the clock reference pcr supplied from the PCR circuit 61 of the real-time streaming encoder 11 at a predetermined cycle and the clock frequency of the system time clock stc may be shifted via the PLL. At this time, the renderer 77 according to the present invention can guarantee the lip sync.

  In the present invention, when a deviation occurs between the system time clock stc, the audio time stamp ATS, and the video time stamp VTS, the continuity of the audio output is given priority as a method of taking a lip sync.

  The renderer 77 compares the count value of the system time clock stc at the output timing Ta2 of the audio frame AF2 with the audio time stamp ATS2, and stores the difference value D2A. On the other hand, the renderer 77 compares the count value of the system time clock stc at the output timing Tv2 of the video frame VF2 with the video time stamp VTS2, and stores the difference value D2V.

  At this time, the clock reference pcr surely reaches the real-time streaming decoder 12 of the second content receiving device 4, and the value of the clock reference pcr and the clock frequency of the system time clock stc of the real-time streaming decoder 12 are transmitted via the PLL. If the values coincide completely and the decoder side including the monitor 13 is synchronized with the system time clock stc, the difference values D2V and D2A are “0”.

  If the difference D2A is a positive value, the audio frame AF2 is determined to be early, and if the difference D2A is a negative value, the audio frame AF2 is determined to be delayed. Similarly, if the difference D2V is a positive value, it is determined that the video frame VF2 is early, and if the difference D2V is a negative value, it is determined that the video frame VF2 is delayed.

  Here, the renderer 77 gives priority to maintaining the continuity of the audio output regardless of whether the audio frame AF2 is early or late, and controls the output of the video frame VF2 relative to the audio frame AF2 as follows. .

  For example, when D2V-D2A is greater than the threshold value TH at the time Tv2, the renderer 77 forms a GOP because the video does not catch up with the audio if the difference value D2V is greater than the difference value D2A. For example, a video frame Vf3 (not shown) corresponding to a B picture is skipped without being decoded and the next video frame Vf4 is output.

  In this case, the renderer 77 does not skip the “P” picture stored in the output video buffer 79 and generates the next picture without skipping since it becomes a reference frame when the video decoder 76 decodes the next picture. By skipping the “B” picture, which is a non-reference frame that does not become the reference frame, lip sync is performed while preventing image quality deterioration.

  On the other hand, if D2V-D2A is larger than the threshold value TH and the difference value D2A is larger than the difference value D2V, the audio has not caught up with the video, so the renderer 77 determines the video frame Vf2 currently being output. Repeated output is made.

  When D2V-D2A is smaller than the threshold value TH, it is determined that the video gap with respect to the sound is within the allowable range, and the renderer 77 outputs the video frame VF2 to the monitor 13 as it is.

  By the way, in the renderer 77, if the “B” picture to be skipped does not exist in the output video buffer 79 and there are only “I” picture and “P” picture, the “B” picture cannot be skipped. Because it is not possible, it will not be able to catch up with the video.

  Therefore, in the renderer 77, similarly to the renderer 37 of the streaming decoder 9 in the first content receiving device 3, when there is no “B” picture to be skipped, the monitor output timing of the monitor 13 is 60 [Hz], for example. Using the fact that the picture refresh timing of the video frame VF2 to be output from the output video buffer 79 is 30 [Hz], the picture refresh timing is shortened.

  Specifically, the renderer 77 monitors the monitor output timing when the difference value between the system time clock stc synchronized with the clock reference pcr and the video time stamp VTS exceeds 16.666... [Msec], that is, with respect to the audio output timing. Is delayed by one frame or more, the picture refresh timing is changed from 30 [Hz] to 60 [Hz] instead of skipping one video frame to shorten the display interval.

  In other words, the renderer 77 shortens the picture refresh interval from 1/30 seconds to 1/60 seconds for the “I” picture and “P” picture that are affected by the image quality degradation due to the skip, so that the “I” picture and “ It is possible to make the video catch up with the audio without causing image quality degradation due to skipping the “P” picture.

(7-2) Lip Sync Adjustment Processing Procedure in Live Streaming As described above, when the renderer 77 of the real-time streaming decoder 12 performs live streaming playback, the output timing of the video frame VF2 is adjusted based on the audio frame AF2. When the output timing adjustment method for lip-syncing video and audio is summarized by the above, as shown in the flowchart of FIG. 14, the renderer 77 of the real-time streaming decoder 12 enters from the start step of the routine RT2, and then the next step SP21. Move on.

  In step SP21, the renderer 77 of the real-time streaming decoder 12 in the second content receiving device 4 receives the clock reference pcr from the PCR circuit 61 of the real-time streaming encoder 11 in the first content receiving device 3, and proceeds to the next step SP22. Move.

  In step SP22, the renderer 77 synchronizes the clock reference pcr and the system time clock stc with the PLL configured via the subtracting circuit 81, the filter 82, the voltage controlled crystal oscillator circuit 83, and the STC circuit 84, and thereafter. Then, the system time clock stc synchronized with the clock reference pcr is used as a reference for adjusting the output timing, and the process proceeds to the next step SP23.

  In step SP23, the renderer 77 calculates a difference value D2V between the count value of the system time clock stc and the video time stamp VTS at the timings Tv1, Tv2, Tv3,..., And the time points Ta1, Ta2, Ta3,. The difference value D2A between the count value of the system time clock stc and the audio time stamp ATS at the timing is calculated, and the process proceeds to the next step SP24.

  In step SP24, the renderer 77 obtains a negative result when D2V-D2A calculated based on the difference values D2V and D2A calculated in step SP23 is smaller than a threshold value TH (for example, 100 [msec]), and proceeds to the next step SP25. Move.

  In step SP25, when D2A-D2V is larger than the threshold value TH (for example, 100 [msec]), the renderer 77 obtains a positive result and determines that the video is proceeding with respect to the sound, and proceeds to the next step SP26.

  In step SP26, the renderer 77 repeats and outputs the video frame VF2 constituting the picture currently being output so that the audio catches up with the video because the video is ahead of the audio, and then proceeds to the next step SP31. The process is terminated.

  On the other hand, if D2A-D2V does not exceed the threshold value TH in step SP25, a negative result is obtained, and it is determined that there is no feeling that there is a deviation between the audio and the video, Control goes to the next step SP27.

  In step SP27, the renderer 77 does not have a time that does not feel that there is a difference between video and audio. In this case, the video frame VF2 is based on the system time clock stc synchronized with the clock reference pcr. Is output as it is to the monitor 13 in accordance with the video time stamp VTS, and the process proceeds to the next step SP31.

  The renderer 77 outputs the sound to the monitor 13 as it is according to the audio time stamp ATS based on the system time clock stc synchronized with the clock reference pcr in any of the above cases in order to maintain the continuity of the sound. It is made like that.

  On the other hand, if a positive result is obtained in step SP24, this indicates that D2V-D2A is larger than a threshold value TH (for example, 100 [msec]), that is, the video is delayed with respect to the sound. At this time, the renderer 77 proceeds to the next step SP28.

  In step SP28, the renderer 77 determines whether or not a “B” picture exists in the output video buffer 79. If a positive result is obtained, the process proceeds to the next step SP29, and if a negative result is obtained, the process proceeds to the next step SP30. Move.

  In step SP29, the renderer 77 determines that the video is behind the audio and confirms that the “B” picture is present in the output video buffer 79, so the B picture (video frame Vf3) is not decoded. By skipping to the output, the delay of the video with respect to the audio can be recovered and the lip sync can be performed, and the process proceeds to the next step SP31 to end the process.

  On the other hand, in step SP30, the renderer 77 uses the fact that the monitor refresh timing of the monitor 13 is 60 [Hz] while the picture refresh timing of the video frame VF2 is 30 [Hz]. By shortening according to the monitor output timing of the monitor 13, the video is caught up with the sound without causing picture quality deterioration due to skipping the picture, and the process proceeds to the next step SP 31 to end the process.

  As described above, the renderer 77 of the real-time streaming decoder 12 in the second content receiving device 4 synchronizes the clock reference pcr of the real-time streaming encoder 11 and the system time clock stc of the real-time streaming decoder 12 in the first content receiving device 3. Therefore, even if the clock reference pcr for that purpose does not arrive without being retransmitted because the real time property is important in UDP, the audio time stamp ATS with respect to the system time clock stc is realized. By executing the lip sync adjustment process according to the deviation of the video time stamp VTS, it is possible to ensure the playback while performing live streaming playback. It is configured so as to be able to Pushinku.

(8) Operation and Effect In the above configuration, the streaming decoder 9 of the first content receiving device 3 uses the preset video time stamp VTSp and the system time clock stc when the content type is composed of audio and video. After presetting the values, be sure to re-set with the preset audio time stamp ATSp, so that the system time clock stc will eventually match the preset audio time stamp ATSp (ATSp1, ATSp2, ATSp3, ...) Let

  The renderer 37 of the streaming decoder 9 includes the count value of the system time clock stc preset by the preset audio time stamp ATSp and the video time stamp VTS (VTS1, VTS1,...) Attached to the video frame VF1 (Vf1, Vf2, Vf3,...). By calculating a difference value D1 from VTS2, VTS3,..., A time difference caused by a difference between the clock frequency of the encoder side with the video time stamp VTS and the clock frequency of the decoder system time clock stc is recognized. be able to.

  The renderer 37 of the streaming decoder 9 repeats and outputs the current picture of the video frame VF1 according to the difference value D1, or skips and outputs the B picture of the non-reference frame without decoding, or By shortening and outputting the refresh timing, it is possible to adjust the output timing of the video for the sound while maintaining the continuity of the sound without interrupting the sound output to the monitor 10.

  Of course, if the difference value D1 is equal to or less than the threshold value TH and the user cannot recognize the lip sync deviation, the renderer 37 can shorten the repeat output, skip reproduction process, or picture refresh interval. Since it can also be output to the monitor 10 according to the video time stamp VTS (VTS1, VTS2, VTS3,...), The continuity of the video can naturally be maintained in this case as well.

  Furthermore, the renderer 77 of the real-time streaming decoder 12 in the second content receiver 4 includes a clock reference pcr supplied from the PCR circuit 61 of the real-time streaming encoder 11 in the first content receiver 3 and a system time clock stc on the decoder side. Since the audio frame AF2 and the video frame VF2 can be output to the monitor 13 in accordance with the audio time stamp ATS and the video time stamp VTS, live streaming reproduction can be realized while maintaining real-time characteristics. .

  In addition, the renderer 77 of the real-time streaming decoder 12 in the second content receiver 4 does not retransmit the clock reference pcr supplied from the PCR circuit 61 of the real-time streaming encoder 11 in the first content receiver 3 using UDP. Therefore, even if the clock reference pcr and the system time clock stc are out of synchronization, the difference value D2V between the system time clock stc and the video time stamp VTS, the system time clock stc and the audio time stamp ATS By calculating the difference value D2A and adjusting the output timing of the video frame VF2 according to the gap between the difference values D2V and D2A, the sound output to the monitor 13 is not interrupted. While maintaining the continuity, it is possible to adjust the output timing of the video for the audio.

  Also, the renderer 37 of the streaming decoder 9 in the first content receiving device 3 follows the preset sequence determined in the order of the preset video time stamp VTSp and the preset audio time stamp ATSp, and the preset video time stamp VTSp and the preset use time stamp. Since the system time clock stc is preset using the audio time stamp ATSp, the system time clock stc can be preset with the audio time stamp ATSp for presetting when the type of content is only audio. If the content type is only video, the preset video time stamp VTSp is used as the system time clock stc. It is possible to preset which types of content consists of audio and video, may also be corresponding if made of only the or a video composed of only audio.

  That is, the renderer 37 of the streaming decoder 9 does not necessarily include audio and video, but the content includes only video and the preset audio time stamp ATSp does not exist, or the content includes only audio. Even when the preset video time stamp VTSp does not exist, the video frame VF1 and the audio frame AF1 can be output. Therefore, the video time stamp VTSp can be output to the monitor 10 at an optimum timing according to the content type. it can.

  Further, in the renderer 37 of the streaming decoder 9, when the difference value D1 between the count value of the system time clock stc after the preset and the video time stamp VTS2 is larger than a predetermined threshold value TH and the video is delayed from the audio, When there is a B picture in the output video buffer 39, the B picture that is not affected by image quality degradation is skipped without decoding, and when there is no B picture in the output video buffer 39, a video frame is synchronized with the monitor output timing of the monitor 10. By shortening the picture refresh timing of VF1, it is possible to catch up video with audio without causing image quality degradation due to picture skip.

  According to the above configuration, the renderer 37 of the streaming decoder 9 in the first content receiver 3 and the renderer 77 of the real-time streaming decoder 12 in the second content receiver 4 are based on the output timings of the audio frames AF1 and AF2. Since the output timing of the video frames VF1 and VF2 can be adjusted, it is possible to lip-sync without making the viewer user feel uncomfortable while maintaining the continuity of the audio.

(9) Other Embodiments In the above-described embodiment, the clock frequency on the encoder side and the decoder are adjusted by adjusting the lip sync according to the difference values D1 or D2V, D2A based on the audio frames AF1, AF2. However, the present invention is not limited to this, and the present invention is not limited to this, and the subtle difference between the clock frequency on the encoder side and the clock frequency on the decoder side caused by clock jitter, network jitter, etc. The shift may be absorbed.

  In the above-described embodiment, the case where the content providing apparatus 2 and the first content receiving apparatus 3 are connected via the Internet 5 to realize pre-encoded streaming has been described. The invention is not limited to this, and the content providing apparatus 2 and the second content receiving apparatus 4 are connected via the Internet 5 so as to realize pre-encoded streaming, Pre-encoded streaming may be realized by providing content to the second content receiving device 4 via the content receiving device 3.

  Furthermore, in the above-described embodiment, the case where live streaming is performed between the first content receiving device 3 and the second content receiving device 4 has been described. However, the present invention is not limited to this, and content Live streaming may be performed between the providing device 2 and the first content receiving device 3 or between the content providing device 2 and the second content receiving device 4.

  In this case, the clock reference pcr is transmitted from the streaming server 8 of the content providing device 2 to the streaming decoder 9 of the first content receiving device 3, and the clock decoder pcr and the system time clock stc are synchronized by the streaming decoder 9. Live streaming can be realized.

  Further, in the above-described embodiment, the case where the values of the first video time stamp VTS1 and the audio time stamp ATS1 are pulled back by a predetermined time by the subtractor circuits 44 and 45 has been described, but the present invention is not limited to this. First, when the first video time stamp VTS1 and audio time stamp ATS1 arrive at the comparator circuit 46 via the buffers 42 and 43, the presets are supplied from the STC circuit 41 to the comparator circuit 46 due to delays by the buffers 42 and 43, etc. If the value of the later system time clock stc has not passed the video time stamp VTS1 and the audio time stamp ATS1, the subtractor circuits 44 and 45 do not necessarily provide the first video time stamp VTS. The value of S1 and audio time-stamp ATS1 may not pulled back by a predetermined time period a.

  Furthermore, in the above-described embodiment, the system time clock stc is determined according to a preset sequence determined in the order of the preset video time stamp VTSp and the preset audio time stamp ATSp regardless of the content type before determining the content type. However, the present invention is not limited to this, and the type of content is first determined. When the content is composed of audio and video, the preset video time stamp VTSp, If the system time clock stc is preset with the audio time stamp ATSp and the content is composed only of video, the system time clock is set with the preset video time stamp VTSp. Presetting the stc, when the content is made of only the audio is preset audio time-stamp ATSp may be preset system time clock stc.

  Further, in the above-described embodiment, when no B picture exists in the output video buffers 39 and 79, the picture refresh rate of the video frames VF1 and VF2 is set to 30 [Hz] in accordance with the monitor output rate of the monitors 10 and 13. However, the present invention is not limited to this, and the picture refresh rate of the video frames VF1 and VF2 is changed from 30 [Hz] to 60 [Hz] regardless of the presence or absence of the B picture. You may make it shorten to. Even in this case, the renderers 37 and 77 can recover the delay of the video with respect to the audio and perform lip sync.

  Furthermore, in the above-described embodiment, the case where the B picture is skipped and output has been described. However, the present invention is not limited to this, and the P picture located immediately before the I picture is skipped and output. Anyway.

  This is because if the P picture is located immediately before the I picture, the P picture is not referred to when the next I picture is generated, and even if it is skipped, there is a problem in generating the next I picture. This is because the image quality is not deteriorated.

  Further, in the above-described embodiment, the case where the video frame Vf3 is skipped without being decoded and output to the monitor 10 has been described. However, the present invention is not limited to this, and the video frame Vf3 is output after being decoded. At the stage of outputting from the video buffer 39, the decoded video frame Vf3 may be skipped and output.

  Further, in the above-described embodiment, since the audio frames AF1 and AF2 are used as a reference when adjusting the lip sync, all the audio frames are output to the monitors 10 and 13 without any loss. However, the present invention is not limited to this. For example, when there is an audio frame corresponding to a silent portion, the audio frame may be skipped and output.

  Furthermore, in the above-described embodiment, the content receiving apparatus of the present invention includes audio decoders 35 and 74, video decoders 36 and 76 as decoding means, input audio buffers 33 and 73 as output storage means, output audio buffer 38, 78, the input video buffers 34 and 75, the output video buffers 39 and 79, and the renderers 37 and 77 serving as calculation means and timing adjustment means have been described. However, the present invention is not limited to this. The content receiving device may be formed with various other circuit configurations.

  The content receiving apparatus, the video / audio output timing control method, and the content providing system according to the present invention can be applied, for example, to downloading and displaying moving image content with sound from a server.

1 is a schematic block diagram illustrating an overall configuration of a content providing system that represents the entire contents of a streaming system. It is a rough block diagram which shows the circuit structure of a content provision apparatus. It is a basic diagram which shows the structure of the time stamp (TCP protocol) in an audio packet and a video packet. It is a basic block diagram which shows the module structure of the streaming decoder in a 1st content receiver. 2 is a schematic block diagram illustrating a configuration of a timing control circuit. FIG. It is a basic diagram which shows the time stamp compared with STC after a preset. It is a basic diagram provided when demonstrating the output timing of the video frame and audio frame in pre-encoded streaming. It is an approximate line figure used for explanation of video frame output control processing in the case of I picture and P picture. It is a flowchart which shows the lip-sync adjustment process sequence in pre-encoded streaming. It is a basic-line block diagram which shows the circuit structure of the real-time streaming encoder in a 1st content receiver. It is a basic diagram which shows the structure of PCR (UDP protocol) in a control packet. It is a basic-line block diagram which shows the circuit structure of the real-time streaming decoder in a 2nd content receiver. It is an approximate line figure used when explaining the output timing of the video frame and audio frame in live streaming. It is a basic flowchart which shows the lip-sync adjustment process sequence in live streaming.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Content provision system, 2 ... Content provision apparatus, 3 ... 1st content receiver, 4 ... 2nd content receiver, 5 ... Internet, 7 ... Encoder, 8 ... Streaming server, 9: Streaming decoder, 10, 13: Monitor, 11: Real-time streaming encoder, 12: Real-time streaming decoder, 14: Web server, 15: Web browser, 21, 51 ... Video input unit, 22, 52... Video encoder, 23... Video ES storage section, 24 and 53... Audio input section, 25 and 54. Audio encoder, 26... Audio ES storage section, 28 and 57. 58 …… Audio frame counter, 27, 56 …… Packet generation , 30, 59... Packet data storage unit, 31, 71... Input packet storage unit, 32, 72... Packet division unit, 33, 73... Input audio buffer, 34, 75. 74 ... Audio decoder, 36, 76 ... Video decoder, 37, 77 ... Renderer, 38, 78 ... Output audio buffer, 39,79 ... Output video buffer, 40 ... Crystal oscillator circuit, 81 ... Subtraction Circuit, 82... Filter, 83... Voltage controlled crystal oscillator, 41, 60, 84... STC circuit, 61.

Claims (8)

Providing content on the encoder side with a plurality of encoded video frames sequentially attached with video time stamps based on a reference clock on the encoder side and a plurality of encoded audio frames sequentially attached with audio time stamps based on the reference clock Decoding means for receiving and decoding from the device;
Storage means for storing a plurality of video frames and a plurality of audio frames obtained as a result of decoding the encoded video frame and the encoded audio frame by the decoding means;
Calculating means for calculating a time difference caused by a deviation between the clock frequency of the reference clock on the encoder side and the clock frequency of the system time clock on the decoder side;
Timing adjustment means for adjusting a video frame output timing when sequentially outputting the plurality of video frames in units of frames with reference to an audio frame output timing when the plurality of audio frames are sequentially output in units of frames according to the time difference A content receiving apparatus comprising: 2. The content according to claim 1, wherein the timing adjustment unit outputs the video frame according to the video time stamp based on a system time clock on the decoder side when the time difference is shorter than a predetermined time. Receiver device. Receiving means for receiving a reference clock on the encoder side transmitted by UDP (User Datagram Protocol), which requires real-time performance from the content providing device;
The calculation means synchronizes the reference clock on the encoder side with the system time clock on the decoder side, and then calculates the difference between the clock frequency of the reference clock on the encoder side and the clock frequency of the system time clock on the decoder side. The content receiving apparatus according to claim 1, wherein a time difference that occurs is calculated. The content receiving device is:
In accordance with a preset sequence determined by the order of the video time stamp and the audio time stamp, preset means for presetting the system time clock on the decoder side using the video time stamp or the audio time stamp,
The content receiving apparatus according to claim 1, wherein the calculating means calculates the time difference caused by a difference between a clock frequency of the reference clock on the encoder side and a clock frequency of the system time clock after the presetting. The content receiving device is:
By following a preset sequence determined in the order of the video time stamp and the audio time stamp, if the content type is only audio, the system time clock on the decoder side is preset using the audio time stamp. Preset means;
The content receiving apparatus according to claim 1, further comprising: an audio output unit that outputs audio of the audio frame based on the system time clock after the preset and the audio time stamp. The content receiving apparatus is
By following a preset sequence determined in the order of the video time stamp and the audio time stamp, the system time clock on the decoder side is preset using the video time stamp when the content type is only video. Preset means;
2. The content receiving apparatus according to claim 1, further comprising video output means for outputting the video frame based on the preset system time clock and the video time stamp. A plurality of encoded video frames sequentially attached with video time stamps based on a reference clock on the encoder side and a plurality of encoded audio frames sequentially attached with audio time stamps based on the reference clock to the decoding means. A decoding step for receiving and decoding from the content providing apparatus on the encoder side;
A storage step of storing a plurality of video frames and a plurality of audio frames obtained as a result of decoding the encoded video frame and the encoded audio frame in the decoding step;
A difference calculating step for causing the calculating means to calculate a time difference caused by a difference between the clock frequency of the reference clock on the encoder side and the clock frequency of the system time clock on the decoder side;
Video frame output when sequentially outputting the plurality of video frames in units of frames with reference to the audio frame output timing when the plurality of audio frames are sequentially output in units of frames in response to the time difference. A video audio output timing control method comprising: a timing adjustment step for adjusting timing. A content providing system having a content providing device and a content receiving device,
The content providing apparatus includes:
Encoding means for generating a plurality of encoded video frames with video time stamps based on a reference clock on the encoder side, and a plurality of encoded audio frames with audio time stamps based on the reference clock;
Transmitting means for sequentially transmitting the plurality of encoded video frames and the plurality of encoded audio frames to the content receiving device;
The content receiving apparatus is
Decoding means for receiving and decoding a plurality of encoded video frames sequentially attached with the video time stamp and a plurality of encoded audio frames sequentially attached with the audio time stamp from the content providing apparatus on the encoder side; ,
Storage means for storing a plurality of video frames and a plurality of audio frames obtained as a result of decoding the encoded video frame and the encoded audio frame by the decoding means;
Calculating means for calculating a time difference caused by a deviation between the clock frequency of the reference clock on the encoder side and the clock frequency of the system time clock on the decoder side;
Timing adjustment means for adjusting a video frame output timing when sequentially outputting the plurality of video frames in units of frames with reference to an audio frame output timing when the plurality of audio frames are sequentially output in units of frames according to the time difference A content providing system characterized by comprising:

Images