CN100473158C

CN100473158C - Method and apparatus for processing, transmitting and receiving dynamic image data

Info

Publication number: CN100473158C
Application number: CN 200610009301
Authority: CN
Inventors: 山口孝雄; 栄藤稔; 荒川博
Original assignee: Matsushita Electric Industrial Co Ltd
Current assignee: Panasonic Holdings Corp
Priority date: 1997-03-17
Filing date: 1998-03-13
Publication date: 2009-03-25
Anticipated expiration: 2018-03-13
Also published as: CN101616317A; CN1941889A; CN100589545C; CN100473159C; CN1941888A; CN1941890A; CN1819654A; CN1819655A; CN1941888B; CN1941889B

Abstract

The invention comprises a reception control section (11) for receiving information including data and sending format information thereof from a memory or a communication channel; a separation section (12) for analyzing and separating the received information; a sending section (13) for sending the information to the memory or a transmission channel; an image extension section (14) for extending an image; an image extension control section (15) for controlling the treatment state of the video extension section (14) to extend at least one or more images; image synthesis equipment consisting of image synthesis sections (16) for synthesizing image according to extension information; an output section (17) for outputting the synthesis result; and a terminal control section (18) for controlling the above devices to simultaneously synthesize a plurality of images corresponding to the dynamic range of the sending format information.

Description

The method of transmission and receiving dynamic image data

The application is to be that March 13, application number in 1998 are 200410032365.9 divide an application the applying date.

Technical field

The present invention relates to a kind of audio-video transmitter and audio-video receiver, data processing equipment and method, Wave data sending method and equipment and Wave data method of reseptance and equipment and video transmission method and equipment and video receiving method and equipment.

Background technology

Existing a kind of equipment, it has satisfied the in esse requirement of the other side's image of feeling that you occur in front, its purpose is to carry out real world images communication, this is by extract someone image from the landscape image that you are for example arranged, then, with this person's the image (being someone image that the other side sends) and the virtual space image mutual superposition that will show, and shown (the Japanese patent application No.4-24914) that realizes with the other side's image of storage in advance.

Specifically, under the situation of prior art, had that to relate to the accelerogram picture synthetic and reduce some inventions (Official Journal of the open No.5-46592 of Japan Patent for example: image synthesizer) of the method for memory.

Though prior art has proposed to utilize the synthetic communication system that bidimensional rest image or three-dimensional CG data are synthesized of image, but from following viewpoint, synthetic simultaneously multiple image and multiple sound and the system that shows, its implementation is not also specifically discussed.

That is, the method for enumerating from below, existing does not have the concrete problem of discussing.

(A1) a kind of method, its utilizes the environment that transmits data and control information (by the information of the data packet transmission different with control terminal side deal with data) independently at many logical transport circuits that are made of software on the one or more actual transmissions circuit to transmit (communicate by letter and broadcast) and control chart picture and sound down.

(A2) a kind of method, it dynamically changes will be added to image to be sent or the header message in the voice data (corresponding to control information of the present invention);

(A3) a kind of method, it dynamically changes the header message (corresponding to control information transmission of the present invention) that will add, so that transmit;

(A4) a kind of method, it by dynamic moving multiple connection with separate many logical transport circuits and come transmission information;

(A5) a kind of method, it has considered reading of program or data and growth cycle, with images and sound;

(A6) a kind ofly quick switching images and sound method have been considered.

Yet, with regard to the data volume that dynamic adjustment will be transferred to network, method (the distribution self-adaptation type VOD systematic research that H.Jinzenji and T.Tajiri write that has proposed to change the method for coded system and frame data are discussed according to the type of picture frame, D-8 publishes in the D-8 (1995) of electronics, information and society of system of communication engineering association (IEICE)).

The dynamic throughput conversion algorithm that high quality graphic can be provided under the restricted processing time has proposed as a kind of throughput of adjusting coder side that (the dynamic throughput conversion of the utilization algorithm that T.Osako, Yajima, H.Kodera, H.Watanabe, K.shimamura etc. write is encoded to software video, publish paper periodical D-2 in IEICE, Vol.80-D-2, the second phase, the 444-458 page or leaf, 1997).

And, exist the MPEG1/MPEG2 system as the example of realizing reproduced in synchronization image and sound.

(B1) conventional method of image being discussed corresponding to the type of picture frame has such problem, promptly by handling a plurality of video data streams or a plurality of audio data stream, be difficult to sound synchronously emphasis reproduce important scene, reflection editor's intention, this is because the grade of manageable information is arranged in single information flow.

(B2) and, because its necessary condition is to realize MPEG1/MPEG2 with hardware, so decoder must be decoded to each bit stream that provides.Therefore, its problem is exactly how to adapt to the situation of decoder by output that exceed.

In addition, for images, more existing systems, for example H.261 (the professional video coder-decoder of ITU-T suggestion px.64 acoustic image H.261), and they use hardware composing.Therefore, owing to considered the upper limit of necessary performance during hardware, so there not be generation to finish the situation of decoding in can not be at the appointed time in design.

The top fixed time represents to send the needed time of bit stream that image of coding obtains.If decoding can not be finished in this time, its extra time just becomes delay.If this delay accumulation, then transmitter side to receiver side just increase of delay, system just can not be used as video telephone.This state must be avoided.

And, when finishing decoding in can not be at the appointed time because communication counterpart has produced the bit stream beyond the regulation, then with regard to produce can not images problem.

The problems referred to above not only are created in the video data, and are created in the voice data.

Yet, in recent years, because the expansion that the layout of personal computer (PC) network environment that forms is brought the Internet and ISDN so transmission rate improves, can utilize PC and Network Transmission image.And the requirement of user's images also grows at top speed.Again since cpu performance improve, so can come decoding with software fully to vision signal.

Yet, because different personal computer on the structure, for example the different personal computers in aspect such as CPU, highway width or accelerator can be carried out identical software, are difficult so consider the upper limit of necessary performance in advance, therefore, produced the at the appointed time problem of interior decoding of image.

And, when sending the video data encoder that length surpasses the receiver throughput, finish coding in just can not be at the appointed time.

Problem (C1): the delay of decoded picture in reducing at the appointed time.

When the Wave data that vision signal is entered as claim C1 of the present invention, perhaps the Wave data that vision signal is output into claim C7 of the present invention is when dealing with problems 1 means, remaining issues is not utilize the real work efficient reduction of transmission line because part is transmitted bit stream.And, exist some according to last coded image (for example P image), produce the coded system of current decoded video.Yet,, change in time and make the serious problem of deterioration in image quality so exist because last decoded picture recovers fully by 1 the means of dealing with problems.

Problem (C2): under the situation of 1 means of dealing with problems, the real work efficient of transmission line reduces.And, enlarged deterioration of image quality.

Under the situation of installing with software, the frame rate of image is by the needed time appointment of once encoding.Therefore, when the frame rate of user's appointment surpasses the throughput of computer, can not meet this appointment.

Problem (C3): when the frame rate of user's appointment surpasses the throughput of computer, can not meet this appointment.

The problem (A1) of considering first kind of prior art the object of the present invention is to provide a kind of audio-video transmitter and audio-video receiver and data processing equipment and method, one of to address the above problem at least to (A6).

And the problem (B1) of considering second kind of prior art another object of the present invention is to provide a kind of data processing equipment and method, one of to address the above problem at least to (B2).

Again, the problem (C1) of considering the third prior art is to (C3), a further object of the present invention is to provide a kind of Wave data method of reseptance and equipment, Wave data sending method and equipment and video transmission method and equipment and video receiving method and equipment, one of to address the above problem at least.

Summary of the invention

The present invention according to claim 1 is a kind of processing method, it is characterized in that,

The priority that utilization is added in the described coded message is transmitted described coded message, thereby compare the result with decoding that is added to described coded message or output cycle and judge that described coded message is written to transmit buffer to be postponed, and then makes the minimizing of transmission information beginning to transmit the back elapsed time if (1) actual transfer rate surpasses video or audio-frequency information target transmission speed or (2).

A kind of data processing equipment according to claim 2 is characterized in that,

Having (1) actual transfer rate surpasses video or audio-frequency information target transmission speed or (2) and compares the result with decoding that is added to described coded message or output cycle and judge that described coded message is written to transmit buffer to be postponed, and then makes the minimizing of transmission information beginning to transmit the back elapsed time.

A kind of data processing method according to claim 3 is characterized in that,

Order input category time series data and precedence information thereof;

(1) when described classification time series data information is damaged, carries out re-send request may and handle, with the described data that are damaged of request repeat; (2) when described classification time series data is lost continuously or often lost, only high-priority data is carried out described re-send request may and handle.

A kind of data processing equipment according to claim 4 is characterized in that,

Order input category time series data and precedence information thereof;

A kind of data processing method according to claim 5 is characterized in that,

Order input category time series data and precedence information thereof;

According to the amount of the described classification time series data that will transmit, the described high-priority data of prioritised transmission.

A kind of data processing equipment according to claim 6 is characterized in that,

Order input category time series data and precedence information thereof;

A kind of data processing method according to claim 7 is characterized in that,

When the actual transfer rate of the time series data information of transmitting encoded video or audio frequency surpasses target transmission speed, with priority in the time series data of priority and the processing priority of the interframe of the time series data of representing to constitute described video between time series data additional, that represent the processing priority between described time series data in the described encoded information, reduce transmission to the described information of the transmitting device of transmission time sequence data.

A kind of data processing equipment according to Claim 8, it is characterized in that, comprise with lower device, this device carries out when the actual transfer rate of the information of the time series data of transmitting encoded video or audio frequency surpasses target transmission speed, with additional in the described encoded information, represent that priority and expression between the time series data of the processing priority between described time series data constitute priority in the time series data of processing priority of interframe of time series data of described video, reduce transmission to the described information of the transmitting device of transmission time sequence data.

A kind of data processing method according to claim 9 is characterized in that,

The time series data of order input video or audio frequency, and represent that priority and expression between the time series data of the processing priority between described time series data constitute priority in the time series data of processing priority of time series data interframe of described video, (1) when the information of described time series data is damaged, with priority between priority in the described time series data and described time series data, carry out handling for the requirement of sending again of sending again that requires its damaged data, or (2) are when described time series data is lost continuously or frequently, with priority between priority in the described time series data and described time series data, only the high data of processing priority are carried out described sending again and require to handle.

A kind of data processing equipment according to claim 10, it is characterized in that, have with lower device, the time series data of this device sequence input video or audio frequency, and represent that priority and expression between the time series data of the processing priority between described time series data constitute priority in the time series data of processing priority of time series data interframe of described video, (1) when the information of described time series data is damaged, with priority between priority in the described time series data and described time series data, carry out handling for the requirement of sending again of sending again that requires its damaged data, or (2) are when described time series data is lost continuously or frequently, with priority between priority in the described time series data and described time series data, only the high data of processing priority are carried out described sending again and require to handle.

A kind of data processing method according to claim 11 is characterized in that,

Time series data with video or audio frequency, and represent that priority and expression between the time series data of the processing priority between described time series data constitute that priority lumps together as output in the time series data of processing priority of interframe of time series data of described video, according to the traffic volume of described time series data, the preferential high data of processing priority that send based on priority is obtained between priority and time series data in the described time series data.

A kind of data processing equipment according to claim 12, it is characterized in that, this device has the time series data with video or audio frequency, and represent that priority and expression between the time series data of the processing priority between described time series data constitute that priority lumps together as output in the time series data of processing priority of interframe of time series data of described video, according to the traffic volume of described time series data, the preferential high data of processing priority that send based on priority is obtained between priority and time series data in the described time series data.

A kind of data processing method according to claim 13 is characterized in that,

Utilization is arranged at the processing priority of the additional described time series data of priority appendix in the encoding process means of encoding process of the time series data of carrying out video or audio frequency, simultaneously, relatively be additional to the period that maybe should export that should decode in the information of time series data of encoded video or audio frequency, with elapsed time from transmission process, be judged to be when writing of the encoded information of transmission buffer lags behind

With being additional to priority between time series data in the described encoded information, that represent the processing priority between described time series data, constitute priority in the processing priority time series data of in-frame encoding picture interframe of time series data of described video with expression, reduce transmission to the information of described encoding process means.

A kind of data processing equipment according to claim 14; It is characterized in that; The processing priority of the additional described time series data of priority appendix in the coding processing means that the coding that this device utilization is arranged at the time series data of carrying out video or audio frequency is processed; Simultaneously; The information of time series data of encoded video or audio frequency of relatively the being additional to total period that maybe should export that should decode, with elapsed time from transmission process; Be judged to be when lagging behind to writing of the encoded information of transmission buffer

With being additional to priority between time series data in the described encoded information, that represent the processing priority between described time series data, constitute priority in the processing priority time series data of interframe of time series data of described video with expression, reduce transmission to the information of described encoding process means.

A kind of data processing method according to claim 15 is characterized in that,

When the message transmission rate of information that comprises the time series data of encoded audio frequency or in-frame encoding picture frame video in transmission surpasses target transmission speed, with be additional to priority between time series data in the described encoded information, that represent the processing priority between described time series data, with the time series data of the processing priority of the in-frame encoding picture interframe of the time series data of representing to constitute described video in priority, extract and carry out transmission to the described information of the transmission means of transmission time sequence data.

A kind of data processing method according to claim 16 is characterized in that,

Utilization is arranged at the processing priority of the additional described time series data of priority appendix in the encoding process means of encoding process of the time series data of carrying out video or audio frequency, simultaneously, the information of time series data of encoded video or audio frequency of relatively the being additional to total period that maybe should export that should decode, with elapsed time from transmission process, be judged to be when writing of the encoded information of transmission buffer lags behind

A kind of data processing method according to claim 17 is characterized in that,

The order input contains the time series data of audio frequency or in-frame encoding picture frame video, and represent that priority and expression between the time series data of the processing priority between described time series data constitute priority in the time series data of processing priority of in-frame encoding picture interframe of time series data of described video, (1) when the information of described time series data is impaired, with priority between priority in the described time series data and described time series data, carry out handling for the requirement of sending again of sending again that requires its damaged data, or (2) are when described time series data is lost continuously or frequently, with priority between priority in the described time series data and described time series data, only the data that processing priority is high are carried out the described requirement processing of sending again.

A kind of data processing method according to claim 18 is characterized in that,

The time series data that will contain audio frequency or in-frame encoding picture frame video, and represent that priority and expression between the time series data of the processing priority between described time series data constitute priority in the time series data of processing priority of in-frame encoding picture interframe of time series data of described video, lump together as output, according to the traffic volume of described time series data, the preferential high data of processing priority that send based on priority is obtained between priority and time series data in the described time series data.

Summary of drawings

Fig. 1 is the schematic block diagram of the audio-video transceiver of one embodiment of the invention;

Fig. 2 is the schematic diagram that receives control section and separating part;

Fig. 3 is the schematic diagram that utilizes the method for the transmission of a plurality of logical transport circuit and control of video and audio frequency;

Fig. 4 is the schematic diagram that dynamically changes the method that is added to the video wanted or the header message in the voice data;

Fig. 5 (a) and 5 (b) are the schematic diagrames that increases the method for AL information;

Fig. 6 (a) is the schematic diagram of example that increases the method for AL information to 6 (d);

Fig. 7 is by dynamically moving multiplexed and the schematic diagram that separates a plurality of logical transport circuit method of transmitting information;

Fig. 8 is the schematic diagram of transmission broadcast program process;

Fig. 9 (a) shows when program or data occurring at receiving terminal, has considered the image or the sound sending method with rise time of reading of program or data;

Fig. 9 (b) shows when transmission program or data, considered program or data read or rise time transmission of video or the method for audio frequency;

Figure 10 (a) is the schematic diagram corresponding to fast switching method;

Figure 10 (b) is the schematic diagram corresponding to fast switching method;

Figure 11 (a) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 11 (b) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 12 shows the object lesson of the agreement of actual transmissions between terminal;

Figure 13 (a) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 13 (b) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 13 (c) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 14 shows the object lesson of the agreement of actual transmissions between terminal;

Figure 15 shows the object lesson of the agreement of actual transmissions between terminal;

Figure 16 (a) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 16 (b) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 17 shows the object lesson of the agreement of actual transmissions between terminal;

Figure 18 shows the object lesson of the agreement of actual transmissions between terminal;

Figure 19 (a) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 19 (b) shows the object lesson of the agreement of actual transmissions between terminal;

Figure 20 (a) to 20 (c) be the block diagram of the demonstration system of CGD of the present invention;

Figure 21 increases the method for priority when showing the encoder overload;

When Figure 22 shows overload, determine the method for priority at receiving terminal;

Figure 23 shows Iterim Change priority;

Figure 24 shows data flow priority and object priority;

Figure 25 is the schematic block diagram of the video encoder and the Video Decoder of one embodiment of the invention;

Figure 26 is the schematic block diagram of the audio coder and the audio decoder of one embodiment of the invention;

The priority of control processed increased part and priority determining section when Figure 27 (a) and 27 (b) showed overload;

Figure 28 (a) and 28 (b) show the increase priority level;

Figure 29 shows the method for distributing priority to the multiresolution video data;

Figure 30 shows the method that constitutes the communication pay(useful) load;

Figure 31 illustrates the method that produces data corresponding to the communication pay(useful) load;

Figure 32 shows the relation between object priority, data flow priority and the communication data packet priority;

Figure 33 is the block diagram of the transmitter of first embodiment of the invention;

Figure 34 is the schematic diagram of first embodiment;

Figure 35 is the block diagram of the receiver of third embodiment of the invention;

Figure 36 is the block diagram of the receiver of fifth embodiment of the invention;

Figure 37 is the schematic diagram of the 5th embodiment;

Figure 38 is the block diagram of the transmitter of sixth embodiment of the invention;

Figure 39 is the block diagram of the transmitter of eighth embodiment of the invention;

Figure 40 is the flow chart of the sending method of second embodiment of the invention;

Figure 41 is the flow chart of the method for reseptance of fourth embodiment of the invention;

Figure 42 is the flow chart of the sending method of seventh embodiment of the invention;

Figure 43 is the flow chart of the sending method of ninth embodiment of the invention;

Figure 44 is the block diagram of audio-video transmitter of the present invention;

Figure 45 is the block diagram of audio-video receiver of the present invention;

Figure 46 shows the key-drawing to the priority adding set of the video of audio-video transmitter of the present invention and voice data increase priority;

Figure 47 shows and is added to the video of audio-video receiver of the present invention and the priority in the voice data by explanation and determines whether that the priority of decoding determines the key-drawing of device.

Symbol description

11 receive control section

12 separating parts

13 send part

14 video expansions (image spreading part)

15 videos expansion control section (image spreading control section)

16 video composite parts (image synthesizing section)

17 outputs

18 terminal control sections

4011 send control section

4012 video codings parts (image encoding part)

4013 receive control section

4014 video decodes parts (picture decoding part)

4015 video composite parts (image synthesizing section)

4016 outputs

4101 video encoders (image encoder)

4102 Video Decoders (image decoder)

301 receiving systems

302 estimation units

303 Video Decoders (being dynamic image or decoding moving image device)

304 number cutting devices

306 outputs

307 inputs

3031 variable decoding devices

3032 anti-quadrature converting means

3033 converting units

3034 motion compensating devices

3035 time of implementation measurement mechanisms

Embodiments of the present invention

Embodiments of the invention are described with reference to the accompanying drawings.

The embodiment that describes below mainly addresses the above problem each in (A1) to (A6).

Used " image " of the present invention comprises rest image and live image.And the image of proposition can be to resemble the two dimensional image of computer graphics (CG) or the 3 d image data that is made of wire-frame model.

Fig. 1 is the schematic block diagram of the audio-video transceiver of one embodiment of the invention.

In Fig. 1, the reception control section 11 of reception information and the transmission part 13 of the information of transmission are the information carrying means such as coaxial cable, CATV, LAN and modulator-demodulator etc.Communication environment can be the environment that can need not consider multiplex machine with many logical transport circuits, and for example the Internet perhaps can be the environment that must consider multiplex machine, for example analog telephone or satellite broadcasting.

And, as the terminal connected system, can be set forth in, or the system of broadcast-type video or voice data is broadcast in broadcasting via satellite, CATV or the Internet such as the system that carries out two-way transmission video or voice data between the terminals such as video telephone or TeleConference Bridge.The present invention has considered these terminal connected systems.

Separating part 12 shown in Figure 1 is to be used to analyze information that receives and the device that data are separated with control information.Specifically, part 12 is to decomposite transmission header message and the data that are added in the data, perhaps decomposites the Data Control stem and the data content that are added in the data.Image spreading part 14 is to expand the device of the image that receives.For example, image expanding can be the compressed image of standardized activity (dynamically) or rest image, for example H.261, H.263, MPEG 1/2 or JPEG etc.

Image spreading control section 15 shown in Figure 1 is the devices that are used for the monitoring picture extended mode.For example,,, reception buffer can read the sky reception buffer when overflowing when almost causing by the extended mode of monitoring picture, and expanded images not, and be ready to image so that restart expanded images after the expansion.

In Fig. 1, image synthesizing section 16 is the devices that are used for synthetic expanded images.Image combining method can be by using such as scripts such as JAVA, VRML or MHEG, describe image and structural information thereof and come (display position and demonstration time (and, can comprise the display cycle)), the relation between the method to image packets, image display layer (degree of depth), object ID (SSRC that describes later on) and their attribute defines.The script of describing synthetic method inputs or outputs by network or local storage.

Output 17 is display or printer, is used for the synthetic result of output image.Terminal control section 18 is the devices that are used to control each part.In addition, can adopt the structure of expanded sound rather than image (can be by image spreading is partly made into the sound expansion, the image spreading control section is made into sound expansion control section, image synthesizing section is made into speech synthesiser assign to constitute this structure), perhaps adopt expanded images and sound, the structure that the two is also shown synchronously, however keep synchronously temporary transient.

Also can utilize the image compression part of compressed image.Control chart partly sends image and sound as the contract sound compression control of part of the sound compression section of the image compression control section of compression section, compression sound and guide sound sound press.

Fig. 2 shows and receives control section and separating part.

The reception control section 11 of Fig. 1 is made of the control information receiving unit 102 of the control information of Data Receiving part 101 that receives data and reception control data, separating part 12 is explained the transmission form storage area 103 of the transmission structure (back detailed description) that sends content by storage and constitutes according to the transmission information interpretation part 104 that is stored in the transmission interpretation of structure transmission content in the form storage area 103 that this structure can receive data and control information independently.Therefore, for example, be not difficult deletion or mobile these data in receiver, video or voice data.

As mentioned above, the communication environment that can receive control section 11 requirements can utilize the environment (internet-class environment) that can use many logical transport circuits and need not consider multiplex machine, as the Internet, also can utilize the environment (former class environment) of multiplex machine, as considering analog telephone or satellite broadcasting etc.Yet the communication environment in user dwelling has many logical transport circuits (logical channel) (for example, under the situation of the communication environment that can use TCP/IP, using the statement that is called " communication port " usually).

As shown in Figure 2, suppose to receive control section 11 and receive the data transmission link of one or more types and the control logic transmission line of one or more types, to control the data that to transmit.Also can prepare plurality of transmission lines transmission data, only prepare a transmission lines control data.Can also prepare transmission lines control data when each transfer of data, as also being used for RTP/RTCP H.323.When considering to utilize the broadcasting of UDP, can use the communication system of utilizing single communication port (multicast address).

The schematic diagram of Fig. 3 is used to explain the method for utilizing the transmission of many logical transport circuits and control of video and voice data.The data that send are called ES (elementary stream), and it can be the image information of a frame or the image information among the GOB, perhaps under the situation of image less than the macro block of a frame.

Under the situation of voice data, the regular length that can use the user to determine.And, the Data Control header message that be added in the data that will send is called AL (adaptation layer information).The information of priority that whether is information, the information of expression data reproduction time and the expression data processing of starting position that can deal with data to expression is classified AL information as.Data control information of the present invention is corresponding to AL information.And the used ES of the present invention always must be not consistent with MPEG 1/2 defined content with AL.

Whether expression is that the information of starting position that can deal with data specifically comprises two types information.First kind is the random access mark, that is, expression can be read individually and the independent information of reproducing the front and back data, for example (I image) in the frame under the situation of image.Second kind is the information that can be defined as access flag the mark that expression can read individually, that is, the expression starting position is the GOB image header or is the information of macroblock header under the situation of image.Therefore, there is not access flag to represent the middle part of data.These two kinds of random access marks and random labelling are not always to need as the information of the starting position of expression energy deal with data.

Such a case is arranged,, then can not have problems if under situation, do not add this two kinds of marks such as the real time communication of videoconference etc.Yet in order to simplify editing, the random access mark is essential.Also can be before transmitting data, it is essential to mark whether by the communication port decision, perhaps needs which kind of mark.

The information of the time synchronized when the information representation reproduced image of designation data recovery time and sound, under the situation of MPEG1/2, it is called PTS (representing time tag).Owing under the situation such as real time communication such as videoconferences, generally do not consider time synchronized, the information of expression recovery time is always unessential.The time interval between the coded frame can be essential information.

Adjust the time interval at receiver side, can prevent that interframe is separated with than great fluctuation process.Yet the receiver side adjustment is reproduced at interval and may be postponed.Therefore, can judge that the temporal information of representing frame period between the coded frame is unnecessary.

For whether the information of determining the expression data reproduction time represents PTS or frame period, also can determine before the transmission data, the data reproduction time is not added in the data, and this decision is sent to receiving terminal by communication port, and send data with the data control information of decision.

When because the load of the load of receiving terminal or network former thereby can not handle or send the information of priority of expression deal with data the time can stop to handle or sending data, reduce the load of receiving terminal or network.

Receiving terminal can be used image spreading control section 15 deal with data, and network can be with link terminal or router deal with data.Can represent priority with numerical value or mark.And, when the load of receiving terminal or network has unexpected fluctuation, the deviant of the information of expression data processing priority is sent with data as control information or data control information (AL information), and with this deviant and the priority value addition of allocating group video or voice data in advance, thus, can dynamic priority be set corresponding to the mode of operation of system.

And, differentiating whether scrambled is arranged, whether copyright is arranged and be original paper or duplicate etc. information as control information with and the data identifier (SSRC) that separates of data send, be reduced at via node removal scrambled.

The information of expression data processing priority can be added in each data flow with the frame set of many images or sound or every frame video or voice data formation.

Provide the priority adding set for sending terminal unit, this device is under overload situations, utilization is determined the processing priority of coded message such as H.263 or G.723 waiting coding method according to pre-defined rule, and makes coded message corresponding to the priority of determining (seeing Figure 46).

Figure 46 is used for explaining that priority adding set 5201 is added to priority in the schematic diagram of image or sound.

That is, as shown in figure 46,, priority is added in coding video frequency data (being handled by video coding apparatus 5202) and the coding audio data (being handled by audio coding apparatus 5203) according to predetermined rule.Adding the rale store of priority adds in the rule 5204 in priority.These rules comprise that a high priority of priority than P frame (inter coded images frame) adds to the rule of I frame (in-frame encoding picture frame) and the rule that a priority lower than the priority of voice data adds to image.And, can change these rules according to user's appointment.

Under the situation of image, it be to become to change that priority is added object, under the situation of the image frame data of editor or user's appointment or sound, then is ensonified zone and no sound area.

In order in picture frame or audio frame, to add priority, with predetermined processing priority under the situation of overload, consider to use following method: priority is increased to the method for communication stem and priority is embedded into the method in the stem of the bit stream of video or audio data coding when encoding, last method can obtain precedence information under the situation of not decoding, then a method can be handled single bit stream independently with the ground of system independence.

When a picture frame (for example P of the I frame of intraframe coding or interframe encode or B frame) when being divided into a plurality of transmits data packets, under the situation of image, only priority is added to the communication stem, so that the stem of addressable picture frame is sent (when priority is the same in the phase diagram picture frame, can think that priority does not change before the addressable picture frame of the next one occurs) as information independently.

And the scope of the value of making can be represented priority variable (for example, according to purposes, with 16 or 32 bit representation temporal informations), can realize the structure consistent with control information.

Under the situation of decoder, the priority when transshipping according to the various coded message fragments of receive is determined device for the receiving terminal unit provides priority, to determine processing method (seeing Figure 47).

Figure 47 is used for explaining the priority that is added to image or sound and explains determining whether that the priority of decoding determines device 5301 schematic diagrames.

That is, as shown in figure 47, these priority comprise the priority and the priority that is added in every two field picture or the voice data of each data flow that is added to each image or sound.Can utilize these priority independently, perhaps make frame priority corresponding to data flow priority.Priority determines that device 5301 is according to definite data flow or the frame that will decode of these priority.

Decoding is to utilize two types priority to carry out, the processing priority when transshipping with definite terminal.

That is, defined: be used to stipulate data flow priority (priority between time series) such as the relative priority level between the bit stream of image and sound; Be used for stipulating (Figure 24) such as the relative priority level between the decoding units such as picture frame of same data stream (priority in the time series).

Preceding a kind of data flow priority can be handled a plurality of videos or voice data.A kind of frame priority in back makes and can change scene according to editor's intention or same in-frame encoding picture frame (I frame) is increased different priority.

Make data flow priority corresponding to the time of distributing to operating system (OS), image or sound encoded or decoded, or carry out processed, thus the control data flow priority, thus, can be on the OS layer control and treatment time.For example, under the situation of the Windows of Microsoft 95/NT, can on five OS layers, define priority.Realize coding or decoding device with software with the clue section, can determine priority according to the data flow priority of destination data stream distributing on the OS layer of each clue section.

Above-mentioned frame priority and data flow priority can be added on transmission medium or the data medium.For example, is the packet priority definition that will send addressed location priority, can be according to the relation between frame priority and the data flow priority, for example according to addressed location priority=data flow priority-frame priority, determine to send relevant priority and the processing priority of terminal when transshipping with packet.

And, can be used as data medium to floppy disk or CD, determine priority.Moreover, not only can utilize recording medium but also can utilize can logging program determine priority such as objects such as IC-card or ROM boxes.And, can use transponder relay data such as images such as router or gateway or sound.

As the concrete grammar that utilizes priority, when receiving terminal transships, the priority of the priority threshold value of determining coded message to be processed is determined that device is arranged to the image spreading control section or sound is expanded control section, elapsed time compares with beginning to handle afterwards the time that will show (PTS), perhaps the time that will decode (DTS) with begin to handle after through time ratio, change the priority threshold value (insertion that also can reference frame at interval or priority etc. as the information that changes threshold value) of coded message to be processed according to comparative result.

In the example shown in Figure 20 (a), when coding, encode to having the QCIF that catches or the image of CIF size with encoder (H.263), the precedence information of the processing sequence when exporting the time tag (PTS) of the time of representing decode time (DTS) or display image and representing overload (CGD, calculating moderate reduction), frame type and sequence number (SN) with coded message.

And, in the example shown in Figure 20 (b), also carry out record, encode, export the time tag (PTS) of representing decode time (DTS) or reproducing time, precedence information (CGD) and the sequence number (SN) of sound with coded message by encoder (G.721) by microphone.

Shown in Figure 20 (c), in when decoding, image and sound offered separates buffer, their DTS (decode time) separately with begin to handle the back elapsed time and compare.If DTS does not lag behind, then this image and sound are offered their corresponding decoders (H.263 and G.721).

The example of Figure 21 has been described a kind of method of utilizing encoder increase priority when overload.For piece image, " 0 " of high priority and " 1 " is distributed to I frame (in-frame encoding picture frame) (numeral is more little, and priority is low more).The priority of P frame " 2 " is lower than the I frame.Owing to give the I frame two kinds of other priority assignment of level, so when the load of decoding terminal is big, can only reproduce the I frame that priority is " 0 ".And, must be according to the insertion interval of priority increase method adjustment I frame.

The example of Figure 22 shows a kind of when overload, determines the method for priority at receiving terminal.The priority of no frame is arranged to than ending the big value of priority.That is, each picture frame is assumed pending object.The priority maximum that is added to picture frame is sent to receiver side (step 101) from transmitter side can learn this value in advance.

When DTS with begin to handle after elapsed time compare, and consequently elapsed time is longer than DTS (when decoding is not in time carried out), the priority threshold value of image then to be processed or sound reduces, to reduce processing (step 102).Yet if elapsed time is shorter than DTS (in time decoding) after beginning to handle, the threshold value of priority increases, to increase the quantity (step 103) of manageable image or sound.

If the image before this frame has been skipped the P frame, then do not handle.If no, then the priority deviant is added on the priority of picture frame (or audio data frame), so that this priority deviant and priority threshold value are compared.When this deviant is no more than this threshold value, the data that decode are offered decoder (step 104).

The priority skew can be used for the performance of precheck machine, and this deviant sent to receiving terminal (user also can send appointment at receiving terminal), also can be used for changing the video in the data flow and the priority (for example, the deviant of background reduces processing by increasing backmost) of audio data stream.

When having proposed multiple data stream, also can add priority, and determine to skip decoding image or sound to each data flow.And, under the situation of real time communication, can determine that by handling the H263 TR similar (interim benchmark) decoding is in advance or lags behind to DTS, and realization and identical skipping mentioned above.

Figure 23 shows the Iterim Change that utilizes the priority that above-mentioned algorithm carries out.

Figure 23 shows the variation that will be added to the priority in the picture frame.This priority is the priority that whether decision decodes when terminal is transshipped, and it is added in each frame.Priority value is more little, and priority is high more.In the example of Figure 23,0 priority is the highest.When the threshold value of priority is 3, added value and do not used greater than the frame of 3 priority, do not decode, to have added value be 3 or decode less than the frame of 3 priority.According to priority these frames are discussed selectively, load that can control terminal.Also can and to be added to relation between the decode time (DTS) on every frame, dynamically determine priority threshold value according to the current processing time.According to same process, this technology not only can be applied to picture frame, also can be applied to sound.

At transmission line is under the situation of the Internet etc., re-send request may priority determining section is set, according to precedence information, retransmit number of times, information dropout rate, intracoded frame insertion at interval, be subjected to the rank (for example Pyatyi is preferential) of the priority of priority determining section control, to receiving the priority threshold value of the definite coded message that will retransmit of control section, and determine to add to the priority threshold value of the coded message that can be subjected to request repeat, like this, in the time must retransmitting the coded message of losing when transmitting, can only retransmit the image or the voice data of receiver side requirement.If the repeating transmission number of times is a lot of or the information dropout rate is very high, then must improve the priority of wanting retransmission of information, reduce and retransmit number of times or reduce Loss Rate.And, understand the priority that the priority determining section is used, can prevent to send information to be processed.

Sending terminal, when actual transfer rate surpasses the target transmission rate of information of transmitting terminal, perhaps when beginning to transmit the time ratio handling the back elapsed time and be added to the coded message that to decode or to show, when coded message is written to the process lag of transmit buffer, the available priority that is added in the coded message as one man sends image or voice data with targeted rate, and receiving terminal when overload, its priority determining section is used this one-tenth level earlier, thus the transmission of the information of minimizing.When receiving terminal transships, also be incorporated into the transmission terminal handling skip functions, thus, can control and send the fault that the terminal overload causes.

By only sending information necessary in the above-mentioned AL information as required, can adjust and to send to such as the amount of information on the narrow band communication passages such as analog of telephone line.Before sending data, determine on sending terminal, will be increased to the data control information in the data, the data control information that will use is sent to receiving terminal as control information (for example only using the random access mark), and, on receiving terminal, rewrite and be stored in the relevant information of transmission structure (seeing Figure 16) in the transmission form storage area 103 according to the control information that obtains.

Fig. 4 is a width of cloth schematic diagram, is used for explaining that dynamic change will be added to the method for the header message of image to be sent or voice data.In the example of Fig. 4, the data that will send (ES) are resolved into data slice and whether serve as the information (marker bit) and the temporal information (time mark) relevant of starting position that can the deal with data sheet, suppose that wherein above-mentioned each information is corresponding to control information transmission of the present invention with the transmission of data slice with identifying information (sequence number), expression that the form of communication stem increases expression data order in these data slice.

Specifically, (real time transport protocol RFC1889) utilizes information, marker bit, time tag, object ID (being called SSRC) and the version number of above-mentioned sequence number as the stem of communicating by letter to RTP.Though can expand the header message item, above-mentioned everyly always add as fixterm.Yet, to transmit a plurality of different coded images or the following time of environment of sound simultaneously when appearing at together such as real time communication such as video telephone and transmission such as accumulation medium such as VODs, because the implication of communication stem differs from one another, so identification device must be arranged.

For example, under MPEG 1/2 situation, as mentioned above, time tag information represents that PTS is the recovery time.Yet, H.261 or H.263, the blanking time when time tag information is represented information encoded.For with the voice data Synchronous Processing H.263, must be labeled as PTS information by express time.This is that in the time interval between the time tag information presentation code frame, the time mark that RTP defines first frame is at random because under situation H.263.

Therefore, the mark that must increase a kind of express time sign and whether be PTS as (a) communication header message (must expand the communication stem) or (b) H.263 or header message H.261 (promptly, AL information) (in this case, must expand Payload information).

Whether as expression is that the marker bit of the information of starting position that can the deal with data sheet adds as the RTP header message.And, as mentioned above, often must provide expression be the access flag of starting position that can visit data and expression can the random access data to obtain the random access mark of AL information.Because the dual mark of communication stem that provides has reduced efficient, substitutes the method for AL mark so also considered a kind of mark of being prepared by the communication stem.

(c) newly provide a kind of mark, the stem that expression AL mark is added to the communication stem substitutes, and do not increase a mark to the AL of communication stem, or the marker bit of definition communication stem and AL's is identical, (compare with the situation that the AL mark is provided, intended interpretation can carry out quickly), thus, address the above problem.That is, adopt a kind of mark, whether the expressive notation position has the implication identical with the AL mark.In this case, think and improved the communication stem or in the scope that expands, described stem.

Yet, (d) also can explain the implication of marker bit of communication stem, to represent that one of random access mark and access flag appear among the AL at least.In this case, can learn that by the version number of communication stem the implication of explanation is different with traditional situation.And, only by provide access flag or random access mark just to simplify processing (for the former, consider that two stems all provide the situation of this mark, but must expand the communication stem again) for communicate by letter stem or AL stem.

The information of adding expression data processing priority has been described, as AL information.By data processing priority is added to the communication stem, processing that can the specified data processing priority, and need not explain the still data content on network.And, under the situation of IPv6, can on the layer lower, add this priority than RTP level.

By increase timer or the counter of the effective period of expression data processing to RTP communication stem, can determine how to change the state of the packet of sending out.For example, when essential decoding software is stored in the memory of low access rate, can determines the needed information of decoder and when need information by counter or timer.In this case, according to purposes, for AL information, the precedence information of timer or counter or the precedence information of data processing are optional.

Fig. 5 (a) and 5 (b) and Fig. 6 (a) are the schematic diagram of explaining the method that increases AL information to 6 (d).

Shown in Fig. 5 (b), send control information to receiving terminal, transmit the stem that whether shown in Fig. 5 (a), AL only is added to the data that will send, whether perhaps pass on AL is increased in each data slice after the data (ES) that will send are resolved into one or more data slice, can select to handle the grade of transmission information like this.When the visit hysteresis was a problem, it was effective that AL is added in the partition data.

As mentioned above, for in advance sending receiving terminal in the recombination of receiver side data control information or the variation that data control information is configured to the method in the data, receiving terminal can utilize the statement of mark, counter or timer to respond smoothly, thereby this statement is prepared to AL information or communication stem, so that it is sent to receiving terminal.

In above-mentioned example, described and avoided with the method for AL information reproduction RTP stem (or communication stem) and the method for expansion RTP or AL information communication stem.Yet, for the present invention, always must not use RTP.For example, also can utilize UDP or TCP to redefine former communication stem or AL information.Though the internet-class environment has utilized RTP sometimes, in former class environment, do not define such as multi-functional stems such as RTP.Considered that following four types notion is used for AL information and the stem of communicating by letter (referring to Fig. 6 (a) to 6 (d)).

(1) header message of RTP or AL information is proofreaied and correct and expand, so that distributed to the header message of RTP and distributed to that the header message of AL is not overlapping (specifically to be, the information overlap of time mark, the precedence information of timer, counter or data processing becomes extend information).Perhaps, can utilize the method that does not expand the RTP stem and do not consider to use RTP information reproduction AL information.They are corresponding to the content that has so far illustrated.Because H.323 reality has been used a part of RTP, be effective so expand RTP with compatibility.(referring to Fig. 6 (a).)

(2) irrelevant with RTP, simplify communication stem (for example, only using sequence number), remaining content is offered AL information, as multi-functional control information.And, make the project that AL information is used can be set before communication changeably, can specify flexible send form like this.(referring to Fig. 6 (b).)

(3) irrelevant with RTP, simplify AL information (extreme example is that AL does not add information), and each control information is offered the communication stem.Sequence number, time mark, marker bit, payload types and the object ID of maintenance through being commonly used for the communication stem is as fix information, and provide the identifier that whether exists with the expression extend information respectively to data processing precedence information and timer information, as with extend information so that when defining this information with reference to (referring to Fig. 6 (c).)

(4) irrelevant with RTP, simplify communication stem and AL information, the packet of formal definition one-tenth, to send this form with communicate by letter stem or AL unpack.For example, also consider a kind of like this method, to AL information only defined label position, time mark and object ID, to communication stem defined nucleotide sequence number only, pay(useful) load information, data processing precedence information and timer information definition are become information packet (second packet) with above-mentioned unpack, and send (referring to Fig. 6 (d).)

As mentioned above, when considering purposes and be added to header message in image or the voice data, according to this purposes, can freely define (customization) will with the packet (second packet) of the stem of communicating by letter, AL unpack transmission, this is preferable mode.

The schematic diagram of Fig. 7 be used for explaining by dynamic multipath multiplexing with separate many logical transport circuits and come method of transmitting information.For providing the information multiplexing section that can begin and finish multiplexed information, the logical transport circuit can reduce the quantity of logical transport circuit, with according to user's appointment or send the quantity of part and the used logical transport circuit of information separated part, send a plurality of data or control information, this separated part is separated multiplexed information for receiving control section.

In Fig. 7, the information multiplexing section is called " group MUX ", and specifically, it can use the multiplex system such as H.223.Can provide group MUX for the transmission terminal.By providing group MUX to relay router or terminal, can be corresponding to the narrow band communication passage.And, using H.223 realization group MUX, can make H.223 and H.324 interconnection.

For the control information (multiplexed control information) that obtains the information multiplexing section apace, by the control information in another logical transport circuit transmission information multiplexing section, and the information multiplexing section can reduce the multiplexed delay that causes not with control information and data multiplexing.Thereby, by passing on and sending, to send behind control information relevant and the data multiplexing with the information multiplexing section, perhaps control information not with data multiplexing, send by another logical transport circuit, it is to keep still reducing because the multiplexed delay that causes with traditional multiplexed consistent that the user can select.In this case, the multiplexed control information relevant with the information multiplexing section is expression and how the information multiplexing section carries out the information of multiplexed relevant multiplexed content to every data.

Equally, as mentioned above, according to expression method, for example according to mark, counter or timer, can transmit the notice of sending method, the information of the combination that this method sends the information of passing on multiplexed beginning and end at least, pass on the multiplexing logic transmission line with one of multiplexed relevant control information (multiplexed control information), as control information, perhaps can data control information be sent to receiving terminal, thereby reduce the time that is provided with of receiver side with data.And, as mentioned above, can provide the project of mark, counter or the timer of expression transmission RTP stem.

When having a plurality of information multiplexing sections or a plurality of information separated part, with identifier communicating control information (multiplexed control information), with authentication information multiplexing section or information separated part, can differentiate which information multiplexing section control information belongs to.Control information (multiplexed control information) comprises multiplexing modes.And, utilize the table of random numbers, thereby determine information multiplexing section or information separated part between the terminal, can produce the identifier of information multiplexing section.For example, can send and receiving terminal between produce random number in the scope determined, and make the identifier (differentiate number) of information multiplexing section adopt its maximum.

Because the multiplexed data of information multiplexing section are different with the multimedia type that defines among the RTP traditionally, so necessary definition information is the multiplexed information of multiplexing section (the definition new media type H.223) of RTP pay(useful) load type to represent it.

Configuration will to improve the access speed to multiplexed data, seek to analyze apace multiplexed information in the information of transmission of information multiplexing section or record in the sequence of control information and data message.And fixedly basis is added to the item that the data control information in the control information is described, and increase and the multiplexed identifier different with data (unique pattern), can rapid analysis go out header message.

The schematic diagram of Fig. 8 is used to explain the transmission course of broadcast program.The information of the relation between the identifier of the identifier of logical transport circuit and broadcast program as broadcast program, thereby send control information or make data increase the identifier of broadcast program, as data control information (AL information), can identify is the data which programming transmits by plurality of transmission lines.And, relation between the identifier (for example LAN port numbers) of receiving terminal transmission data identifier (being SSRC under the situation at RTP) and logical transport circuit, as control information, and confirmed receiving terminal can receiving control information after (Ack/Reject), send corresponding data, even control information and data all respectively by independently transmission line transmission, also can form the corresponding relation between the data slice.

The identifier of the transfer sequence of combination expression broadcast program and have the expression broadcast program or data can be as the counter of valid expiration date of information or the data of timer information.Combination identifiers and information is increased in broadcast program or the data sent, can realize broadcasting, and not need not Return Channel (when effective time limit almost lost efficacy,, also beginning to reproduce the information or the data of broadcast program) even information is not enough.And, can consider a kind of like this method, wherein, utilize single communication port (multicast address), broadcast control information and data and do not make them separated from one another.

In the communication of Return Channel, must before sending data, send control information fully, so that receiving terminal can be learnt data structure.And control information should send by not producing the high transmission channel of data-bag lost, reliability.Yet, when utilizing the low transmission channel of reliability, must circulation send the control information of identical traffic sequence number.This is not limited to send the situation of the control information relevant with the time that is provided with.

And, by selecting to can be used as the item that data message increases (for example access flag, random access mark, data reproduction time (PTS), or data processing precedence information), determine whether that the identifier (SSRC) with data sends data control information by the transmission line Different Logic transmission line with data, perhaps determined whether before sending data, data control information is sent as the data of data control information with transmitter side, and these data are passed on as control information and sent to receiver side, can control and send data neatly.

Thereby, can not increase information and send data message to AL.Therefore, send image or voice data, needn't enlarge definition defined pay(useful) load so far in order to utilize RTP.

Fig. 9 (a) and 9 (b) are for considering reading and the image of rise time or the schematic diagram of sound sending method of program or data.Specifically, when as the unidirectional satellite broadcasting or portable terminal device that do not have Return Channel, when terminal resource is restricted, or program or data appear on the receiver side terminal and are used, and necessary programs (for example H.263, MPEG 1/2, or audio decoder software) or data (for example video data or voice data) be stored in need longer time for reading memory (VD for example, file server on hard disk or the network) time, can be according to expression methods such as identifier such as discriminating program or data, identifier of the data flow of wanting (for example SSRC or logical channel number) or the necessary particular point in time mark of estimation receiving terminal, counter (lifting/lowering counting) or timer, by in advance program being received as control information, or receive the program that reduces early stage and need or the time that is provided with (Figure 18) of data as data control information with data.

When sending program or data, from transmitter side will represent program or data in the information of the memory target (for example hard disk or memory) of receiving terminal, begin or read the type of required time, terminal or storage target with beginning or read relation (for example relation between cpu power, memory device and the average response time) between the required time and the sequence of use sends with program or data, if actual so necessary program of receiving terminal or the data of requiring then can the plan program or the storage target and the time for reading of data.

The schematic diagram of Figure 10 (a) and 10 (b) is used for explaining the method corresponding to switching (switching TV channel) fast.

The situation that only receives image with traditional satellite broadcasting is different, when must be at the receiving terminal executive program, the fetch program and begin to carry out before the time that is provided be a big problem.Also there is same problem in the situation of the portable terminal device that is restricted for available resource.

The time that is provided with of wishing the receiver side terminal can reduce in the following manner: (a) utilize the user to watch and the main audiovisual part listened to and receiving terminal circulation monitoring user are not watched and the auxilliary audiovisual part of the program listened to, and the relativeness between the identifier of the program of reception discriminating requirement in advance or data, the mark of the special time that is used to estimate that receiving terminal is required, the information of counter or timer and program are as control information (sending to the information that control terminal is handled by the packet different with the packet of data) or as data control information (AL information), and preparation reads program or data with data, and the user does not watch or the required programs listened to or a kind of solution of data as occurring in the memory of the longer time for reading of needs.

As second kind of solution, broadcasting channel is set, only broadcasting is by the title image of the image of multifrequency channel radio, and switch program by the user, can prevent that screen from stopping when being provided with, thereby, when program that needing in the memory of the longer time for reading of needs, to occur or data, the interim title image of selecting the program of customer requirements, and it is shown to the user, perhaps show current program or the data of reading, and read the program or data that needs by memory after, restart the program of customer requirements.

Top title image comprises the broadcast image that the program cycle sampling by multichannel broadcasting is obtained.

And timer embodies a kind of time, and expression must need the special time of the required program needs of data flow that decoding sends out defeated side and send.Counter is the basic time unit that determines between transmission and the receiving terminal, and it can be the information of which time of expression.Mark and the data that sent before time necessary is set or control information (information that the different packet of packet by the data handled with control terminal sends) send and pass on.So long be embedded into them in the data and send, perhaps they be sent as control information.

And, in order to determine to be provided with the time, can be when utilizing when being operated in the transmission lines such as ISDN on the base, the transmission sequence number of differentiating transfer sequence is used as control information, to want the specific of program or data to receiving terminal reception and registration from sending terminal, thereby sequence number is sent to receiving terminal with data as data control information or control information, thus, estimate the time that is provided with.And, when the transmission time owing to as the beating or postpone and fluctuate of the Internet, then consider the propagation delay of transmission, must the transmission time be added in the time of setting according to the beating or time delay of the device of realizing RTCP (media transmission protocol that the Internet uses).

Figure 11 (a) shows the object lesson of the actual agreement that transmits between the terminal to 19 (b).

Transmission form and transmission course have been described in ASN.1.And, send the H.245 expansion of form according to ITU.Shown in Figure 11 (a), the object of image and sound can have hierarchy.Under the situation of this example, each Target id has the attribute and the structural information of broadcast program identifier (program ID) and object ID (S SRC), and the synthetic method between the image is used such as scripts such as Java or VRML and described.

Figure 11 (a) shows the example that concerns between the object.

In Figure 11 (a), to liking such as medium such as sound-image, CG and texts.In the example of Figure 11 (a), object constitutes hierarchy.The program number of each object " program ID " is corresponding to the TV channel, and object identifier " object ID " is used to differentiate object.When transmitting each object, making object identifier can easily identify object corresponding to SSRC (synchronous source identifier) according to RTP (media transmission protocol of the transmission medium that the Internet uses, RTP).And, can utilize such as the structure between the descriptive language description objects such as JAVA or VRML.

Consider method with two types connection object.A kind of is broadcast type, and object is unilaterally sent from the transmitter side terminal.Another kind is the type (communication type) of transmission object between transmission and receiving terminal (terminal A and B).

For example, under the situation of the Internet, can be RTP as transmission method.Under the situation of video telephone standard, utilize the transmission channel that is called LCNO, control information transmission.In the example of Figure 11 (a), transmit with a plurality of transmission channels.(program ID) distributes to these passages identical program passage.

The schematic diagram of Figure 11 (b) is used to explain how to realize this agreement that has realized the function that the present invention describes.The used host-host protocol (H.245) of video telephone standard (H.324 and H.323) is described below.H.245, the function that the present invention describes realizes by expansion.

Describing method shown in the example among Figure 11 (b) is the protocol description method that is called ASN.1." the performance of terminal capabilities collection (" Terminal Capabilitycet ") statement terminal.In the example of Figure 11 (b), the function that is described as " mpeg4 performance " is to tradition expansion H.245.

In Figure 12, " mpeg4 performance " (" MPEG4 Capability ") described and can also describe the maximum multiplexed function number (" Max Number ofMux ") that terminal can realize simultaneously by maximum image data of terminal processes (" Max Namber of Video ") and maximum acoustic number of signals (" Max Numbeof Sounds ").

In Figure 12, be expressed as manageable largest object amount (" Number of ProcessObject ").And, the mark of representing whether can change communication stem (being expressed as AL in Figure 12) has been described.When the value of mark is a true time, can change the communication stem.For utilize " MPEG 4 confirming performances (" MEG 4Capbility ACK ") pass on manageable number of objects between terminal each other; can accept (processing) these objects if be communicated side; be communicated side " MPEG 4 Capbil ity ACK " are sent back to the terminal of transmission " MPEG 4Capbility "; if can not, then to this terminal loopback " MEPG 4 performances refusal " (" MPEG 4Capbility Reject ").

Figure 13 (a) shows how to describe and utilizes above-mentioned group of MUX that a plurality of logics are believed that passages are multiplexed into the agreement of a transmission channel (being the LAN transmission channel in this example), so that logical channel common transmitted passage.In the example of Figure 13 (a), make multiplex machine (group MUX) corresponding to LAN (local area network (LAN)) transmission channel (LAN port numbers, " LAN Poxt Number ")." group Mux ID " (" Group Max ID ") is the identifier that is used to differentiate multiplex machine.For " establishment group Max " (" Create group Mux "), make the shared multiplex machine of terminal, and between terminal, transmit mutually, can accept (use) this multiplex machine if be transmitted side, just to the terminal loopback " Greate Group Mux Ack " that sends " Greate Group Mux ", if can not accept, just to this terminal loopback " Greate Group Mux Reject ".Can realize separator with identical method, as carrying out the device opposite with the operation of multiplex machine.

At Figure 13 (b), the situation (" Destory GroupMux ": " destruction group Mux ") of having deleted the multiplex machine that has produced has been described.

In Figure 13 (c), the relation between LAN transmission channel and a plurality of logical channel has been described.

According to the transmission channel of " LAN Port Numbef " description LAN, (" Logical Port Number ") described logical channel according to " logical port number ".

Under the situation of Figure 13 (c), can make corresponding nearly 15 logical channels of transmission channel of a LAN.

In Figure 13, when the quantity of operable MUX only was one, group Mux ID was just optional.And, in order to use a plurality of Mux, group Mux ID all must be arranged for H.223 each order.Can usage flag, concern to pass between the port that uses between multiplex machine and the separator.Can also utility command make that can select is control information to be carried out multiplexed, still passes through another logical transport line and transmits information.

Figure 13 (a) is to the explanation example shown in Figure 13 (c), and transmission channel is utilized LAN.Yet, also can use utilization for example H.223 or the system of the Internet protocol of MPEG 2 etc.

In Figure 14, the protocol description of " opening logical channel " (" Open Logical Channel ") display definition transmission channel attribute.In the example of Figure 14, " MPEG 4 logical channel parameters " (" MPEG4 LogicalParameters ") is according to H.245 protocol extension and definition.

Figure 15 shows program number (corresponding to the TV channel), makes program number be made into transmission channel (" MPEG 4 logical channel parameters ") corresponding to LAN.

And " broadcast channel program " (" Broadcesting Channel Program ") expression sends the describing method of corresponding relation between LAN transmission channel and the program number according to broadcast type in Figure 15.The example of Figure 15 makes to send and reaches corresponding relation between 1023 transmission channels and the program number.Because under the situation of broadcasting, transmitter side unilaterally sends to receiver side,, must circulation send these message slots so consider the phenomenon of losing during the transmission.

In Figure 16 (a), having described will be as the attribute (" definition of MPEG4 object type ") (" MPEG4 Object Calssdefinition ") of the object (for example image or sound) of program transmission.Object information (" object structure unit ") (" Object Structure Element ") corresponding to program identifier (" ProgramID ").Can make and reach 1023 objects corresponding to program identifier.For object information, described LAN transmission channel (" LAN port numbers "), expression and whether used the mark (" scrambled mark ") (Scramble Flag) of scrambled, define the field (" CGD skew ") (" CGD Offset ") of the deviant that changes processing priority and the identifier (" medium type ") (" Media type ") of differentiating the medium type (image or sound) that will transmit when terminal is transshipped.

In the example shown in Figure 16 (b), increased AL (in this example, being defined as the required additional information of a frame decoding image) with the decoding of control to ES (in this example, be defined as corresponding to the image of a frame serial data).For AL information, be defined as follows.

(1) random access mark (mark whether expression can independently reproduce, the picture frame of true respresentation intraframe coding)

(2) represent time tag (time that frame shows)

(3) CGD priority (determining the priority value of the processing priority when terminal is transshipped)

This example shows the situation of utilizing RTP (by the agreement of the continuous medium of the Internet transmission, i.e. real time transport protocol) to transmit the serial data of a frame." AL reconstruct " is to change the peaked transmission statement that can represent with above-mentioned AL.

The example of Figure 16 (b) makes and can be shown as " random access mark maximum number of digits " (" Random Access Flag Max Flag ") reaching 2 bit tables.For example, when not having, do not use the random access mark.In the time of two, maximum equals 3.

And, can represent (for example 3 ∧ 6) with real part and mantissa part.When data not being set, can under default definite state, work.

In Figure 17, " request is set " (" Setup Request ") expression transmission is provided with the transmission statement of time." request is set " and before the transmission program, transmits, make the transmission channel number (" logical channel number (LCN) ") that to transmit, the program ID that will carry out (" executive program number ") (" execative Program the Number ") data ID (" data Number ") that will use and order ID that will carry out (" fill order number ") (" excutiveCommand Number ") become to correspond to each other, and be transferred to receiving terminal.And, as other expression method, can adopt and carry out authorization mark (" flag "), be described in the timer value (" timer ") that begins to carry out after receiving the counter (" counter ") that begins to carry out when the request how many times is set and being illustrated in, and make them corresponding to transmission channel number through how long.

As the example of the request that will propose, can enumerate the rise time that rewrites AL information and guarantee to organize Mux.

The schematic diagram of Figure 18 is used for explaining the transmission statement, and the AL that whether utilizes Figure 16 (b) to describe transmits AL (" control AL definition ") (" Control AL Definstion ") from sending terminal to receiving terminal.

In Figure 18,, then use the random access mark if " use of random access mark " (" Random Access Flag Use ") is true.If be not true, just do not use.Can transmit as control information the AL change notification by the passage that separates with data transmission channel, or, transmit with data by the transmission channel identical with data.

As the program that will carry out, can list decoding program.Can be used for broadcasting and communicate by letter request is set.According to above-mentioned request, which of control information receiving terminal specified as AL information.And, can specify which as the communication stem, which as AL information, which as control information to receiving terminal.

Figure 19 (a) shows a kind of transmission statement, it utilizes the information frame identifier (" stem ID ") (" header ID ") between transmission and the receiving terminal, changes the structure (data control information, control information transmission and control information) of header message according to purposes.

In Figure 19 (a), " classification ES stem " (" Calss ES_header ") be according to the information frame identifier, the structure of the data control information that differentiation will be by the channel transfer identical with data transmission channel and control information transmission in order to send and receiving terminal between the structure channel of information transmitted identical.

For example, as " header ID " when value is 0, only use " ES of buffer scale " (" bufferSize ES "), and work as " header ID " when value is 1, increase (" reserved ") item.

And, utilize default designation symbol (" use header extension ") (" use Header Extension "), determine whether to use the default type information frame.When " use header extension " is true time, use the item in the if statement.Suppose between transmission and receiving terminal, to have determined these structural informations in advance.And, can use a kind of structure, to use one of information frame identifier and default designation symbol.

In the example of Figure 19 (b), " AL configuration " (" AL Configuration ") represents such example, it according to purposes by with transmission and receiving terminal between the different transmission channel of the data transmitted change the control information structure that will transmit.The effect of information frame identifier is identical with the situation of Figure 19 (a) with the effect of default designation symbol.

Under situation of the present invention, specifically describe realization from following viewpoint and synthesize and show a plurality of images and a plurality of sound method simultaneously.

(1) by many logical transport circuits transmission (passing on and broadcasting) image and sound and control their method.Especially, described by the method for control information transmission and data respectively of logical transport circuit independently.

(2) dynamically change the method that is added to the header message (AL information) in the data of wanting image transmitted or sound.

(3) dynamically change the communication header message of increase so that the method for transmission.

Specifically, for (2) and (3), described a kind of control and overlapped the method for the information on the AL information and the stem of communicating by letter and the method that AL information is transmitted as control information.

(4) dynamically multiplexed with separate many logical transport circuits and method of transmitting information.

A kind of method and effective multiplexed method of realization of saving the port number of transmission line described.

(5) consider and read program or data rise time, and images and sound method.And, described the apparent that reduces various functions and user time method has been set.

(6) image or the sound transmission method that switches fast at channel.

The present invention is not limited in synthetic two dimensional image.Also can use the expression method of two dimensional image and 3-D view combination, can comprise that perhaps synthetic multiple image is so that their image combining methods adjacent to each other as wide field-of-view image (panoramic picture).

And the present invention is an object with the communication system such as two-way catv and B-ISDN not only.For example, radio wave (for example VHF frequency range or uhf band) be can use, perhaps the analog of telephone line of information or N-ISND (always must multiplexed image, sound or data) be transmitted to the broadcasting satellite of home terminal images and sound and from home terminal to central terminal from central terminal.

And, can use radio communications system, such as IrDA, PHS (personal hand-held phone), perhaps radio LAN.And the object terminal can be the portable terminal of for example portable data assistance or desktop terminals such as box or personal computer for example are set.And, as range of application, also have video telephone, multiple spot surveillance, multimedia system, multimedia database retrieval system and game machine.The present invention not only comprises receiving terminal, but also comprises server and the transponder that is connected to receiving terminal.

In above-mentioned example, described the method for avoiding RTP (communication) stem and AL information overlap and expanded the communicate by letter method of stem or AL information of RTP.Yet the present invention always must not use RTP.For example, also can utilize UDP or TCP to redefine former communication stem or AL information.Though the internet-class environment uses RTP sometimes, do not define such as multi-functional stems such as RTP for former class environment.As mentioned above, there are four kinds about the notion of AL information with the stem of communicating by letter.

Therefore, the control information that information frame, control information transmission or transmission and receiving terminal by dynamically moving specified data control information uses (for example comprise the information sequence that will increase and at first the random access mark is distributed into 1 label information and thereafter the random access mark is distributed into the information frame of 16 figure place with the form of sequence number) can only change information frame corresponding to situation according to purposes or transmission line.

The frame of each information can be to have shown any as Fig. 6 (a) to 6 (d), under the situation of RTP, data control information (AL) can be every kind of medium header message (for example, under situation H.263, can be the header message of video header message or H.263 intrinsic pay(useful) load), control information transmission can be the RTP header message, and control information can be the information of control such as RTP such as RTCP.

And, between transmission and receiving terminal, set in advance under the situation of known information frame, by the control information that transmits and receive data respectively, control information transmission and control information (by the information of the different data packet transmission of the packet of the data handled with control terminal), default identifier is provided, whether represent process information, can understand information frame and whether change.Only when changing, according to method shown in Figure 16, the content (for example time mark information changes over 16 from 32) that default designation symbol and transmission have changed is set, this can prevent to transmit when the frame information of information from not changing unnecessary configuration information.

For example, two kinds of methods below considering, the information frame of change data control information.At first, when being described in data itself and changing the method for information frame of data control information, default designation symbol (will be written to fixed area or position), the changing content of descriptor frame then of the information that occurs in the data that the information frame of data control information describes are set.

In order to change the information frame of data control information, when describing another kind the method for the information frame that only changes the data in the control information (information frame control information), the default designation that control information provides is set, the content of the information frame of the data control information that description will change, and send receiving terminal to according to ACK/ refusal, after confirming the information frame change of data control information, the data that the transmission information frame has changed.Also can change the information frame (Figure 19) of control information transmission and control information according to above-mentioned two kinds of methods.

More particularly, though the header message of MPEG2 is fixed, by default designation symbol (by the PSI definition) is provided for Program Map Table, the video data stream (transmission data flow) of MPEG2-Ts is associated with its audio data stream, and the definition configuration data stream, description changes the method for the frame of video data stream and voice data stream information, can at first explain configuration data stream, according to the content that default designation when symbol configuration data stream is set, explain the stem of video and audio data stream then.Can make configuration data stream have content shown in Figure 19.

The of the present invention content (transmission format information) relevant with the structure of transmission method and/or the data that will transmit is corresponding to the information frame in the foregoing description for example.

For the foregoing description, the relevant content that will change of structure with transmission method and/or the data that will transmit has been described mainly.Yet, also can use the structure of only transmitting content designator.In this case, as shown in figure 44, also can use the audio-video transmitter, this transmitter setting: (1) dispensing device 5001, be used for by transmission line identical or the transmission line different with last transmission line with data, the relevant interior identifier of perhaps representing content of structure of transmission and transmission method and/or the data that will send is as sending format content; (2) storage device 5002, be used to store the polytype content relevant with the content of the transmission method and/or the data that will send and polytype identifier of these contents, wherein identifier is included in one of information that data control information, control information transmission and control terminal side handle at least.And, as shown in figure 45, can use a kind of audio-video receiver, this receiver is provided for receiving audio-video transmitter receiving system 5101 that sends and the transmission format information interpreting means 5102 of explaining the transmission format information that receives.And, the audio-video receiver can be made has storage device 5103 formations, be used to store the polytype content relevant and polytype identifier of these contents with the structure of the transmission method and/or the data that will send, when receiving identifier, with the content that is stored in the content interpret identifier in the storage device as the transmission format information.

More particularly, be ready for sending and receiving terminal between predetermined various types of information frame, and transmit identifier and polytype data control information of above-mentioned information frame with data, the information frame identifier of polytype control information transmission and polytype control information (transmission frame control information), as control information, like this, can differentiate polytype data control information according to the capacity of medium type that will transmit or transmission line, polytype control information transmission and polytype control information choose at random the information frame of every type of information.Identifier of the present invention is corresponding to above-mentioned information frame identifier.

Even information frame has changed in the receiver side terminal, also can read and explain these information identifiers and default designation symbol by identifier being added in the predetermined regular length zone of wanting information transmitted or on the precalculated position.

And, except the structure that the foregoing description is described, also can use a kind of like this structure, when program of spending a lot of times to be provided with to need or data, utilize broadcast channel, only broadcast title image, thereby select the image of catching of program that the user will watch and listen to temporarily, and switch the program that the user will watch and listen to by the image of multichannel broadcasting.

As mentioned above, the present invention can be according to purposes or transmission line, dynamically determine to send and frame, control information transmission or the control information of the data control information of receiving terminal use, thereby corresponding to the frame of state change message.

And, can be by the default designation symbol be provided, whether expression utilizes the known information of the data control information, control information transmission and the control information that set in advance between transmission and the receiving terminal to send or receive and process information respectively, thereby can learn whether information frame changes, even and the information frame of information only just transmits the content of change owing to be provided with default designation symbol and not have to change and also can prevent to transmit unnecessary configuration information when having carried out change.

Moreover, by preparing predetermined multiple information frame between transmission and receiving terminal, and transmit information frame identifier, polytype control information transmission and the polytype control information of differentiating polytype data control information with data, can differentiate polytype data control information, polytype control information transmission and polytype control information, and, can optionally select various types of information frames according to the type of the medium that will transmit or the capacity of transmission line.

Even information frame has changed on the receiver side terminal,, also can read and explain these information identifiers and default designation symbol by identifier being added in the predetermined regular length zone of wanting information transmitted or preposition.

In this example, explained a problem in the problems referred to above (B1) to (B3).

Used " image " of the present invention comprises rest image or live image.And the image of proposition can be a two dimensional image, and for example computer graphics (CG) image perhaps can be the 3 d image data that constitutes with the wire frame type.

Figure 25 is the schematic block diagram of the image encoder or the image decoder of embodiments of the invention.

Sending or write down various coded messages transmission control sections 4011 is devices of the information of transmission shaft cable, CATV, LAN or modulator-demodulator etc.Image encoder 4101 has image encoding part 4012, be used for to such as H.263, MPEG1/2, jpeg image information is encoded or carry out Huffman coding, also has the control section 4011 of transmission.And the output 4016 of image encoder 4102 is made of following part: receive the image synthesizing section 4015 of the reception control section 4013 of various coded messages, the picture decoding part 4014 of the various image informations that receive of decoding, synthetic one or more decoded picture and the output that constitutes by the display and the printer of output image 4016,

Figure 26 is the schematic block diagram of the audio coder and the audio decoder of one embodiment of the invention.

G.721 or the audio-frequency information of the MPEG1 audio signal audio coding part 4022 of encoding audio coder (vocoder 4021) is made of following part: send or write down the transmission control section 4021 of various coded messages and to such as.Audio decoder (voice decoder) 4202 is made of following part: receive the reception control section 4023 of various coded messages, the audio decoder part 4024 of the above-mentioned audio-frequency information of decoding, audio frequency composite part (the speech synthesiser branch) 4025 of synthetic one or more decoded audio signals and the output device 4026 of output audio signal.

Above-mentioned coding or decoder carry out concrete coding or decoding to the time series data of sound or image.

Figure 25 and 26 communication environment can be can use many logical transport circuits and the environment of not considering multiplex machine, as the situation of the Internet, or the essential environment of considering multiplex machine, as the situation of analog telephone or satellite broadcasting.As the terminal connected system, can enumerate system such as both sides' images or sound between terminals such as video telephone or video conference, or on satellite broadcasting, CATV or the Internet the system of broadcast types of image or sound.

Can use such as scripts such as JAVA, VRML or MHEG and speak, describe image and sound, image and sound structural information (display position and demonstration time), audio-video group technology, image display layer (degree of depth) and object ID (ID of discriminating such as various objects such as image or sound) and and their attribute between relation, can define a kind of composograph or sound method like this.The script of describing synthetic method obtains from network or local storage.

And image encoder, image decoder, audio coder and audio decoder that can the combination in any any amount constitute and send or receiving terminal.

The schematic diagram of Figure 27 (a) is used to explain the priority interpolation part and the priority determining section of control overload processing priority.For being provided with priority, image encoder 4101 and audio coder 4201 add part 31, according to predetermined standard, H.263 or coding method G.723 utilization is handled the priority of coded message such as when determining overload, and coded message is associated with definite priority.

The benchmark that adds priority be under the situation of image for scene changes, under the situation of picture frame, data flow or the sound of editor or user's appointment, be the ensonified zone and do not have the sound area.

To the communication stem add the method for priority and priority is embedded into during at coding the video that will encode or the stem of the bit stream of audio frequency in the priority adding method of method priority can be considered as the definition overload time.Preceding a kind of method can obtain the information relevant with priority without decoded information, and a kind of method in back can be handled a bit stream independently, and and system independence.

Shown in Figure 27 (b), when precedence information being added to the communication stem, and a picture frame (for example P of the I frame of intraframe coding or interframe encode or B frame) when being divided into a plurality of transmits data packets, under the situation of image, only priority is added in the communication stem, with the stem of addressable picture frame as single message transmission (, can suppose) until when addressable picture frame occurring, priority all not have variation next time when when same image medium priority is identical.

Again, under the situation of decoder, according to the priority of the various coded messages that the time receive in overload, for image decoder 4102 and audio decoder 4202 are provided with the processed part 32 of determining processing method.

The schematic diagram of Figure 28 (a) to 28 (c) is used to explain the grade of adding priority.Decoding is to utilize two types priority to carry out, the priority of handling when transshipping with definite terminal.

That is, defined: data flow priority (the data flow priority of the priority of handling when being used to be defined in bit stream (such as the bit stream of image and sound) overload; Priority between time series data); Frame priority (the frame priority of the priority of handling when being used for stipulating same data flow frame (for example picture frame) overload; Priority in the time series data) (referring to Figure 28 (a)).

Preceding a kind of data flow priority can be handled a plurality of videos or voice data.A kind of frame priority in back can be according to editor's purpose, changes or the picture frame (I frame) of same intraframe coding increases different priority to the image scene.

The value representation that data flow priority is represented is processed into it relative value and it is processed into the situation (referring to Figure 28 (b) and 28 (c)) of absolute value.

On network, by transmitting terminal processes data flow priority and frame priority such as router or gateway etc.; In terminal, then handle by transmission and receiving terminal.

Consider with two kinds of method representation absolute values or relative value.A kind of is the method shown in Figure 28 (b), and another kind is the method shown in Figure 28 (c).

In Figure 28 (b), when being the expression editor increases or machine increases image data stream (video flowing) or audio stream overload, absolute value priority handles the value (but not being the value of having considered the load variations of real network or terminal) of the order of (maybe will handle).Relative value priority is the value according to the changing load absolute priority value of terminal or network.

According to load fluctuation of network etc. priority composition relative value and absolute value are carried out controlling value, thereby only change relative value at transmitter side or by transponder, can keep the absolute priority that is added in video or the audio data stream like this, simultaneously with its value record to hard disk or on the VTR.Therefore, when the value of record absolute priority, can reproduce image or sound that the load fluctuation that is not subjected to network etc. influences.And, can be independent of data and transmit relatively or absolute priority by control channel.

In Figure 28 (b), can carry out meticulous grade relatively with data flow priority, processed frame priority, the frame processing priority during overload is defined as relative priority level value, perhaps it is processed into the absolute priority value.For example, absolute frame priority and the relative frame priority of describing corresponding to the absolute priority in the picture frame in the stem of the communication data packet that is added to transfer encoding information in the coded image are described, the load fluctuation of reflection network or terminal, like this, even under the situation of frame one-level, also can when keeping former priority, add priority corresponding to the load of network or terminal.

Can also be independent of data, in control channel, rather than the communication stem is described and the relation of frame, transmits the relative priority level.Like this, can keep the absolute priority that was added to image or audio data stream originally, simultaneously data record on hard disk or VTR.

And, in Figure 28 (b), when reproducing data at receiving terminal, and simultaneously by transmitted data on network and not when the receiving terminal identifying recording layer, owing to needn't separate absolute value and relative value is controlled at receiving terminal, can after the value and relative priority level of transmitter side, only transmit absolute value to frame and data flow two-stage calculating absolute priority.

In Figure 28 (c), absolute value priority is from well-determined value between the frame of the acquisition of the relation between data flow priority and the frame priority.Image data stream that relative value priority is that the expression editor adds or machine adds or audio data stream are handled the value of the order of (maybe will handle) when overload.In the example of Figure 28 (c), the frame priority that increases image or audio data stream is (relative; Relative value) or the data flow priority of each data flow.

Obtain absolute frame priority (definitely from relative frame priority and data flow priority sum; Absolute value) (that is absolute frame priority=relative frame priority+data flow priority).In order to obtain absolute priority, also can use subtractive method or the constant method that multiplies each other.

Absolute frame priority is mainly used in network.This is because utilize the statement of absolute value not need to determine to determine the priority of every frame by considering data flow priority and frame priority such as transponders such as router or gateways.Utilize absolute frame priority, the processing of discarded frame of transponder and so on is simplified.

And, can wish relative frame priority is mainly used in accumulation system, to write down or to edit.Under the situation of edit operation, can handle a plurality of images and audio data stream simultaneously.In this case, can limit the quantity of energy reproduced image data flow and the quantity of frame according to the load condition of terminal or network.

In these cases, with only different by data flow priority is separated the situation of explaining absolute value with frame priority, needn't recomputate each frame priority, promptly only the editor thinks preferentially to show or the data flow priority of the data flow that the user will see gets final product by changing.Therefore, must be according to absolute statement of purposes utilization or statement relatively.

By description is that data flow priority is used as relative value or as absolute value, can represents the priority of transmitting and accumulating effectively

Under the situation of Figure 28 (b), its difference is that the value of utilizing expression data flow priority to explain is absolute value or the mark of relative value or identifier, and the value of streams of trace data priority statement is the data flow priority of relative value or absolute value.Under the situation of frame priority, do not need mark or identifier, this is because described relative value in the communication stem, has described absolute value in coded frame.

In the example of Figure 28 (c), used and differentiated that frame priority is absolute value or the mark of relative value or identifier.Under the situation of absolute value, frame priority is the priority that calculates according to data flow priority and relative frame priority, and therefore, transponder or terminal are not carried out this calculating.And, when terminal known calculations formula, can be according to the relative frame priority of absolute frame priority with data flow priority backwards calculation.For example, can also obtain to want the absolute priority (addressed location priority) of data packets for transmission from following relative statement:

" addressed location priority=data flow priority-frame priority "

In this case, because its acquisition after from data flow priority, having deducted, so also can represent also to reduce priority to frame priority.

One or more data flow priority are associated with the priority of data processing by TCP/IP logical channel (port numbers of LAN), also can handle by control data.

Moreover, give image or sound by handle than character or low data flow priority or the frame priority assignment of control information, expection can reduce repeating transmission.Even this is because parts of images or sound are lost, in most of the cases can not have problems yet.

The schematic diagram of Figure 29 is used to explain the method for priority assignment being given the multiresolution video data.

When a data flow is made of a plurality of sub data flows, can define a kind of sub data flow processing method, data flow priority is added in the sub data flow, and logic and or logic product when describing accumulation or transmission.

Under the situation of wavelet, a picture frame can be resolved into the picture frame of a plurality of different resolutions.And, even in the method for DCT based encode, by picture frame being divided into high fdrequency component and low frequency component, and they are encoded, also can resolve into a picture frame picture frame of a plurality of different resolutions.

The data flow priority in a plurality of image data streams that the picture frame that is added to by a series of decomposition constitutes, the relation between the image data stream with " with " (logic product) and " or " (logic and) define, to describe this relation.Specifically, when the data flow priority of data flow A is 5, the data flow priority of data stream B is 10 o'clock (numerical value is more little, and priority is high more), and the relation between the image data stream is defined by discarding under the situation of data flow data according to priority, discarded data stream B, but by the relation between the data of description stream, " with " situation under, even the priority of data stream B is lower than threshold value priority, also can transmit and data streams B, and not discarded.

Thereby, can handle related data flow and discarded." or " situation under, be defined as and can discard related data flow.Can carry out waste treatment at transmission or receiving terminal even transmitting terminal.

And, as the operator of describing relation, when same video clipping is encoded into 24Kbps and 48Kbps respectively, there is the situation (describing relation) of reproduction 24 or 48Kbps with different logic and distance.

When the priority that is configured to 10 latter when the former priority was configured to 5, the user can reproduce the latter according to priority, does not perhaps follow priority, selects the latter.

The schematic diagram of Figure 30 is used to explain the constructive method of communication pay(useful) load.

When constituting by a plurality of sub data flows, for example,, constitute transmits data packets successively from having the sub data flow of limit priority according to the data flow priority that is added in the sub data flow, then on the transmits data packets one-level, discard and become easy.And, subtly the information of the object that has high frame priority is respectively classified and component units, can make and discardedly on the communication data packet one-level become easy thereby constitute communication data packet.

Be associated with communication data packet by the piece cutting structure that makes image, the packet that loopback is omitted becomes easy.That is, be associated with packet structure, need be used for again synchronous sync mark again by the piece cutting structure that makes image.Except the piece cutting structure consistent, must increase sync mark again (informing the mark of loopback location), so that because the missing data bag when making information impaired, also can carry out synchronous again with the communication data pack arrangement.

State according to mentioned above, can think high mistake proofing measure is applied to have the communication data packet of high priority.And the piece cutting structure of image is represented the integrated images information unit such as GOB or MB etc.

The schematic diagram of Figure 31 is used to explain the method that data are associated with the communication pay(useful) load.By data flow or object are transmitted with control information or data with the method that communication data packet is associated, can produce data format arbitrarily according to communications status or purposes.For example under the situation of RTP (real time transport protocol), to the pay(useful) load of each coding definition RTP to be processed.Existing RTP form is fixing.Under situation H.263, as shown in figure 31, the three kind data formats of Mode A have been defined to pattern C.Under situation H.263, not having definition is the communication pay(useful) load of object with the multi-resolution image form.

In the example of Figure 31, level number and above-mentioned relation are described (" with ", " or ") be added in the data format of Mode A, and defined.

The schematic diagram of Figure 32 is used for the relation between interpreted frame priority, data flow priority and the communication data packet priority.

And Figure 32 shows the priority that is added on transmission line in the communication data packet is used as the communication data packet priority, and makes data flow priority and frame priority and the corresponding example of communication data packet priority.

Usually, under the situation of utilizing IP to communicate, must make the frame priority or the data flow priority and the corresponding data of transmitting of low side IP priority of data packets that are added in image or the voice data.Because being divided into the IP packet, image or voice data transmit, so these priority are corresponded to each other.In the example of Figure 32, because the value of data flow priority is 0 to 3, the value of frame priority from 0 to 5 is so the priority of high end data can from 0 to 15.

Under the situation of Ipv6, kept 0 to 7 priority (4), be used for blocking the control business.Keep priority 8 to 15 and be used for real time communication business or unblock control business.Priority 15 is limit priorities, and priority 8 is lowest priorities.This is illustrated in the priority of IP packet one-level.

Under the situation of the transfer of data of utilizing IP, must make 0 to 15 high-end priority corresponding with 8 to 15 low side IP precedence.For these priority are corresponded to each other, can eliminate the method for some high-end priority, perhaps can utilize the performance evaluation function that these priority are corresponded to each other.Set up that the corresponding relation of high end data and low side IP precedence carries out on via node (router or gateway) or transmission and receiving terminal.

The transmission means are not limited in IP.Can have TS (transmission data flow), use to have the whether packet of discardable mark of expression as ATM or MPEG2.

So far frame priority and the data flow priority described can be applied to transmission medium or data medium.Can use floppy disk or CD as data medium.

Not only can use floppy disk or CD, also can use, as long as can recorded program on this medium such as medium such as IC-card or ROM boxes.Moreover, also can use audio-video transponder (for example, router or gateway) to come relay data.

And, according to the information of data flow priority (priority between time series data) or frame priority (priority in the time series data), the definite time series data that will retransmit, thus realize preferential the repeating transmission.For example, when when receiving terminal is decoded according to precedence information, can prevent that the data flow of non-process object or frame are retransmitted.

Also can separate, determine to have the data flow or the frame of answering important priority according to the relation of retransmitting number of times and successfully send between the number of times with the present preferential utmost point to be processed.

In addition, under the situation of transmitter side terminal,, thereby realize preferential the transmission according to the definite time series data that will send of information of data flow priority (priority between time series data) or frame priority (priority in the time series data).For example, according to average transfer rate or retransmit number of times, determine the frame that will send or the priority of data flow, even also can send adapting to image or sound during network over loading.

It is synthetic that the foregoing description is not limited to two dimensional image.That also can use makes up two dimensional image and 3-D view and the expression method of acquisition, perhaps comprise the image combining method of synthetic multiple image, thereby as wide view field image (panoramic picture), each image is adjacent to each other.And the communication system that the present invention proposes is not limited to two-way catv or B-ISDN.For example, therefrom entreat lateral terminal to be transferred to home terminal and can use radio wave (for example VHF or UHF wave band) image and sound, perhaps use satellite broadcasting, the information that comes from family's lateral terminal is sent to the center side terminal can use analog phone line or N-ISDN (always not needing multiplexed image, sound or data).Can also use and utilize wireless communication system, for example IrDA, PHS (personal hand-held phone) or radio LAN.

And the purpose terminal can be portable terminal device (for example portable information terminal) or desktop terminal (for example top setting box or personal computer).

As mentioned above, the present invention can easily handle a plurality of video data streams and a plurality of audio data stream, and emphasis ground and the main scene fragment of sound reproduced in synchronization are arranged, with reflection editor's intention.

Following embodiment has solved the problems referred to above (C1) to one of (C3).

Figure 33 shows the structure of the transmitter of first embodiment.Symbol 2101 presentation video inputs, the size of an image are 144 pixels * 176 pixels.Symbol 2102 expression video encoders constitute (referring to recommending H.261) by four

parts

1021,1022,1023 and 1024.

Symbol 1021 expression converting units, input picture is divided into macro block (square region of 16 pixels * 16 pixels), and determine these pieces are carried out intraframe coding or carry out interframe encode, 1022 expression motion compensating devices, according to can be according to this machine decoded picture that last time, coding result calculated, produce the motion compensation image, calculate the difference between motion compensation image and the input picture, and with macro block form output result.Motion compensation comprises half-pix prediction with long process time and the both full-pixel prediction with weakness reason time.Symbol 1023 expression orthogonal converters carry out dct transform to each macro block, and 1024 expression variable-length code devices carry out scrambled to dct transform result and other coded message.

Symbol 2103 expression counting devices are counted the execution number of times of four parts of video encoder 2102, and are exported the count results of each input picture to converting means.In this case, count from the execution number of times of 1022 pairs half pixel predictions of motion compensating device and the execution number of times of both full-pixel prediction.

Symbol 2104 expression converting means, output serial data as shown in figure 34.Symbol 2105 expression dispensing devices are multiplexed into a serial data to the serial data of the variable-length code (VLC) of video encoder 2102 and converting means 2104, and to data output end 2109 these data of output.

According to said structure, can send the necessary execution number of times of handling (converting unit 1021, orthogonal converter 1023 and variable-length code device 1024) and inessential processing (motion compensating device 1022) to receiver.

The transmitter of first embodiment is corresponding to claim 68.

Figure 40 is the flow chart of the sending method of second embodiment.

Because the operation of present embodiment is similar to first embodiment, thus note units corresponding.At step 801 (image input 2101) input picture, and segment the image into macro block in step 802.After this, repeating step 803 is to the processing of step 806, up to the processing of having finished according to the conditional branching in the step 807 corresponding to each macro block.And, handle when having carried out each, when enabling the number of times with the concrete processing of variable record, relevant variable is increased 1 from step 803 to step 806.

At first, determine macro block to be processed is carried out coding or inter codes in the piece in step 803 (converting unit 1021).When macro block is carried out inter codes, carry out motion compensation in step 804 (motion compensating device 1022).After this, carry out dct transform and variable length code in step 805 and step 806 (orthogonal converter 1023 and variable-length encoder 1024).When the processing finished each macro block (in step 807 under the situation of "Yes"), read the variable of expression in step 808 corresponding to each execution number of times of handling, produce serial data as shown in Figure 2, and multiplexed and this serial data of output and code.As long as continuing has input picture, with regard to the processing of repeated execution of steps 801 to step 808.Go up brown structure and can transmit the execution number of times of each processing.

The sending method of second embodiment is corresponding to claim 67.

Figure 35 shows the receiver structure of the 3rd embodiment.

In Figure 35, symbol 307 expression inputs are imported the output of the transmitter of first embodiment, and 302 expression receiving systems are multiplexed by going according to the output of the transmitter of first embodiment, obtain variable-length code (VLC) and serial data, and output.In this case, suppose to measure and exported the reception needed time of one page data.

Symbol 303 expressions are the Video Decoder of variable-length code (VLC) as input, and it is made of five parts.Symbol 3031 expression variable length decoding apparatus, obtain DCT coefficient and other coded message from variable-length code (VLC), 3032 expression anti-quadrature converting means, the DCT coefficient is carried out anti-dct transform, 3033 expression conversion equipments are that the intraframe coding or the coded message of interframe encode are changed the output of each macro block up or down according to the expression macro block.Symbol 3034 expression motion compensating devices utilize decoded picture and mobile coding information last time, produce the motion compensation image, and with the output addition of anti-quadrature converting means 3032 after export.Symbol 3035 expression time of implementation measurement mechanisms are measured from variable-length code (VLC) being input to decoder 303 backs to the time of implementation of finishing decoding and output image and add output.

Symbol 302 expression estimation units, the serial data that reception transmits according to receiving system 301, the execution number of times of each unit (code device 3031, anti-quadrature converting means 3032, converting unit 3033 or motion compensating device 3034) and the time of implementation of time of implementation measurement mechanism 305 are to estimate the time of implementation of each unit.

In order to estimate the time of implementation of each unit, can use linear regression method, the time of implementation of estimating is made as purpose variable y, the execution number of times of each parts is made as explanatory variable xui.In this case, the time of implementation that can regard regression parameter aui as each unit.And, under the situation of linear regression, the abundant former data that must add up, the result has wasted many memories.Yet,, also can use Kalman filter to estimate the internal state variable in order to avoid waste many memories.Can regard above-mentioned situation as such a case: observed value is made as the time of implementation, the time of implementation of each unit is made as the internal state variable, and each step of each unit execution number of times all changes the execution Matrix C.Symbol 304 is represented the number of times cutting devices, changes the execution number of times of each unit, to cut down the execution number of times of both full-pixel prediction, increases execution number of times one analog value of half-pix prediction.Show the method for calculating analog value below.

At first, receive the execution number of times of each unit and the time of implementation of estimation, to estimate the time of implementation from estimation unit 302.Surpass when receiving system 301 receives required time of data when the time of implementation, increase the execution number of times of both full-pixel prediction, reduce the execution number of times of half-pix prediction, be no more than one time of back up to the last time.The output of symbol 306 expression decoded pictures.

Such a case is arranged, promptly specify motion compensating device 3034, to carry out the half-pix prediction according to coded message.In this case, when surpassing the predetermined execution number of times of half-pix prediction, half-pix is moved the both full-pixel that rounds off move, to carry out the both full-pixel prediction.

According to the above-mentioned first and the 3rd embodiment, estimate time of implementation of decoding according to the estimation time of implementation of each unit, may surpass reception when needed time of one page data (fixed time) when the decoding time of implementation, replace long half-pix prediction of time of implementation with the both full-pixel prediction.Thereby, can prevent that the time of implementation from surpassing the fixed time, deal with problems (C1) (corresponding to claim 68 and 74).

And, two groups example is regarded as corresponding to claim 66 and 72 in necessary and inessential processing section, the example of video section being regarded as Wave data is corresponding to claim 64 and 70.

Receiver does not utilize high fdrequency component in IDCT calculates, thereby can reduce the processing time that IDCT calculates.That is, necessary processing is found out in the calculating of low frequency component, the calculating of high fdrequency component is regarded as the inessential processing of IDCT in calculating, also can reduce the calculation times that IDCT calculates the medium-high frequency component.

Figure 41 is the flow chart of the method for reseptance of the 4th embodiment.

Because the operation and the 3rd embodiment of present embodiment are similar, so the note units corresponding.In step 901, the variable a_i (estimation unit 302) of the time of implementation of each unit is represented in initialization.In step 902, the input multiplexed data, and measure the required time of multiplexed these data (receiving system 301).In step 903, multiplexed data is divided into variable-length sign indicating number and serial data, and output (receiving system 301).In step 904, obtain from serial data (Fig. 2) and respectively to carry out number of times, and be arranged to x_i.In step 905, according to the time of implementation a_i and the actual number of times (number of times cutting device 304) of carrying out of each execution number of times x_i calculating of each unit.In step 906, begin to measure the decoding time of implementation.In step 907, begin to carry out following decoding subprogram.After this, in step 908, finish to measure the decoding time of implementation (Video Decoder 303 and time of implementation measurement mechanism 3035).In step 908, estimate time of implementation of each unit according to the actual execution time of each unit of decoding time of implementation of step 908 and step 906, to upgrade a_i (estimation unit 302).Multiplexed data to each input carries out above-mentioned processing.

And, in the decoding subprogram of step 907, carry out length-changeable decoding (variable length decoding apparatus 3031) in step 910, after step 911 is carried out anti-quadrature conversion (anti-quadrature converting means 3032), processing in the piece that step 912 obtains according to the processing by step 910/information that inter codes is handled, carry out bifurcated (converting unit 3033).Under the situation that the piece interframe encode is handled, carry out motion compensation (motion compensating device 3034) in the step 913.In step 913, the time of implementation of double pixel prediction counts.When the time of implementation of counting surpasses the actual execution number of times that obtains in the step 905, replace the half-pix prediction with the both full-pixel prediction, so that carry out.After above-mentioned processing is applied to each macro block (step 914), finish this subprogram.

According to the above-mentioned second and the 4th embodiment, estimate time of implementation of decoding according to the estimation time of implementation of each unit, may surpass reception when needed time of one page data (fixed time) when the time of implementation, replace long half-pix prediction of time of implementation with the both full-pixel prediction.Thus, can prevent that the time of implementation from surpassing the fixed time, deal with problems (C1) (corresponding to claim 67 and 73).

And, two groups example is found out corresponding to claim 65 and 71 in the inessential and necessary processing section of part, the example of video section being found out Wave data is corresponding to claim 63 and 69.

Figure 36 shows the structure of the receiver of the 5th embodiment.

Most of parts of this embodiment are identical with the described parts of second embodiment.Yet, two parts that increase parts and a change are described below.

Symbol 402 expression estimation units, the estimation unit 302 that it changes embodiment 2 descriptions obtains, with the time of implementation of output according to each unit of result's acquisition of the output independent estimations of number of times cutting device 304.Symbol 408 expression dispensing devices, according to the time of implementation generation serial data shown in Figure 37 of each unit, and output.When being microsecond that unit, can represent up to about 65 milliseconds during the time of implementation with 16 bit representations.Therefore, these about 65 milliseconds will be enough.Symbol 409 expressions send serial data to the output of dispensing device.

And can only be right after increases the steps that produce serial data shown in Figure 37 and just can obtain method of reseptance corresponding to the 5th embodiment in symbol 808 back of Figure 40.

Figure 38 shows the structure of the transmitter of the 6th embodiment.

Most of parts of this embodiment are described identical with first embodiment.Yet, the parts of two increases are described below.Symbol 606 is represented inputs, receives the serial data of the receiver output of the 3rd embodiment 607, and 607 expression receiving systems receive serial data, and the time of implementation of exporting each unit.Device is determined in symbol 608 expressions, obtains the time of implementation of each unit, and its procurement process is described below.At first, converting unit 1021 is handled each macro block of image, thereby obtains the execution number of times of converting unit 1021 at this special time.And then, can determine the execution number of times of motion compensating device 1022, orthogonal converter 1023 and variable-length encoder 1024 uniquely according to result at this special time.Therefore, these execution number of times and times of implementation of utilizing receiving system 607 to send, estimate in the required time of implementation of receiver side decoding.The estimation decode time that obtains as time of implementation of each unit and carry out product between the number of times and obtain.If the decode time of estimating is equal to or greater than the required time of the yardage (for example 16Kbit) of this image generation that transmits appointments such as through-rate controller (for example when transmission rate is 64Kbits/s, it is 250 milliseconds), then increase the execution number of times of both full-pixel prediction, reduce the execution number of times of half-pix prediction, transmit the required time so that the decoding time of implementation of estimating is no more than.(because the time of implementation of both full-pixel prediction is short, so reduce the time of implementation that the number of times of both full-pixel prediction can reduce full prediction.)

In addition, video encoder 2102 carries out various processing according to the execution number of times of determining device 608 appointments.For example, after motion compensating device 1022 has been carried out the half-pix prediction with the predetermined execution number of times of half-pix prediction, only carry out the both full-pixel prediction.

Can improve system of selection, the half-pix prediction is uniformly dispersed in image.For example, can use following method: each macro block that at first obtains to carry out the half-pix prediction, calculate the result (3) that the number (for example 12) of above-mentioned macroblock number obtains divided by the execution number of times of half-pix prediction (for example 4), the serial number that only macro block that carries out the half-pix prediction from needs is begun does not have the macro block (0 of remainder divided by The above results then, 3,6 or 9) carry out the half-pix prediction.

According to the above-mentioned the 5th and the 6th embodiment, the unit time of implementation of each estimation is sent to transmitter side, estimate the time of implementation of decoding at transmitter side, and replace long half-pix prediction of time of implementation, thereby make the decoding time of implementation of estimation be no more than the time (fixed time) that a page data may need that receives with both full-pixel prediction.Thus, in the coded message of transmission, not discarded half-pix information of forecasting, and can prevent that the time of implementation from surpassing the fixed time, deal with problems (C2) (corresponding to claim 76 and 78).

In addition, under the situation of inessential processing, can be divided into three kinds of motion compensations to coding between macro block: normal motion compensation; 8 * 8 motion compensations and overlapping motion compensation.

Figure 42 is the flow chart of the sending method of the 7th embodiment.

Because the operation of present embodiment is similar to the 6th embodiment, so the note units corresponding.In step 1001, time of implementation of each processing is provided with initial value.At step 801 input picture (input 2101), and it is divided into macro block in step 802.In step 1002, determine each macro block is carried out coding or inter codes (converting unit 1021) in the piece.The result is the execution number of times that obtains each processing from step 1005 to step 806.Therefore, in step 1003, according to the execution number of times and the actual number of times (determining device 608) of carrying out of time of implementation calculating of above-mentioned each processing.

After this, repeating step 1005 is to the processing of step 806, up to the processing of having finished each macro block according to the conditional branching of step 807.

And, when carrying out each processing, variables corresponding is added 1, step 1005 to the number of processes of step 806 can be recorded in the specific variable.At first in step 1005, carry out bifurcated (converting unit 1021) according to definite result of step 1002.Under the situation of inter codes, carry out motion compensation (motion compensating device 1022) in step 804.In this case, the number of times of double pixel prediction is counted.When institute's metering number surpasses the actual frequency that obtains in step 1003, carry out the both full-pixel prediction and replace carrying out the half-pix prediction.After this, in

step

805 and 806, carry out dct transform and variable length code (orthogonal converter 1023 and variable-length encoder 1024).When the processing finished each macro block, (in step 807 under the situation that is) reads the variable of expression corresponding to the execution number of times of each processing in step 808, produces serial data shown in Figure 2, and multiplexed and dateout string and code.In step 1004, receive serial data, and be the time of implementation of each processing of taking-up and be provided with from this serial data.

As long as the processing of image input with regard to repeated execution of steps 801 to 1004 arranged.

According to the 5th embodiment and the 7th embodiment describe part last " and " paragraph of beginning, the time of implementation of the estimation of each unit is sent to transmitter side, estimate the decoding time of implementation at transmitter side, predict the half-pix prediction that replaces time of implementation length with both full-pixel, make the decoding time of implementation of estimation be no more than the needed possibility time (fixed time) of a page data that receives.Thus, in the coded message of transmission, the information of discarded half-pix prediction, and can prevent that the time of implementation from surpassing the fixed time, solved problem (C2) (corresponding to claim 75 and 77).

Figure 39 shows the structure of the transmitting apparatus of eighth embodiment of the invention.

Most of parts of this embodiment are identical with first embodiment description.Therefore, four parts of increase are described below.

Symbol 7010 expression time of implementation measurement mechanisms are used for measurement image and encode and time of implementation of output image to finishing to encoder 2102 input pictures after, and export the time of implementation that records.Symbol 706 expression estimation units, be used for receiving the execution number of times of each unit (converting unit 1021, motion compensating device 1022, orthogonal converter 1023 and variable length decoding apparatus 1024) from the serial data of counting device 2103, receive the time of implementation from time of implementation measurement mechanism 7010, and estimate the time of implementation of each unit.Can utilize the identical method of estimation of describing with the estimation unit 302 of second embodiment.The input of the frame rate value that symbol 707 expression input users send, device is determined in 708 expressions, is used to obtain the time of implementation of each unit.The process prescription that obtains is as follows.

At first, each macro block that converting unit 1021 is handled in the image is to obtain the execution number of times of converting unit 1021 at this special time.After this, can determine the execution number of times of motion compensating device 1022, orthogonal converter 1023 and variable-length encoder 1024 uniquely according to result to this special time.Then, each unit is obtained the execution number of times of each unit that estimation unit 706 sends and the summation of the product between estimated time.When the scramble time of estimating is equal to or greater than inverse from symbol 707 frame rate that transmits time that obtain, that can be used for one page image encoding, increase the execution number of times of both full-pixel prediction, the execution number of times of minimizing half-pix.

Repeat the increase and decrease of above-mentioned execution number of times, and the scramble time of calculate estimating, equal or be shorter than the available time up to the scramble time of estimating, thereby determine respectively to carry out number of times.

And video encoder 2102 carries out various processing according to the execution number of times of determining device 608 appointments.For example, after the predetermined execution number of times that motion compensating device 1022 is predicted with half-pix has carried out the half-pix prediction, only carry out the both full-pixel prediction.

And, also can improve system of selection, so that half-pix prediction dispersed structure in image.For example, can use following method: acquisition need be carried out each macro block of half-pix prediction, calculating need be carried out the result (3) that the macroblock number (for example 12) of half-pix prediction obtains divided by the execution number of times of half-pix prediction (for example 4), the sequence number that only macro block that carries out the half-pix prediction from needs is begun does not have the macro block (0 of remainder divided by The above results then, 3,6 or 9) carry out the half-pix prediction.

Above the 8th embodiment can deal with problems (C3), it estimates the time of implementation of each processing, according to the required time of implementation of estimating of time of implementation estimated coding, and determine to carry out number of times, make the scramble time of estimation equal or be shorter than the image of determining according to frame rate is encoded the available time (corresponding to claim 80).

And, because motion compensating device 1022 detects mobile vector,, in the vector in 15 levels and vertical pixel scope, detect the vector that makes SAD (the absolute difference sum of each pixel) minimum so a kind of full search method of detecting motion vector is arranged.Also have a kind of three moved further vector detection methods (in appendix H.261, description being arranged).Three moved further vector detection methods are carried out the processing of selecting equally distributed nine points in above-mentioned range of search, to select to have the point of minimum SAD, select nine points again in the approaching narrow scope of above-mentioned point, with the point of selecting one to have minimum SAD once more.

Also can suitably reduce the time of implementation of full search method of detecting motion vector, suitably increase the execution number of times of three moved further vector detection methods, these two kinds of methods as inessential processing method, and estimate time of implementation of each method in these two methods, according to the needed time of implementation of estimating of time of implementation estimated coding so that the time of implementation of estimating equal or be shorter than the time of user's appointment.

And, can use the fixedly method of detecting motion vector of retrieving number of times, further simplify and handle, perhaps with three frequency displacement dynamic vector detection methods, dual-purpose only returns the consequent method of detecting motion vector of mobile vector (0,0).

Figure 43 is the flow chart of the sending method of the 9th embodiment.

Because the operation of present embodiment is similar to the 8th embodiment, so the note units corresponding.To the description of the detailed description of each flow process with reference to corresponding units.

And, because present embodiment is almost identical with second embodiment, so the following difference of only explaining.

In step 1101, the initial value of the time of implementation of each processing is set to variable a_i.In step 1102, input frame speed (input 707).In step 1103, according to the frame rate of each processing of step 1102 and time of implementation a_i and in 1002 piece/piece sound encoding process determines that each execution number of times of handling that the result obtains determines the actual number of times (determining device 708) of carrying

out.In step

1105 and 1106, measure the time of implementation of coding.In step 1104, estimate time of implementation of each processing, more new variables a_i (estimation unit 706) according to the actual execution number of times of the time of implementation that obtains in the step 1106 and each processing.

According to above-mentioned the 9th embodiment, estimate the time of implementation of each processing earlier, can be according to the required time of implementation of estimated time of implementation predictive coding.Therefore, determine to carry out number of times, make the scramble time of estimation equal or be shorter than the image of determining according to frame rate is encoded the available time (corresponding to claim 79), thereby can deal with problems (C3).

Under the situation of second embodiment, when in step 808, producing serial data, also can after opening code shown in Figure 2, increase the zone of two bytes, increase the binary system label of code length to this zone.

And, in the 4th embodiment, when at step 902 input multiplexed data, also can be from two byte area code fetch length, and the execution number of times that will be used for step 905 from the sign indicating number transmission time of this code length and the acquisition of sign indicating number transmission rate calculates (reducing the execution number of times of half-pix prediction, to be no more than a yard transmission time).This is corresponding to claim 81 and 83.

And, in first embodiment,, also can behind opening code shown in Figure 2, increase the zone of two bytes when when step 2104 produces serial data, the binary system that increases code length in this zone indicates.

And, in the 3rd embodiment, when at step 301 input multiplexed data, also can be from this two byte area code fetch length, and the execution number of times that will be used for step 304 from the sign indicating number transmission time of this code length and the acquisition of sign indicating number transmission speed calculates (reducing the execution number of times of half-pix prediction, to be no more than a yard transmission time).This is corresponding to claim 82 and 84.

And, in the 4th embodiment, be right after actual execution number of times, to calculate maximum in the record half-pix prediction of step 909 back.When maximum is equal to or less than enough little value (for example 2 or 3), also can produce serial data (serial data that comprises concrete bit pattern), the half-pix prediction is not used in expression, and sends the serial data that produces.And, in a second embodiment, confirm whether to be right after and after step 808, receive this serial data, when receiving expression when not using half-pix data predicted string, motion compensation is handled always predicted as both full-pixel in step 808.This is corresponding to claim 93 and 91.

And above-mentioned principle also can be applied to not be the situation of motion compensation.For example DCT calculating can be without high fdrequency component, to reduce the computing time of DCT.That is, under the situation of method of reseptance,, transmit the serial data of this ratio of expression above a certain value to transmitter side when IDCT calculates the time of implementation when ratio of whole computing time is surpassed a certain value.When transmitter side receives this serial data, also can calculate and only calculate low frequency component, and all high fdrequency components are reduced to zero by DCT.This is corresponding to claim 89.

And, though the present embodiment that utilized iamge description also can be applied to audio frequency to every kind of method rather than frequently looks.This is corresponding to claim 85 and 87.

And, in the 3rd embodiment, in step 3034, write down the actual execution number of times of half-pix prediction, to calculate the maximum number of times maximum of carrying out.Then, when this maximum is enough little value or more hour (for example 2 or 3), can produce and send expression and not use half-pix data predicted string (serial data that comprises concrete bit pattern).And, in first embodiment,, the motion compensation of step 1022 is handled always predicted as both full-pixel when receiving expression when not using half-pix data predicted string.This is corresponding to claim 94 and 92.

And above-mentioned principle can be applied to not be the situation of motion compensation.For example DCT calculates and does not utilize high fdrequency component can reduce the processing time that DCT calculates.That is, in method of reseptance,, transmit the serial data of this ratio of expression above a certain value to transmitter side when IDCT calculates the time of implementation when ratio of whole time of implementation is surpassed a certain value.

When transmitter side receives this serial data, can calculate by DCT and only calculate low frequency component, all high fdrequency components are reduced to zero.This is corresponding to claim 90.

And, though above the embodiment that utilized iamge description, also can be applied to sound rather than image to said method.This is corresponding to claim 86 and 88.

As mentioned above, according to claim 68 and 74 (as the first and the 3rd embodiment), estimate the time of implementation of decoding according to the estimation time of implementation of each unit, and, may surpass when receiving required time (fixed time) of a page data when the decoding time of implementation of estimating, replace long half-pix prediction of time of implementation with the both full-pixel prediction.Thus, can prevent that the time of implementation from surpassing the fixed time, (C1) deals with problems.

And, according to claim 75 and 77 (for example the 5th and the 7th embodiment), the estimation time of implementation of each unit is sent to transmitter side, estimate the decoding time of implementation at transmitter side, and,, the decode time of estimating is suitable for receiving the needed time of a page data (fixed time) so that being no more than with the long half-pix prediction of both full-pixel prediction replacement time of implementation.Thus, the information of discarded half-pix prediction in the coded message that sends, and can prevent that the time of implementation from surpassing the fixed time, (C2) deals with problems.

And, according to claim 79 (for example the 9th embodiment), (C3) can deal with problems, its method is to estimate the time of implementation of each processing, according to the estimated needed time of implementation of time of implementation estimated coding, and determine to carry out number of times, make the scramble time of estimation be equal to or less than the available time that the image of determining according to frame rate is encoded and can be done.

Therefore, even the present invention can realize the calculated load increase function (CGD: calculating moderate reduction) of quality deterioration that also slows down, thereby obtain very big interests.

And, computer utilization such as magnetic recording medium or optical recording media can be realized above-mentioned operation, wherein make the program of computer realize all or part of of each step (or each device) (the perhaps operation of each device) of describing among arbitrary embodiment in the foregoing description.

Industrial applicability

As mentioned above, the present invention can change information frame corresponding to state, purposes or transmission line, dynamically determines Data control information, control information transmission and be used for the frame of the control information of sending and receiving terminal. And, be easy to Process a plurality of video data streams or a plurality of audio data stream, and have emphasis ground and synchronous sound ground to reproduce main field The dough sheet section, reflection editor's intention. In addition, according to the time of implementation of the element of each estimation, when the solution of estimating The code time of implementation replaces holding with the both full-pixel prediction in the time of may surpass receiving required time (fixed time) of a page data The half-pix prediction that line time is long can prevent that the time of implementation from surpassing the time of appointment.

Claims

1, a kind of data processing method is characterized in that,

The time series data of paired input video or audio frequency, and represent that priority and expression between the time series data of the processing priority between described time series data constitute priority in the time series data of processing priority of time series data interframe of described video, (1) when the information of described time series data is damaged, with priority between priority in the described time series data and described time series data, carry out handling for the requirement of sending again of sending again that requires its damaged data, or (2) are when described time series data is lost continuously or frequently, with priority between priority in the described time series data and described time series data, only the high data of processing priority being carried out described sending again requires to handle

In above-mentioned (2), according to the described at least one side who send again in the loss late that requires the number of times of handling and lose described time series data, the described threshold value of sending the processing priority that is suitable for when requiring to handle is again carried out in decision.

2, a kind of data processing method is characterized in that,

The paired time series data of importing audio frequency or containing I-frame video, and represent that priority and expression between the time series data of the processing priority between described time series data constitute priority in the time series data of processing priority of I interframe of time series data of described video, (1) when the information of described time series data is impaired, with priority between priority in the described time series data and described time series data, carry out handling for the requirement of sending again of sending again that requires its damaged data, or (2) are when described time series data is lost continuously or frequently, with priority between priority in the described time series data and described time series data, only the high data of processing priority being carried out described sending again requires to handle