JP6105717B2 - Improved block request streaming system for handling low latency streaming - Google Patents

Description

  The present invention relates to an improved media streaming system and method, and more particularly to adapting to network and buffer conditions to optimize the presentation of streamed media, and to efficiently and efficiently streamed media data. The present invention relates to systems and methods that enable simultaneous delivery or timely delivery.

  As high quality audio and video is more commonly delivered over packet-based networks such as the Internet, cellular and wireless networks, power line networks, and other types of networks, streaming media delivery is increasingly Can be important. The quality at which delivered streaming media can be presented includes the resolution (or other attributes) of the original content, the encoding quality of the original content, the ability of the receiving device to decode and present the media, the receiver Can depend on a number of factors, including the timeliness and quality of the signal received at. In order to create a perceived good streaming media experience, the transport and timeliness of the signal received at the receiver may be particularly important. Good transport can provide fidelity of the stream received at the receiver to what the sender sends, while timeliness is how fast the receiver is after the initial request for content. Can indicate whether or not the reproduction of the content can be started.

  A media distribution system may be characterized as a system having a media source, a media destination, and a channel (in time and / or space) that separates the source and destination. Typically, a source is a transmitter that has access to an electronically manageable form of media and an electronic control of the reception of the media (or the like) to control the media (e.g., receiver, A storage device or element, another channel, etc., and a receiver capable of providing to a user having a display device coupled in some manner.

  Although many variations are possible, in a general example, a media distribution system has one or more servers with access to media content in electrical form, and one or more client systems Or the device makes a request for media to the server, the server uses the transmitter as part of the server to carry the media, and the reception at the client so that the received media can be utilized by the client in some manner. To the machine. In a simple example, there is one server and one client for a given request and response, but this need not be the case.

  Traditionally, media distribution systems can be characterized as either a “download” model or a “streaming” model. The “download” model can be characterized by timing independence between the delivery of media data and the playback of the media to the user or recipient device.

  As an example, the media is downloaded well before the media is needed or will be used, and when the media is used, the amount needed is already at the recipient. Is available. Delivery in the download context is often performed using a file transport protocol such as HTTP, FTP, or File Delivery over Unidirectional Transport (FLUTE), and the delivery rate is behind, such as TCP / IP Depending on the flow and / or congestion control protocol in The operation of the flow or congestion control protocol may be independent of media playback for the user or destination device, and media playback may occur at the same time as the download or otherwise.

  The “streaming” mode can be characterized by a close coupling between the timing of delivery of the media data and the timing of playback of the media to the user or recipient device. Distribution in this context is often performed using streaming protocols such as Real Time Streaming Protocol (RTSP) for control and Real Time Transport Protocol (RTP) for media data. The delivery rate may be determined by the streaming server and often matches the data playback rate.

L. Rizzo, "Effective Erasure Codes for Reliable Computer Communication Protocols", Computer Communication Review, 27 (2): 24-36 (April 1997) Bloemer et al., `` An XOR-Based Erasure-Resilient Coding Scheme '', Technical Report TR-95-48, International Computer Science Institute, Berkeley, California (1995)

  Some disadvantages of the “download” model are that due to the timing independence of delivery and playback, none of the media data may be available when needed for playback (e.g., media Caused by a temporary suspension of playback (`` stall ''), resulting in a bad user experience, or media data downloaded much before playback Consume storage resources of the receiving device, which may be scarce (e.g., due to greater available bandwidth than the media data rate), and A valuable network resource for delivery that can be wasted if the content is not finally played or otherwise used It may be to consume.

  The advantage of the “download” model is that the technologies required to perform such downloads, such as HTTP, are very mature, widely deployed and applicable across a wide range of uses. possible. Download servers, and solutions for such file download great scalability (eg, HTTP web servers and content delivery networks) may be readily available, making deployment of services based on this technology simple and low cost To.

  Some drawbacks of the “streaming” model are that the rate of delivery of media data generally does not match the bandwidth available in the server-to-client connection, and special streaming that provides bandwidth and delay guarantees There may be a need for servers or more complex network architectures. There are streaming systems that respond to variations in delivery data rate depending on available bandwidth (e.g. Adobe Flash Adaptive Streaming), but these are generally similar to TCP in all available bandwidth usage. Not as efficient as the download transport flow control protocol.

  Recently, new media distribution systems based on the combination of “streaming” and “download” models have been developed and deployed. An example of such a model is referred to herein as a “block request streaming” model, where a media client requests a block of media data from a serving infrastructure using a download protocol such as HTTP. The challenge in such systems is the ability to initiate playback of the stream, eg, using a personal computer to decode and render the received audio and video streams, display the video on a computer screen and through the built-in speakers Play audio or, as another example, use a set-top box to decode and render received audio and video streams, display video on a television display device, and play audio through a stereo system Could be.

  It is also a problem to be able to decode the source block fast enough to keep up with the source streaming rate, for example, to minimize decoding latency and reduce the use of available CPU resources. . Another challenge is to provide a robust and scalable method of streaming delivery that does not adversely affect the quality of the stream delivered to the receiver if a system component fails. As the presentation is distributed, other problems may arise due to the fast converting information about the presentation. Therefore, it is desirable to have an improved process and apparatus.

  Block request streaming systems provide improved user experience and bandwidth efficiency for such systems, and capture systems that typically generate data in the format (HTTP, FTP, etc.) provided by traditional file servers , The capture system captures the content and prepares the content as a file or data element to be provided by the file server, which may or may not include a cache.

  According to one embodiment, the media server of the block request streaming system enables low latency streaming of media presentation content. A relatively large segment file for live profile streaming can be integrated from a relatively small media fragment for low latency streaming. Media segments and media fragments are encoded according to the same encoding protocol.

  The following detailed description, together with the accompanying drawings, provides a further understanding of the nature and advantages of the present invention.

FIG. 3 illustrates elements of a block request streaming system according to an embodiment of the present invention. FIG. 2 illustrates the block request streaming system of FIG. 1 showing in more detail elements of a client system coupled to a block serving infrastructure (“BSI”) that receives data processed by a content capture system. It is a figure which shows the hardware / software implementation form of an acquisition system. It is a figure which shows the hardware / software implementation form of a client system. FIG. 2 illustrates a possible structure of the content storage device shown in FIG. 1, including segment and media presentation description (“MPD”) files, as well as the breakdown, timing, and other structure of the segments in the MPD file. FIG. 6 is a diagram showing details of a normal source segment that can be stored in the content storage device shown in FIGS. 1 and 5. FIG. 6 illustrates simple and hierarchical indexing within a file. FIG. 6 illustrates simple and hierarchical indexing within a file. FIG. 6 illustrates variable block sizing with search points aligned across multiple versions of a media stream. FIG. 6 illustrates variable block sizing with search points that are not aligned across multiple versions of a media stream. It is a figure which shows a metadata table. It is a figure which shows transmission of the block and metadata table from a server to a client. It is a figure which shows the block independent of the boundary of RAP. It is a figure which shows the continuous timing and discontinuous timing over a some segment. It is a figure which shows the certain aspect of a scalable block. FIG. 3 shows a graphical representation of the evolution over time of some variables in a block request streaming system. FIG. 6 shows another graphical representation of the evolution over time of some variables in a block request streaming system. FIG. 6 shows a cell grid of states as a function of threshold. FIG. 4 is a flowchart of a process that may be performed at a receiver that may request a single block and multiple blocks per request. Figure 5 is a flowchart of a flexible pipeline process. FIG. 4 is a diagram illustrating an example of a candidate set of requests, request priority, and on which connection a request can be issued at a certain time. FIG. 4 is a diagram illustrating an example of a candidate set of requests, request priority, and on which connection a request can be issued, changed from one time to another. FIG. 6 is a flowchart of selecting a consistent caching proxy server based on a file identifier. It is a figure which shows the syntax definition with respect to a suitable formula language. It is a figure which shows the example of a suitable hash function. It is a figure which shows the example of a file identifier construction rule. It is a figure which shows the fluctuation | variation of the bandwidth of a TCP connection. It is a figure which shows the some HTTP request with respect to source data and repair data. FIG. 3 illustrates an exemplary channel zapping time with FEC and without FEC. FIG. 2 shows details of a repair segment generator that generates a repair segment from source segments and control parameters as part of the capture system shown in FIG. It is a figure which shows the relationship between a source block and a repair block. It is a figure which shows the procedure for the live service in a different time in a client. It is a figure which shows the relationship between the media fragment for low latency streaming, and a media fragment.

  In the drawings, similar items are referred to by similar numerals, and sub-indexes are given in parentheses to indicate multiple instances of similar or identical items. Unless otherwise indicated, the last subindex (e.g., `` N '' or `` M '') is not intended to be limited to any particular value, and the number of instances of an item is indicated by the same number. Even if the subindex is reused, it can be different from the number of instances of another item.

  As described herein, the purpose of a streaming system is to use the media from its storage location (or the location where the media is generated), the location where the media is utilized, i.e., the media is presented to the user, It is moved to a location that is otherwise “used up” by a human or electronic user. Ideally, a streaming system can achieve uninterrupted playback (or more generally, uninterrupted “consumption”) at the receiver, as soon as the user requests one or more streams. In addition, playback of one stream or a collection of streams can be started. For efficiency reasons, when the user indicates that the stream is no longer needed, such as when switching from one stream to another, or when the user follows a presentation of a stream, for example a “subtitle” stream, It is also desirable that each stream be stopped. If a media component such as video continues to be presented, but a different stream is selected to present this media component, it is preferable to occupy the limited bandwidth with the new stream and stop the old stream There are many.

  The block request streaming system according to the embodiments described herein provides many advantages. Some applications may provide a reasonably satisfactory experience with less than all of the features described herein, so that a viable system has all of the features described herein. It should be understood that it is not necessary to include.

HTTP streaming
HTTP streaming is a specific type of streaming. For HTTP streaming, the source may be a standard web server and content delivery network (CDN) and may use standard HTTP. This technique may involve streaming segmentation and the use of multiple streams, all within the context of a standardized HTTP request. Media such as video may be encoded at multiple bit rates to form different versions or representations. The terms “version” and “representation” are used interchangeably in this document. Each version or representation can be broken down into smaller pieces, sometimes in the order of seconds, to form segments. Each segment can then be stored on the web server or CDN as a separate file.

  On the client side, requests can be made using HTTP for individual segments that are seamlessly stitched together by the client. The client can switch to a different data rate based on the available bandwidth. The client can also request multiple representations, each presenting a different media component, and the media in these representations can be presented together and simultaneously. What causes the switch can be, for example, buffer occupancy and network measurement results. When operating in a steady state, the client can adjust requests to the server to maintain the target buffer occupancy.

  The advantages of HTTP streaming may include bit rate adaptation, fast startup and searching, and minimal unnecessary delivery. These benefits include controlling delivery to be in front of playback time, maximizing available bandwidth (through variable bit rate media), and streaming segmentation and intelligent By optimizing client-side procedures.

  A media presentation description is provided to the HTTP streaming client so that the client can provide streaming services to the user using a collection of files (eg, in the format defined by 3GPP, referred to herein as the 3gp segment) Can be done. The media presentation description and, in some cases, updates to this media presentation description, allow clients to present the included media in a synchronized manner, and include advanced search, bit rate switching, and combined presentation of media components in different representations. Describes a media presentation, which is a structured collection of segments, each containing media components so that it can provide functionality. The client can use the media presentation description information in various ways for service provisioning. Specifically, from the media presentation description, the HTTP streaming client determines which segments in the aggregate can be accessed so that the data is useful to the client capabilities and users in the streaming service. be able to.

  In some embodiments, the media presentation description may be unchanged, but the segments may be created dynamically. The media presentation description can be as small as possible to minimize access and download times for the service. Other dedicated server connectivity, eg, regular or frequent timing synchronization between the client and server, can be minimized.

  Media presentations can be constructed to allow access by terminals with different capabilities, such as access to different types of access networks, different current network conditions, display sizes, access bit rates, and codec support. The client can then extract the appropriate information and provide the streaming service to the user.

  The media presentation description may also allow deployment flexibility and miniaturization according to requirements.

  In the simplest case, each Alternative Representation is either a single 3GP file, i.e. a file that conforms to that defined in 3GPP TS26.244, or ISO / IEC 14496-12 or a derived specification (3GPP Technical It can be stored in any other file that conforms to the ISO-based media file format as defined in 3GP file format described in Specification 26.244. In the rest of this document, when referring to 3GP files, ISO / IEC 14496-12 and derived specifications are defined in ISO / IEC 14496-12 or any derived specification. It should be understood that this can be mapped to a more general ISO-based media file format. The client then requests the first part of the file and knows the media metadata (usually stored in the Movie header box, also called the “moov” box), along with the time and byte offset of the movie fragment. it can. The client can then issue an HTTP partial get request to obtain the requested movie fragment.

  In some embodiments, it may be desirable to divide each representation into several segments. If the segment format is based on the 3GP file format, the segment includes non-overlapping time slices of movie fragments, called “temporal partitioning”. Each of these segments may include multiple movie fragments, each of which can be a valid 3GP file in itself. In another embodiment, the representation is divided into an initial segment containing metadata (usually a Movie header “moov” box) and a set of media segments, each media segment containing media data, the initial segment and any media The concatenation of segments forms a valid 3GP file, and the concatenation of all media segments of the initial segment and one representation forms a valid 3GP file. The entire representation can be formed by playing each segment and then associating a local timestamp in the file with a global presentation time according to the start time of each representation.

  Throughout this description, references to “segments” are fully or partially constructed, or read from a storage medium, or otherwise obtained as a result of a file download protocol request including, for example, an HTTP request. Note that it should be understood to include any data object. For example, in the case of HTTP, the data object may be stored in an actual file that resides on disk or other storage medium connected to or forming part of an HTTP server, or the data object is It can be built by CGI scripts or other dynamically executed programs that are executed in response to HTTP requests. The terms “file” and “segment” are used interchangeably in this document unless otherwise specified. In the case of HTTP, a segment may be considered the entity body of an HTTP request response.

  The terms “presentation” and “content item” are used interchangeably in this document. In many examples, the presentation is an audio, video, or other media presentation with a defined “play” time, but other variations are possible.

  The terms “block” and “fragment” are used interchangeably in this document, unless otherwise specified, and generally refer to the smallest collection of indexed data. Based on available indexing, the client can request different parts of a fragment with different HTTP requests, or request one or more consecutive fragments or parts of a fragment with one HTTP request Can do. If an ISO-based media file format-based segment or 3GP file format-based segment is used, the fragment is usually defined as a combination of a movie fragment header (“moof”) box and a media data (“mdat”) box Refers to a movie fragment.

  In this document, the network carrying the data is assumed to be packet-based to simplify the description herein, and after reading this disclosure, one skilled in the art will be described herein. It will be appreciated that embodiments of the present invention can be applied to other types of transmission networks, such as continuous bitstream networks.

  In this specification, FEC codes are assumed to provide protection against long and variable delivery times of data to simplify the description herein, and after reading this disclosure, one of ordinary skill in the art will recognize the present invention. It will be appreciated that this embodiment can be applied to other types of data transmission problems such as bit reversal corruption of data. For example, in the absence of FEC, if the last part of the requested fragment arrives much later, or if the arrival time variation is larger than the previous part of the fragment, the content zapping time can be large and variable However, with FEC and parallel requests, only the majority of the data requested for a fragment needs to be reached before the data can be recovered, so content zapping time can be reduced and content zapping time can vary. Get smaller. In this description, it may be assumed that the data to be encoded (eg, source data) has been decomposed into equal length “symbols” that can be of arbitrary length (minimum single bit). The symbols may be of different lengths for different parts of the data, for example, different symbol sizes may be used for different blocks of data.

  In this description, to simplify the description herein, it is assumed that the FEC is applied to a “block” or “fragment” of data at a time, ie, the “block” is the FEC encoding and It is a “source block” for decoding. The client device can help determine the source block structure of the segment using segment indexing as described herein. One skilled in the art can apply embodiments of the present invention to other types of source block structures, for example, a source block may be part of a fragment, or one or more fragments or fragment parts Can be included.

  FEC codes that are considered for use with block request streaming are typically systematic FEC codes, i.e. the source symbols of the source block can be included as part of the source block encoding so that the source symbols are transmitted. The As those skilled in the art will appreciate, the embodiments described herein apply equally well to non-systematic FEC codes. A systematic FEC encoder generates a certain number of repair symbols from a source block of source symbols, and at least some combinations of source symbols and repair symbols are encoded symbols transmitted over a channel representing the source block. Some FEC codes may be useful in efficiently generating as many repair symbols as needed, such as "information add code" or "fountain code", examples of these codes include There are “chain reaction codes” and “multi-stage chain reaction codes”. Other FEC codes, such as Reed-Solomon codes, can actually only produce a limited number of repair symbols for each source block.

  In many of these examples, it is assumed that the client is coupled to a media server or multiple media servers and that the client requests streaming media from the media server or multiple media servers through one or more channels. The However, more complex configurations are possible.

Example Benefits In block request streaming, the media client maintains a relationship between the timing of these block requests and the timing of media playback for the user. While this model retains the advantages of the “download” model described above, it can avoid some of the disadvantages that usually arise from the elimination of the relationship of media playback to data delivery. The block request streaming model takes advantage of the rate and congestion control mechanisms available in transport protocols such as TCP to ensure that the maximum available bandwidth is used for media data. In addition, the division of the media presentation into blocks allows each block of encoded media data to be selected from a plurality of available encoding sets.

  This choice means that the media data rate matches the available bandwidth, even if the available bandwidth changes over time, the resolution of the media or the complexity of the decoding for the capabilities or configuration of the client. May be based on any number of criteria, including being adapted or adapted to user preferences such as language. This selection may also include downloading and presenting auxiliary components such as accessibility components, closed captions, subtitles, sign language videos, and the like. Examples of existing systems that use the block request streaming model are Move Networks ™, Microsoft Smooth Streaming, and Apple iPhone ™ Streaming Protocol.

  In general, each block of media data may be stored on the server as a separate file, and then a protocol such as HTTP is used with HTTP server software running on the server to request the file as a unit. . Typically, the client is given a metadata file, which can be, for example, Extensible Markup Language (XML) format or playlist text format or binary format, and available encodings (e.g., required bandwidth, This document describes the characteristics of the media presentation, usually called “representation” in this document, such as resolution, coding parameters, media type, language), and the manner in which the encoded material is divided into blocks. For example, the metadata may include a Uniform Resource Locator (URL) for each block. To indicate that the protocol to be used to access the documented resource is HTTP, the URL itself may provide a scheme such as the one with the string “http: //” appended it can. Another example is “ftp: //” to indicate that the protocol to be used is FTP.

  In other systems, for example, media blocks may be built in time by the server “on the fly” in response to a request from a client indicating the portion of the media presentation that is being requested. For example, in the case of “http: //” style HTTP, execution of the request for this URL provides a request response that includes some specific data in its entity body. The implementation in the network of how to generate this request response can vary greatly depending on the implementation of the server serving such a request.

  In general, each block may be independently decodable. For example, for video media, each block may start at a “search point”. In some coding schemes, the search points are referred to as “random access points” or “RAPs”, but not all RAPs may be designated as search points. Similarly, in other coding schemes, search points start with an “Independent Data Refresh” frame or “IDR” in the case of H.264 video encoding, but not all IDRs may be designated as search points. . A search point is a position in the video (or other) media where the decoder can start decoding without requiring any data about the previous frame or data or sample, and the frame or sample being decoded is standalone If not encoded in a manner, for example, encoded as the difference between the current frame and the previous frame, the decoder may need data for the previous frame or data or samples.

  The challenge in such systems is the ability to initiate playback of the stream, eg, using a personal computer to decode and render the received audio and video streams, display the video on a computer screen and through the built-in speakers Play audio or, as another example, use a set-top box to decode and render received audio and video streams, display video on a television display device, and play audio through a stereo system Could be. The main challenge is when the user decides to see the new content delivered as a stream and takes action to represent that decision, for example, when the user clicks a link in a browser window or plays a remote control device It may be to minimize the delay from when the button is pressed to when the content begins to be displayed on the user's screen, and this delay is hereinafter referred to as “content zapping time”. Each of these challenges may be addressed by the improved system elements described herein.

  An example of content zapping is that the user is watching the first content delivered via the first stream, then the user decides to watch the second content delivered via the second stream, This is when the operation for starting to view the second content starts. The second stream may be sent from the same set of servers as the first stream or from a different set of servers. Another example of content zapping is when a user visits a website and starts watching the first content delivered via the first stream by clicking a link in the browser window It is time to decide. In a similar manner, the user can decide to start playing the content at some time in the stream rather than from the beginning. The user indicates to the client device to search for a time position and the user can expect the selected time to be rendered instantly. Minimizing content zapping time is important for video viewing to enable users to have a high-quality, high-speed content surfing experience when searching and experimenting with a wide variety of available content. is there.

  Recently, it has become common to consider using forward error correction (FEC) codes to protect streaming media during transmission. When transmitted through a packet network that includes the Internet and wireless networks, such as those standardized by groups such as 3GPP, 3GPP2, and DVB, the source stream is routed to packets as they are generated or made available As placed, the packets can be used to carry the source or content stream in the order that they are generated and made available to the recipient.

  In general application of these types of FEC codes to scenarios, the encoder can use the FEC codes in creating repair packets, which are then transmitted in addition to the original source packet including the source stream. Is done. The repair packet has the property that when a loss of the source packet occurs, the received repair packet can be used to recover the data contained in the lost source packet. A repair packet can be used to recover the contents of a completely lost lost source packet, but is either a fully received repair packet, or even a partially received repair packet, Repair packets can be used to recover from the occurrence of partial packet loss. Thus, fully or partially received repair packets can be used to recover completely or partially lost source packets.

  In yet another example, other types of corruption may occur in the transmitted data, for example, the value of a bit may be reversed, so that the repair packet corrects such corruption and allows for the source packet It can be used to achieve as accurate recovery as possible. In other examples, the source stream is not necessarily transmitted in separate packets, but may instead be transmitted, for example, as a continuous bit stream.

  There are many examples of FEC codes that can be used to provide source stream protection. A Reed-Solomon code is a well-known code for correcting errors and erasures in a communication system. For example, a well-known and efficient implementation of Reed-Solomon codes for erasure correction over packet data networks is L. Rizzo, “Effective Erasure Codes for Reliable Computer Communication Protocols”, Computer Communication Review, 27 ( 2): 24-36 (April 1997) (hereinafter `` Rizzo ''), Bloemer et al., `` An XOR-Based Erasure-Resilient Coding Scheme '', Technical Report TR-95-48, International Computer Science Institute, California The Cauchy matrix or van der Monde matrix is used as described in State Berkeley (1995) (hereinafter "XOR-Reed-Solomon") or elsewhere.

  Other examples of FEC codes include LDPC codes, chain reaction codes as described in Luby I, and multi-stage chain reaction codes as in Shokrollahi I.

  Examples of FEC decoding processes for Reed-Solomon code variants are described in Rizzo and XOR-Reed-Solomon. In those examples, decoding may be applied after sufficient source and repair data packets are received. The decryption process can be computationally intensive and depending on the available CPU resources, the decryption process can take a significant amount of time to complete for the length of time that the media spans in the block There is. The receiver may consider this length of time required for decoding when calculating the required delay between the start of receiving the media stream and playing the media. This delay due to decoding is perceived by the user as a delay from the user's request for a particular media stream to the start of playback. Therefore, it is desirable to minimize this delay.

  In many applications, the packet may be further subdivided into symbols to which the FEC process is applied. A packet may contain one or more symbols (or may contain less than one symbol, but typically symbols are known to be highly correlated with error conditions between groups of packets. Unless otherwise split across groups of packets). The symbols can have any size, but the size of the symbol is often at most equal to the size of the packet. A source symbol is a symbol that decodes data to be transmitted. A repair symbol is a symbol that is generated directly or indirectly from the source symbol in addition to the source symbol (i.e., the data to be transmitted is available to all of the source symbols and any of the repair symbols If not possible, it can be fully restored).

  Some FEC codes may be block-based in that their encoding operation depends on symbols that are in the block and may be independent of symbols that are not in the block. In block-based coding, the FEC encoder can generate repair symbols for a block from the source symbols in that block and then move on to the next block, other than for the current block being coded There is no need to reference the source symbol. In transmission, a source block that includes source symbols may be represented by an encoded block that includes encoded symbols (which may be some source symbols, some repair symbols, or both). Due to the presence of repair symbols, not all of the source symbols are required in each coding block.

  For some FEC codes, especially Reed-Solomon codes, the encoding and decoding times can become impractical as the number of encoded symbols per source block increases. Therefore, in practice, especially in the typical case where the Reed-Solomon encoding or decoding process is performed by custom hardware, for example included as part of the DVB-H standard to protect the stream against packet loss If the MPE-FEC process using a Reed-Solomon code is implemented on special hardware in a mobile phone that is limited to a total of 255 Reed-Solomon encoded symbols per source block, There is often a practical upper limit to the total number of encoded symbols that can be obtained (255 is an approximate practical limit in some applications). This places a practical upper limit on the maximum length of the source block to be encoded, since symbols are often required to be placed in a separate packet payload. For example, if the packet payload is limited to 1024 bytes or less and each packet carries one encoded symbol, the encoded source block may be at most 255 kilobytes, which is of course the size of the source block itself. It is also the upper limit.

  Minimize the decoding latency introduced by FEC decoding, such as being able to decode the source block fast enough to keep up with the source streaming rate, and the available CPU at the receiving device at any point during FEC decoding. Other challenges for using only small pieces are addressed by the elements described herein.

  What is needed is to provide a robust and scalable stream delivery method that does not adversely affect the quality of the stream delivered to the receiver if a system component fails.

  Block request streaming systems can make changes to the structure or metadata of the presentation, eg, changes to the number of available media encodings, or media encodings such as bit rate, resolution, aspect ratio, audio or video codec, or codec parameters. Need to support changes to other parameters, or other metadata changes such as URLs associated with content files. Such changes may be required as a result of announcements in different segments of larger presentations, such as configuration changes, equipment failures or recovery from equipment failures, or other reasons that change the serving infrastructure May be required for a number of reasons, including editing together content from different sources, such as modifying the parameters of

  There are methods where the presentation can be controlled by a playlist file that is continuously updated. Since this file is continuously updated, at least some of the changes described above can be made in these updates. Disadvantages of the traditional method, also called “polling”, are that the client device has to continuously retrieve the playlist file, which puts a load on the serving infrastructure and It is impossible to be cached for a long time, and the task of serving infrastructure is much more difficult. The above is addressed by the elements herein so that the type of update described above is performed without the need for continuous polling by the client to the metadata file.

  Another problem commonly known from broadcast delivery, especially in live services, is that the user cannot see content that was broadcast before the user joined the program. Typically, local personal recording is unnecessarily consuming local storage capacity or impossible because the client was not matched to the program, or prohibited by content protection rules. Network recording and time-shifted viewing are preferred, but require individual connections to the server and a delivery protocol and infrastructure that is separate from the live service, resulting in infrastructure duplication and large server costs. This is also addressed by the elements described herein.

System Overview One embodiment of the present invention will be described with reference to FIG. 1, which shows a simplified diagram of a block request streaming system embodying the present invention.

  In FIG. 1, a block streaming system 100 is shown that includes a block serving infrastructure (“BSI”) 101 that includes a capture system 103 that captures content 102 and HTTP streaming with the capture system 103. By storing the content in a content storage device 110 accessible to both servers 104, the content is prepared and packaged for service by the HTTP streaming server 104. As shown, the system 100 may also include an HTTP cache 106. In operation, a client 108, such as an HTTP streaming client, sends a request 112 to the HTTP streaming server 104 and receives a response 114 from the HTTP streaming server 104 or the HTTP cache 106. In each case, the elements shown in FIG. 1 may be implemented at least in part in software that includes program code executed on a processor or other electronic circuit.

  Content may include movies, audio, 2D flat video, 3D video, other types of video, images, timed text, timed metadata, and the like. Some content is presented or used in a timely manner, such as data to present auxiliary information (such as station identification information, advertisements, stock prices, Flash (TM) sequences, etc.) along with other media being played. Can be accompanied by data to be done. Other hybrid presentations that combine other media and / or go beyond just audio and video may also be used.

  As shown in FIG. 2, the media block may be, for example, an HTTP server, content delivery network device, HTTP proxy, FTP proxy or server, or some other media server or system, block serving infrastructure 101 (1) Can be stored within. Block serving infrastructure 101 (1) is connected to a network 122, which may be, for example, an Internet Protocol (“IP”) network such as the Internet. The block request streaming system client is given the six functional components, ie, the block or partial block to be requested from among the multiple available blocks indicated by the metadata given the metadata described above. A block selector 123, which performs a function to select, receives a request instruction from the block selector 123, and sends a request for a specified block, block portion, or blocks to the block serving infrastructure 101 (1 The block requestor 124, and also the block buffer 125, buffer monitor 126, media decoder 127, and media are easily used to perform the operations necessary to receive the data containing that block as a reply. One or more media transactions to It is shown the inducer 128, as having.

  Block data received by the block requestor 124 is passed to a block buffer 125 that stores media data for temporary storage. Alternatively, the received block data may be stored directly in the block buffer 125 as shown in FIG. Media decoder 127 is provided with the media data by block buffer 125 and performs the necessary transformations on this data to provide the appropriate input to media transducer 128, which can use the media for user utilization or Make it suitable for other uses. Examples of media transducers include visual display devices, such as found in cell phones, computer systems or televisions, and may also include audio presentation devices such as speakers or headphones.

  Examples of media decoders are analog or digital for video frames, such as YUV format pixel maps with associated presentation timestamps for each frame or sample, in the format described in the H.264 video coding standard. It will be a function that converts it into a presentation.

  The buffer monitor 126 receives information about the contents of the block buffer 125 and, based on this information and possibly other information, can determine the selection of the block to request, as described herein. An input is given to the block selector 123 to be used.

  In the terminology used herein, each block represents a “play time” or “duration” that represents the amount of time that the receiver will take to play the media contained in that block at normal speed. Is included. In some cases, playback of media in a block may depend on data that has already been received from the previous block. In rare cases, playback of a portion of media in a block may depend on subsequent blocks, where the playback time of the block is defined with respect to media that can be played within the block without reference to the subsequent block. The playback time of the subsequent block is increased by the playback time of the media in this block that can be played back only after the subsequent block is received. Since it is rare to include media in a block that depends on a succeeding block, the rest of this disclosure assumes that the media in one block does not depend on the succeeding block. It should be noted that those skilled in the art will recognize that they can be readily added to the described embodiments.

  The receiver may have controls such as “pause”, “fast forward”, “rewind” and the like. This may result in block consumption due to playback at different rates, but the receiver will acquire and decode each successive sequence of blocks in a total time that is less than or equal to the total playback time excluding the last block in the sequence. If possible, the receiver can present the media to the user without stalling. In some descriptions herein, a particular location in the media stream is referred to as a particular “time” in the media, which reaches a particular location in the video stream from the beginning of media playback. Corresponds to the time elapsed up to the time. Time or position in the media stream is a conventional concept. For example, if the video stream contains 24 frames per second, the first frame may be said to have a position or time of t = 0.0 seconds, and the 241st frame will have a position or time of t = 10.0 seconds. May be said to have. Of course, in a frame-based video stream, the position or time need not be contiguous, as each of the bits in the stream from the first bit of the 241st frame to just before the first bit of the 242nd frame Because they can all have the same time value.

  Adopting the above terms, a block request streaming system (BRSS) includes one or more clients that make requests to one or more content servers (eg, HTTP servers, FTP servers, etc.). The capture system includes one or more capture processors that can receive content (in real time or otherwise), process the content for use by BRSS, and access the content server Content is stored in a separate storage device, possibly with metadata generated by the capture processor.

  The BRSS may also include a content cache that cooperates with the content server. A content server and content cache may be a traditional HTTP server and HTTP cache that receives requests for files or segments in the form of HTTP requests that contain URLs, fewer than all of the files or segments indicated by the URL May also include byte ranges. Clients may include traditional HTTP clients that make HTTP server requests and process responses to such requests, where the HTTP client creates a request, passes the request to the HTTP client, and from the HTTP client. Operated by a new client system that obtains the response and processes (eg, stores, transforms, etc.) the response to provide it to the display player for playback by the client device. Typically, the client system does not know in advance what media will be needed (since the user's input will change because the need may be in response to the user's input, etc.) and is therefore received This is called a “streaming” system in that the media is “consumed” as soon as it is received or shortly after reception. As a result, response delays and bandwidth constraints can cause delays in the presentation, for example, if the stream catches up where the user is watching the presentation, the presentation is paused.

  In order to provide a display that is perceived as being of good quality, some details may be implemented in the BRSS either on the client side, on the capture side, or both. In some cases, the details that are implemented are made to account for and correspond to the client-server interface in the network. In some embodiments, both the client system and the capture system are aware of the improvement, and in other embodiments, only one side is aware of the improvement. In such cases, even if one side is not aware of the improvement, the entire system will benefit from the improvement, and in other cases it will only benefit if both sides recognize the improvement, but one side will improve. Even if you don't recognize it, the system will continue to work without breaking down.

  As shown in FIG. 3, in accordance with various embodiments, the capture system may be implemented as a combination of hardware and software components. The capture system may include a set of instructions that may be executed to cause the system to perform any one or more of the methods discussed herein. The system can be implemented as a unique machine in the form of a computer. The system may be a server computer, a personal computer (PC), or any system capable of executing a set of instructions (sequential instructions or other instructions) that define the actions to be taken by the system. Further, although only a single system is shown, the term `` system '' refers to a set (or sets) of instructions for performing any one or more of the methods discussed herein. Should be construed to include any collection of systems that execute individually or together.

  The capture system may include a capture processor 302 (e.g., a central processing unit (CPU)), a memory 304 capable of storing program code being executed, and a disk storage 306, all of which are connected to the bus 300. Communicate with each other via The system may further include a video display unit 308 (eg, a liquid crystal display (LCD) or a cathode ray tube (CRT)). The system may also include an alphanumeric input device 310 (eg, a keyboard) and a network interface device 312 for receiving and delivering content sources to content storage devices.

  The disk storage unit 306 includes a machine-readable medium on which one or more sets of instructions (e.g., software) may be stored that embodies any one or more of the methods or functions described herein. obtain. The instructions may also be fully or at least partially in memory 304 and / or in capture processor 302 during its execution by the system, and memory 304 and capture processor 302 may also be machine readable media. Configure.

  As shown in FIG. 4, according to various embodiments, a client system may be implemented as a combination of hardware and software components. A client system may include a set of instructions that may be executed to cause the system to perform any one or more of the methods discussed herein. The system can be implemented as a unique machine in the form of a computer. The system may be a server computer, a personal computer (PC), or any system capable of executing a set of instructions (sequential instructions or other instructions) that define the actions to be taken by the system. Further, although only a single system is shown, the term `` system '' refers to a set (or sets) of instructions for performing any one or more of the methods discussed herein. Should be construed to include any collection of systems that execute individually or together.

  The client system may include a client processor 402 (eg, a central processing unit (CPU)), a memory 404 capable of storing executing program code, and a disk storage 406, all of which are connected via the bus 400. Communicate with each other. The system may further include a video display unit 408 (eg, a liquid crystal display (LCD) or a cathode ray tube (CRT)). The system may also include an alphanumeric input device 410 (eg, a keyboard) and a network interface device 412 for sending requests and receiving responses.

  The disk storage unit 406 includes a machine readable medium on which one or more sets of instructions (e.g., software) may be stored that embodies any one or more of the methods or functions described herein. obtain. The instructions may also be fully or at least partially in memory 404 and / or client processor 402 during its execution by the system, and memory 404 and client processor 402 also constitute a machine-readable medium. .

Using the 3GPP file format
Any other file based on 3GPP file format or ISO based media file format such as MP4 file format or 3GPP2 file format may be used as a container format for HTTP streaming with the following features: A segment index may be included in each segment to signal the time offset and byte range so that the client can download the appropriate piece of the file or media segment as requested. The timing of the global presentation of the entire media presentation and the local timing within each 3GP file or media segment can be precisely aligned. Tracks within one 3GP file or media segment can be precisely aligned. Tracks that span multiple representations can also be aligned by assigning each of the representations to a global timeline, so that the switching of representations can be seamless and the combined presentation of media components in different representations can be synchronized.

  The file format may include a profile for adaptive streaming with the following characteristics: All movie data may be contained in a movie fragment, and the “moov” box may not contain any sample information. Audio and video sample data may be interleaved with requirements similar to those for progressive download profiles as specified in TS26.244. The “moov” box may be placed at the beginning of the file, followed by the segment index, containing time and byte range offset information for each fragment or at least a subset of the fragments in the containing segment Fragment offset data may be placed.

  It may be possible for the Media Presentation Description to refer to a file that conforms to an existing progressive download profile. In this case, the client can simply select an appropriate alternative version from a plurality of available versions using the Media Presentation Description. The client may also use an HTTP partial get request with a file that conforms to the progressive download profile to request a subset of each alternative version, thereby implementing a less efficient form of adaptive streaming. In this case, different representations including media in the progressive download profile may still maintain a common global timeline and allow seamless switching between multiple representations.

Overview of Advanced Method In the following section, a method for an improved block request streaming system is described. It should be understood that some of these improvements can be used with or without others of these improvements, depending on the needs of the application. In general operation, a receiver makes a server or other transmitter request for a particular block or portion of data. A file, also called a segment, may contain multiple blocks and is associated with one presentation of a media presentation.

  Preferably, indexing information, also referred to as “segment indexing” or “segment map”, is generated that provides a mapping from playback time or decoding time to the byte offset of the corresponding block or fragment within the segment. This segment indexing may be included within the segment, usually at the beginning of the segment (at least part of the segment map is at the beginning) and is often small. The segment index can also be provided in a separate index segment or file. In particular, if a segment index is included in a segment, the receiver can quickly download some or all of this segment map and then use it to use time offsets and their time offsets in the file Can be determined for the corresponding byte position of the fragment associated with.

  The receiver uses the byte offset to request data from a fragment associated with a specific time offset without having to download all of the data associated with other fragments that are not associated with the time offset of interest. Can do. In this way, the segment map or segment indexing can greatly improve the receiver's ability to directly access the portion of the segment associated with the current time offset of interest, improving content zapping time, There are benefits, including the ability to quickly change from one representation to another as the state changes, and the reduction in waste of network resources downloading media that has not been played at the receiver.

  When switching from one representation (referred to herein as a `` switching source '' representation) to another representation (referred to herein as a `` switching destination '' representation) is considered, playback of the switching destination representation is performed from a random access point. To ensure that seamless switching is possible in the sense that the media in the source representation is downloaded until a certain presentation time so that it can start seamlessly, the segment index is also It can be used to identify the start time of the random access point in and identify the amount of data to be requested in the switch source representation.

  These blocks represent segments of video media or other media that the requesting receiver needs to produce output for the user of the receiver. The media receiver may be a client device, such as when a receiver receives content from a server that transmits the content. Examples are set-top boxes, computers, game consoles, televisions with special equipment, handheld devices, mobile phones with special equipment, or other client receivers.

  A number of advanced buffer management methods are described herein. For example, the buffer management method allows the client to request the highest media quality block that can be received in time to be played continuously. The variable block size feature improves compression efficiency. Improved transmission performance is achieved by having multiple connections to transmit blocks to requesting devices while limiting the frequency of requests. Partially received blocks of data can be used to continue the media presentation. A connection can be reused for multiple blocks without having to first dedicate the connection to a particular set of blocks. The consistency of server selection among multiple possible servers by multiple clients is improved, which reduces the frequency of duplicate content in nearby servers and increases the probability that the server will contain the entire file. The client can request a media block based on metadata (such as available media encodings) embedded in the URL of the file containing the media block. The system can achieve calculation and minimization of the amount of buffering time required before content playback can be started without causing a subsequent pause in media playback. The available bandwidth is shared among multiple media blocks so that a larger portion of the available bandwidth can be allocated to the block with the closest playback time if necessary. It can be adjusted as the playback time approaches.

  HTTP streaming can make use of metadata. Presentation level metadata includes, for example, stream duration, available encodings (bit rate, codec, spatial resolution, frame rate, language, media type), pointers to stream metadata for each encoding, and , Content protection (digital rights management (DRM) information). The stream metadata may be a segment file URL.

  Segment metadata may include byte range versus time information for requests within a segment, and random access point (RAP) or other search point identification information, some or all of which may be segment indexing or segment maps Can be part of

  A stream may include multiple encodings of the same content. Each encoding may then be broken down into segments, each segment corresponding to a storage unit or file. In HTTP, a segment is usually a resource that can be referenced by a URL, and a request for such a URL results in the return of the segment as the entity body of the request response message. A segment may include a group of multiple pictures (GoP). Each GoP may further include a plurality of fragments, where segment indexing provides time / byte offset information for each fragment, ie, the indexing unit is a fragment.

  Fragments or portions of fragments can be requested through parallel TCP connections to improve throughput. This can alleviate problems that occur when sharing connections on bottleneck links or when connections are lost due to congestion, thus improving overall speed and reliability of delivery. Can significantly improve the speed and reliability of content zapping time. Bandwidth can be in exchange for latency due to excessive demand, but care must be taken to avoid making demands on the far future that can increase the risk of starvation.

  Multiple requests for segments on the same server can be pipelined to avoid repetitive TCP startup delays (making the next request before the current request is completed). Requests for consecutive fragments can be combined into a single request.

  Some CDNs prefer large files and can cause a background fetch of the entire file from the original server when the range request is first viewed. However, most CDNs serve range requests from the cache when data is available. Therefore, it may be advantageous to make some part of the client request for the entire segment file. These requests can be canceled later if necessary.

  A valid switching point may be a search point in the target stream, for example, specifically a RAP. Various implementations are possible, such as a fixed GoP structure or alignment of RAPs across multiple streams (based on media initiation or based on GoP).

  In one embodiment, the segments and GoP may be aligned across different rate streams. In this embodiment, the GoP may be of variable size and may include multiple fragments, but the fragments are not aligned between different rate streams.

  In some embodiments, file redundancy can be exploited advantageously. In these embodiments, an erasure code is added to each fragment to generate a redundant version of the data. Preferably, the source formatting is not changed due to the use of FEC, and an additional repair segment containing FEC repair data is generated as an additional step in the capture system, for example as a subordinate representation of the original representation. Made available. A client that is able to reconstruct a fragment using only the source data for that fragment can request only the source data for the fragment in the segment from the server. If the server is unavailable, or if the connection to the server is slow, these can be determined either before or after the request for source data, but additional repair data is not available due to fragments from the repair segment. This may be required, which reduces the time to reliably deliver enough data to recover the fragment and, in some cases, uses FEC decoding to combine the received source and repair data. Use and restore fragment source data. In addition, when a fragment becomes imminent, i.e., when its playback time is imminent, additional repair data may be required to allow the fragment to be restored, which means that for the fragment on the link Increases data occupancy, but is more efficient than disconnecting other connections on the link to free up bandwidth. This may also reduce the risk of starvation due to the use of parallel connections.

  The fragment format may be a stored stream of real-time transport protocol (RTP) packets with audio / video synchronization implemented through the real-time transport control protocol RTCP.

  The segment format may also be a stored stream of MPEG-2 TS packets with audio / video synchronization implemented through MPEG-2 TS internal timing.

Use of signaling and / or block creation to make streaming more efficient A number of features may or may not be used in a block request streaming system to provide improved performance. Performance is the ability to play presentations without stalling, obtaining media data within bandwidth constraints, and / or limited processor resources in client, server, and / or capture systems It may be related to obtaining media data within. Some of these features are described here.

In order to organize a partial GET request for an indexed movie fragment within a segment, the client is responsible for the byte offset and start time in the decoding time or presentation time of all media components contained in the file or fragment within the segment, and also which fragment May be informed of whether to start or include a random access point (and is suitable as an alternative representation switch point), this information is often referred to as segment indexing or segment map. The start time at the decoding time or the presentation time may be expressed directly or may be expressed as a delta with respect to the reference time.

  This time and byte offset indexing information may require at least 8 bytes of data per movie fragment. As an example, for a two hour movie contained in a single file with a 500 ms movie fragment, this is about 112 kilobytes of data in total. Downloading all of this data when starting a presentation can result in significant additional startup delays. However, since time and byte offset data can be encoded hierarchically, the client can quickly find a small chunk of time and offset data associated with the point in the presentation that the client wishes to begin. it can. The information may also be distributed within the segment so that some refinement of the segment index can be placed interleaved with the media data.

  If the representation is segmented into multiple segments in time, the use of this hierarchical coding may not be necessary because the complete time and offset data for each segment may already be small enough. Please keep in mind. For example, if the segment is 1 minute instead of 2 hours in the above example, the time-byte offset indexing information is on the order of 1 kilobyte of data, which can typically fit in a single TCP / IP packet.

  Different options can add fragment time and byte offset data to the 3GPP file.

  First, a Movie Fragment Random Access Box (“MFRA”) can be used for this purpose. MFRA provides a table that can help readers find random access points in a file using movie fragments. To support this function, MFRA then includes the byte offset of the MFRA box that contains the random access point. The MFRA can be placed at or near the end of the file, but this is not always the case. By scanning from the end of the file for the Movie Fragment Random Access Offset Box and using the size information in it, it may be possible to locate the beginning of the Movie Fragment Random Access Box. However, placing MFRA at the end of HTTP streaming usually involves at least 3-4 HTTP requests to access the desired data, i.e. at least one HTTP request to request MFRA from the end of the file. , One HTTP request to get the MFRA and one last HTTP request to get the desired fragment in the file. Thus, it may be desirable to place it first because mfra can be downloaded along with the first media data in a single request. Also, using MFRA for HTTP streaming can be inefficient, as information in "MFRA" is not required except for time and moof_offset, and specifies an offset instead of a length Because it may require more bits.

  Second, an Item Location Box (“ILOC”) may be used. “ILOC” identifies the directory of the metadata resource in this file or other files, the file containing the metadata resource, the offset of the metadata resource within that file, and the length of the metadata resource Provided by. For example, the system can consolidate all externally referenced metadata resources into a single file and readjust file offsets and file references accordingly. However, since “ILOC” is intended to give the location of the metadata, it may be difficult for ILOC to coexist with the actual metadata.

  The last, and sometimes the most appropriate, is a new one called Time Index Box (“TIDX”), which is specifically dedicated to the efficient provision of accurate fragment time or duration and byte offset. It is the specification of the box. This is explained in more detail in the next section. An alternative box having the same function may be a Segment Index Box (“SIDX”). As used herein, unless otherwise indicated, the two may be interchangeable, so that both boxes efficiently provide accurate fragment time or duration and byte offset. Because it provides the ability. The difference between TIDX and SIDX is given below. Since both boxes implement a segment index, it will be clear how to exchange TIDX boxes and SIDX boxes with each other.

Segment indexing A segment has a specified start time and a specified number of bytes. Multiple fragments may be concatenated into a single segment, and the client can make a request to identify a particular range of bytes within the segment corresponding to the requested fragment or subset of fragments. For example, if HTTP is used as the request protocol, the HTTP Range header may be used for this purpose. This approach requires that the client have access to the “segment index” of the segment that defines the position within the segment of the different fragments. This “segment index” may be provided as part of the metadata. This approach results in that far fewer files need to be created and managed compared to the approach where each block is stored in a separate file. Managing the creation, transfer, and storage of very large numbers of files (for example, up to thousands for a one hour presentation) can be complex and error-prone, so reducing the number of files is an advantage It becomes.

  If the client only knows the desired start time for a smaller portion of the segment, the client may request the entire file and then read the entire file to determine an appropriate playback start position. To improve bandwidth utilization, a segment may include an index file as metadata, which associates the byte range of each block with the time range that the block corresponds to, which is the segment This is called indexing or segment map. This metadata may be formatted as XML data, or may be binary, for example according to the atom and box structure of the 3GPP file format. Indexing may be simple, where the time and byte range of each block is absolute with respect to the start of the file, or the indexing may be hierarchical, with some blocks being parent Grouped into blocks (and parent blocks are grouped into grandparent blocks, etc.), and the time and byte range for a given block is relative to the time and / or byte range of the block's parent block Expressed.

Exemplary Indexing Map Structure In one embodiment, the original source data for one representation of a media stream may be included in one or more media files, referred to herein as “media segments”, with each media segment Includes media data used to play continuous time segments of media, eg, 5 minutes of media playback.

  FIG. 6 shows an exemplary overall structure of the media segment. Within each segment, there may also be indexing information including a time / byte offset segment map, either at the beginning or distributed throughout the source segment. The time / byte offset segment map of one embodiment includes time / byte offset pairs (T (0), B (0)), (T (1), B (1)), ..., (T (i), B (i)), ..., (T (n), B (n)), where T (i-1) is the media for the start time of the first media among all media segments. Represents the start time in the segment for the playback of the i-th fragment, and T (i) represents the end time of the i-th fragment (and hence the start time of the next fragment) and the byte offset B ( i-1) is the corresponding byte index at the beginning of the data in this source segment, where the i th fragment of the media starts at the beginning of the source segment and B (i) is the i th The corresponding end byte index of the fragment (and therefore the index of the first byte of the next fragment ) It is. If the segment contains multiple media components, T (i) and B (i) may be given for each component in the segment in an absolute manner, or another media that services the reference media component It may be expressed for a component.

  In this embodiment, the number of fragments in the source segment is n, and n can vary from segment to segment.

  In another embodiment, the time offset in the segment index for each fragment may be determined by the absolute start time of the first segment and the duration of each fragment. In this case, the segment index can record the start time of the first fragment and the duration of all the fragments included in the segment. A segment index may also record only a subset of the fragments. In that case, the segment index records the duration of a subsegment, defined as one or more consecutive fragments, ending at the end of the containing segment or at the beginning of the next subsegment.

  For each fragment, there is also a value that indicates whether the fragment starts at or contains the search point, i.e., the media after that point does not depend on any media prior to that point. Because it can exist, media beyond that fragment can be played independently of the previous fragment. A search point is generally a point in the media where playback can begin independently of all previous media. FIG. 6 also shows a simple example of possible segment indexing for the source segment. In that example, the time offset value is in milliseconds, so the first fragment of this source segment starts 20 seconds from the beginning of the media, and the first fragment has a playback time of 485 milliseconds. The initial byte offset of the first fragment is 0, the end of the first fragment / 2 the initial byte offset of the second fragment is 50,245, and therefore the size of the first fragment is 50,245 bytes. If a fragment or subsegment does not start with a random access point, but a random access point is included in the fragment or subsegment, a decoding time difference or presentation time difference between the start time and the actual RAP time may be given. This allows the client to know exactly the time before switching from the display has to be presented when switching to this media segment.

  In addition to or instead of simple or hierarchical indexing, daisy chain indexing and / or hybrid indexing may be used.

  Because the sample periods for different tracks may not be the same (for example, video samples may be displayed for 33ms, but audio samples may last 80ms), different tracks in a movie fragment will not start and end at exactly the same time In other words, the audio may start slightly before or slightly after the video, and the reverse holds for the preceding fragment to make up for it. To avoid ambiguity, time stamps defined in time and byte offset data may be defined for a particular track, which may be the same track for each representation. Usually this is a video track. This allows the client to accurately identify the next video frame when the client is switching representations.

  Care can be taken during the presentation to maintain a strict relationship between the timeline of the track and the presentation time and to ensure smooth playback and maintenance of audio / video synchronization despite the above issues .

  FIG. 7 shows some examples such as a simple index 700 and a hierarchical index 702.

  Two specific examples of boxes containing segment maps are given below, one called a time index box (“TIDX”) and one called (“SIDX”). This definition follows a box structure with an ISO-based media file format. Other designs of such boxes for defining similar syntax and having the same semantics and functionality will be apparent to the reader.

Time Index Box
Definition

  Box type: “tidx”

  Container: file

  Essentiality: None

  Amount: number of either 0 or 1

  The Time Index Box may provide a set of time and byte offset indexes that associate several areas of the file with several time intervals of the presentation. The Time Index Box may include a targettype field that indicates the type of data being referenced. For example, the Time Index Box with targettype “moof” provides an index to the media fragments contained in the file with respect to both time and byte offsets. A Time Index Box with a target type of Time Index Box may be used to build a hierarchical time index, allowing the user of the file to go quickly to the required part of the index.

  The segment index may include the following syntax, for example.

aligned (8) class TimeIndexBox extends FullBox ('frai') {
unsigned int (32) targettype;

unsigned int (32) time_reference_track_ID;
unsigned int (32) number_of_elements;
unsigned int (64) first_element_offset;
unsigned int (64) first_element_time;
for (i = 1; i <= number_of_elements; i ++)
{
bit (1) random_access_flag;
unsigned int (31) length;
unsigned int (32) deltaT;
}
}

  Semantics

  targettype: the type of box data referenced by this Time Index Box. This can be either a Movie Fragment Header (“moof”) or a Time Index Box (“tidx”).

  time-reference_track_id: indicates the target track for which the time offset at this index is defined.

  number_of_elements: The number of elements indexed by this Time Index Box.

  first_element_offset: The byte offset from the beginning of the file of the first indexed element.

  first_element_time: The start time of the first indexed element using the time axis specified in the Media Header box of the track identified by time_reference_track_id.

  random_access_flag: 1 if the element start time is a random access point. Otherwise it is 0.

  length: The length of the indexed element in bytes.

  deltaT: The difference on the time axis specified in the Media Header box of the track specified by time_reference_track_id between the start time of this element and the start time of the next element.

  Segment Index Box

  The Segment Index Box ("sidx") provides a compact index of movie fragments and other Segment Index Boxes in the segment. There are two loop structures in the Segment Index Box. The first loop records the first sample of the subsegment, i.e. the sample in the first movie fragment referenced by the second loop. The second loop provides the subsegment index. The container of the “sidx” box is directly a file or a segment.

  Syntax

  aligned (8) class SegmentIndexBox extends FullBox ('sidx', version, 0) {

  unsigned int (32) reference_track_ID;

  unsigned int (16) track_count;

  unsigned int (16) reference_count;

  for (i = 1; i <= track_count; i ++)

  {

  unsigned int (32) track_ID;

  if (version == 0)

  {

  unsigned int (32) decoding_time;

  } else

  {

  unsigned int (64) decoding_time;

  }

  }

  for (i = 1; i <= reference_count; i ++)

  {

  bit (1) reference_type;

  unsigned int (31) reference_offset;

  unsigned int (32) subsegment_duration;

  bit (1) contains_RAP;

  unsigned int (31) RAP_delta_time;

  }

  }

  Semantics

  reference_track_ID gives the track_ID of the reference track.

  track_count: Number of tracks (1 or more) to be indexed in the next loop.

  reference_count: The number of elements to be indexed in the second loop (1 or more).

  track_ID: The ID of the track contained in the first movie fragment identified by this index, and exactly one track_ID in this loop is equal to reference_track_ID.

  decoding_time: The track identified by the track_ID in the movie fragment referenced by the first item in the second loop, expressed in the track's time axis (as recorded in the timescale field of the track's Media Header Box) The decoding time of the first sample in

  If set to reference_type: 0, indicates that the reference is for a movie fragment (“moof”) box, and if set to 1, indicates that the reference is for a segment index (“sidx”) box. Show.

  reference_offset: Distance in bytes from the first byte after the containing Segment Index Box to the first byte of the referenced box.

  subsegment_duration: If the reference is to a Segment Index Box, this field carries the sum of the subsegment_duration field in the box's second loop, and if the reference is to a movie fragment, this field is indicated Carries the total sample duration of samples in the reference track in the movie fragment and subsequent movie fragments up to the first movie fragment recorded by the next entry in the loop or the end of the subsegment, whichever comes first However, the duration is expressed on the time axis of the track (as recorded in the timescale field of the track's Media Header Box).

  contains_RAP: If the reference is to a movie fragment, this bit may be 1 if the track fragment in that movie fragment of the track with track_ID equal to reference_track_ID contains at least one random access point, otherwise In this case, this bit is set to 0. When a reference is to a segment index, this bit is set to 1 only if any reference in that segment index has this bit set to 1, otherwise it is 0.

  When RAP_data_time: contains_RAP is 1, the presentation (composite) time of a random access point (RAP) is given, and when contains_RAP is 0, the value 0 is reserved. The time is expressed as the difference between the decoding time of the first sample of the subsegment recorded by this entry and the presentation (composite) time of the random access point in the track with track_ID equal to reference_track_ID.

Difference between TIDX and SIDX
TIDX and SIDX provide the same functionality with respect to indexing. The first loop of SIDX additionally provides global timing for the first movie fragment, but global timing can also be included in the movie fragment itself, either absolutely or relative to the reference track.

  The second loop of SIDX implements the function of TIDX. Specifically, SIDX allows to have a mixture of references for each index referenced by reference_type, while TIDX only refers to TIDX only or MOOF only. TIDX number_of_elements corresponds to SIDX reference_count, TIDX time-reference_track_id corresponds to SIDX reference_track_ID, TIDX first_element_offset corresponds to reference_offset of the first entry in the second loop, TIDX first_element_time corresponds to the first loop TIDX random_access_flag corresponds to SIDX contains_RAP, with the additional freedom that RAP_delta_time is required because TIDX does not necessarily have to be placed at the beginning of the fragment. The length of TIDX corresponds to reference_offset of SIDX, and finally, deltaT of TIDX corresponds to subsegment_duration of SIDX. Thus, the functions of the two boxes are equivalent.

Variable Block Sizing and Sub-GoP Blocks For video media, the relationship between the video coding structure and the required block structure can be important. For example, if each block starts at a search point such as a random access point ("RAP") and each block represents an equal duration of video time, the placement of at least some search points in the video media is fixed And search points occur at regular intervals in the video code. As is well known to those skilled in the art of video coding, compression efficiency is improved when search points are arranged according to the relationship between video frames, especially in frames that have little in common with previous frames. Can be improved. This requirement that the blocks represent an equal amount of time therefore imposes constraints on video coding so that compression can be suboptimal.

  Rather than requiring the search points to be at a fixed location, it is desirable to allow the placement of the search points within the video presentation to be chosen by the video encoding system. By allowing the video encoding system to choose a search point, video compression is improved, so that higher quality video media can be provided using a given available bandwidth, Improve user experience. This is a drawback of existing systems because current block request streaming systems can require that all blocks have the same duration (in terms of video time) and that each block must start at a search point. It is.

  A novel block request streaming system that provides advantages over the above will now be described. In one embodiment, the video encoding process of the first version of the video component may be configured to choose the placement of search points to optimize compression efficiency, but the length between search points With the requirement that there is a maximum value for This latter requirement restricts the selection of search points by the encoding process and thus reduces the compression efficiency. However, this reduction in compression efficiency requires that a normal fixed position is required for the search point unless the maximum value for the length between the search points is too small (eg, longer than about 1 second). Is smaller than what is caused. Furthermore, when the maximum value for the length between search points is several seconds, the reduction in compression efficiency is generally very small compared to the case where the search points are completely free.

  In many embodiments, including this embodiment, some RAPs are not search points, i.e. there may be a frame that is not a search point and is a RAP between two consecutive search points. This could be because, for example, the RAP is too close in time to the surrounding search points, or because the amount of media data between the search points before or after the RAP is too small .

  The placement of search points within all other versions of the media presentation may be constrained to be the same as the search points of the first (eg, highest media data rate) version. This reduces the compression efficiency for these other versions compared to allowing the encoder to freely select search points.

  The use of search points usually requires that the frame be independently decodable, which generally reduces the compression efficiency of that frame. Frames that are not required to be independently decodable may be encoded with respect to data in other frames, generally in an amount that depends on the amount of commonality between the frame to be encoded and the reference frame , Increase the compression efficiency of that frame. Efficient selection of search point placement is preferentially selected as search point frames for frames that have low commonality with previous frames, so compression caused by encoding the frames in an independently decodable manner Minimize efficiency loss.

  However, the level of commonality between a frame and a possible reference frame is highly correlated with different representations of content because the original content is the same. As a result, constraining search points in other deformations so that they are in the same position as the search points in the first deformation does not make a large difference in compression efficiency.

  The structure of the search points is preferably used to determine the block structure. Preferably, each search point determines the beginning of the block and there may be one or more blocks that contain data between two consecutive search points. For coding with good compression, the length between search points is not fixed, so it is not required that all blocks have the same playback period. In some embodiments, the blocks are aligned between versions of the content, i.e. if there is a block that spans a particular group of frames in one version of the content, it is in the same group of frames in another version of the content. There are blocks that span. The blocks for a given version of content do not overlap, and each frame of content is contained within exactly one block of each version.

  A variable length between search points, and thus an enabling mechanism that allows efficient use of variable length GoPs can be included in the segment or provided to the client by other means Segment indexing or segment map to get, i.e. this is the metadata associated with this segment in this presentation that can be provided, including the start time and duration of each block of the presentation. The client can use this segment indexing data when determining which block the presentation should start if the user requests that the presentation start at a particular point in the segment. If no such metadata is provided, the presentation can start only at the beginning of the content, or randomly or at an approximate point close to the desired point (e.g., the requested starting point ( You can start by picking the starting block) by dividing the (temporal) by the average block duration to give the starting block index.

  In one embodiment, each block may be provided as a separate file. In another embodiment, multiple consecutive blocks can be combined into a single file to form a segment. In this second embodiment, each version of the metadata may be provided, including the start time and duration of each block and the byte offset within the file where the block starts. This metadata may be provided in response to the initial protocol request, i.e. may be available separately from the segment or file, or as a block itself within the same file or segment, for example at the beginning of the file. May be included. As will be apparent to those skilled in the art, this metadata is in a compressed or binary format, such as gzip or delta encoding, to reduce the network resources required to transport the metadata to the client. May be encoded.

  FIG. 6 shows an example of segment indexing where the block is variable size, where the range of the block is a partial GoP, ie a partial amount of media data between one RAP and the next. In this example, the search point is indicated by a RAP indicator, a RAP indicator value of 1 indicates that the block starts at or contains a RAP or search point, and a RAP indicator of 0 indicates that the block is RAP or Indicates that the search point is not included. In this example, the first three blocks, bytes 0-157,033, contain the first GoP, which has a presentation duration of 1.623 seconds, and the presentation time is 20 to 21.623 seconds in the content Continue. In this example, the first of the first three blocks includes a presentation time of 0.485 seconds and includes the first 50,245 bytes of media data in the segment. In this example, blocks 4, 5, and 6 include the second GoP, blocks 7 and 8 include the third GoP, and blocks 9, 10, and 11 include the fourth GoP. Note that other RAPs may be present in the media data that are not specified as search points, so they are not signaled as RAPs in the segment map.

  Referring back to FIG. 6, if the client or receiver wants to access content that starts at a time offset of about 22 seconds in the media presentation, the client will resemble an MPD like that described in more detail later. Other information can first be used to first determine that the associated media data is in this segment. The client can download the first part of the segment to obtain the segment indexing, which is only a few bytes in this case, for example using an HTTP byte range request. Using segment indexing, the client can determine that the first block that the client should download is the first block that starts at the RAP with a time offset of at most 22 seconds, ie, is the search point. . In this example, block 5 has a time offset that is less than 22 seconds, i.e. the block 5 time offset is 21.965 seconds, but segment indexing indicates that block 5 does not start with a RAP, instead Based on the segment indexing, the client chooses to download block 4. That is because the start time of block 4 is at most 22 seconds, that is, the time offset of block 4 is 21.623 seconds and starts with RAP. Thus, based on segment indexing, the client makes an HTTP range request starting at byte offset 157,034.

  If segment indexing is not available, the client may have had to download all previous 157,034 bytes of data before downloading this data, much longer startup time or channel zapping time, and This leads to wasted downloads of unuseful data. Alternatively, if segment indexing is not available, the client can approximate where the desired data begins in the segment, but the estimation may be inaccurate and the client misses the appropriate time. And rework may be required, which again increases the start-up delay.

  In general, each block contains a portion of media data that can be played by the media player along with the previous block. Thus, signaling to clients in a blocking structure and segment indexing blocking structure, either contained within a segment or provided to the client through other means, provides fast channel zapping, network fluctuations and This greatly improves the client's ability to perform seamless playback in the face of disruption. Support for variable length blocks and blocks that contain only the GoP portion, as enabled by segment indexing, can greatly improve the streaming experience. For example, referring again to FIG. 6 and the above example where the client wishes to begin playback at approximately 22 seconds in the presentation, the client requests the data in block 4 through one or more requests. This can then be given to the media player as soon as playback can begin. Thus, in this example, playback begins as soon as 42,011 bytes of block 4 are received at the client, thus allowing fast channel zapping time. Instead, if the client needed to request the entire GoP before playback was about to begin, the channel zapping time would have been longer because this was 144,211 bytes of data.

  In other embodiments, the RAP or search point may also be in the middle of the block, and there may be data during segment indexing that indicates where the RAP or search point is within the block or fragment. In other embodiments, the time offset may be the decoding time of the first frame in the block instead of the presentation time of the first frame in the block.

  Figures 8 (a) and 8 (b) show examples of variable block sizing of search point structures aligned across multiple versions or representations, and Figure 8 (a) is aligned across multiple versions of a media stream. FIG. 8 (b) illustrates variable block sizing with search points that are not aligned across multiple versions of the media stream.

  The time is shown at the top in seconds and the two segment blocks and search points for the two representations are shown from left to right with respect to their timing relative to this time axis, so the length of each block shown The size is proportional to the playback time, not the number of bytes in the block. In this example, segment indexing for both segments of the two representations has the same time offset with respect to the search point, but in some cases has a different number of blocks or fragments between the search point and the search point. , Due to different amounts of media data in each block, have different byte offsets for the block. In this example, if the client wants to switch from representation 1 to representation 2 with a presentation time of about 23 seconds, the client will request up to block 1.2 in the segment for representation 1 and the segment for representation 2 starting at block 2.2. Since the request can be initiated, a switch occurs in the presentation at the same time as search point 1.2 in representation 1, which is simultaneous with search point 2.2 in representation 2.

  As is apparent from the foregoing, the block request stream system described does not constrain video coding to place search points at specific locations within the content, which is a problem with existing systems. Reduce one.

  In the embodiment described above, the search points for different representations of the same content presentation are organized. However, in many cases it is preferable to relax this alignment requirement. For example, an encoding tool that does not have the ability to generate an expression with aligned search points may be used to generate the expression. As another example, content presentations can be independently encoded into different representations without alignment of search points between the different representations. As another example, the representation may be at a lower rate and more often to be switched, or more search points to support trick modes such as fast forward or rewind or fast search. If included frequently, more search points may be included. Accordingly, it would be desirable to provide a method that allows a block request streaming system to efficiently and seamlessly process search points that are not aligned across various representations for a content presentation.

  In this embodiment, the arrangement of search points over a plurality of expressions need not be uniform. Since the blocks are constructed such that a new block starts at each search point, there may be no alignment between blocks of different representations. An example of such an unaligned search point structure between different representations is shown in FIG. 8 (b). The time is shown at the top in seconds and the two segment blocks and search points for the two representations are shown from left to right with respect to their timing relative to this time axis, so the length of each block shown The size is proportional to the playback time, not the number of bytes in the block. In this example, segment indexing for both segments of the two representations sometimes has different time offsets relative to the search point, and in some cases, a different number of blocks between the search points. Or have fragments and have different byte offsets for the blocks due to different amounts of media data in each block. In this example, if the client wants to switch from representation 1 to representation 2 at a presentation time of about 25 seconds, the client will request up to block 1.3 in the segment for representation 1 and the segment for representation 2 starting at block 2.3. Since the request can be initiated, a switch occurs in the presentation at the same time as the search point 2.3 in representation 2 during playback of block 1.3 in representation 1, and thus the media for block 1.2 Some are not played (but media data for frames in block 1.3 that are not played may have to be loaded into a receiver buffer to decode other frames in block 1.3 that are played).

  In this embodiment, whenever it is requested to select a block from a different representation than the previously selected version, its first frame is a frame that follows the last frame of the last selected block. The operation of the block selector 123 can be modified so that the latest, not later, block is selected.

  This last described embodiment can remove the requirement of constraining the placement of search points into versions other than the first version, thus increasing the compression efficiency for these versions and the given available Provides higher quality presentations for bandwidth and improved user experience. A further consideration is that video encoding tools that perform the function of alignment of search points across multiple encodings (versions) of content may not be widely available and are therefore discussed at this end. An advantage of this embodiment is that currently available video encoding tools can be used. Another advantage is that the encoding of different versions of content can proceed in parallel without requiring any coordination between the encoding processes for the different versions. Another advantage is that additional versions of content can be encoded and added to the presentation at a later point in time without having to provide the encoding tool with a list of specific search point locations.

  In general, if a picture is encoded as a group of pictures (GoP), the first picture in the sequence may be a search point, but this need not always be the case.

Optimal block partitioning One issue of concern in block request streaming systems is the interaction between the structure of the encoded media, eg video media, and the block structure used for the block request. As is well known to those skilled in video coding, the number of bits required for the coded representation of each video frame often varies considerably from frame to frame in some cases. . As a result, the relationship between the amount of data received and the duration of the media encoded by that data may not be simple. Furthermore, the division of media data into blocks within the block request streaming system further increases the complexity dimension. Specifically, in some systems, media data in a block may not be played until the entire block is received, for example, the composition of media data in the block, or media in a block that uses an erasure code Dependencies between samples can provide this property. As a result of these complex interactions of block size and block duration and the potential need to receive the entire block before starting playback, the media data is buffered before playback begins Generally, a client system adopts a conservative method. Such buffering results in long channel zapping times and thus a bad user experience.

  Pakzad describes a new and efficient way to determine how to partition a data stream into continuous blocks based on the structure behind the data stream, the “block partitioning method” And further describes some of the advantages of these methods in the context of streaming systems. Further embodiments of the present invention will now be described for applying Pakzad's block partitioning method to block request streaming systems. This method ensures that the playback time of any given element of media data (e.g., a video frame or audio sample) differs from the playback time of any adjacent media data element by less than a given threshold. , Arranging the media data to be presented in a general presentation time order. Such ordered media data may be considered as a data stream in Pakzad wording, and any of the Pakzad methods applied to this data stream identifies block boundaries by the data stream. Data between any pair of adjacent block boundaries is considered a “block” in the language of this disclosure, and the disclosed method is intended to provide a presentation of media data in a block request streaming system. Applied. As will be apparent to those skilled in the art having read this disclosure, several advantages of the method disclosed in Pakzad can then be realized in the context of a block request streaming system.

  As described in Pakzad, the determination of the block structure of a segment, including blocks that contain partial GoPs or more parts than GoP, affects the client's ability to enable fast channel zapping time Can give. In Pakzad, when a target start time is given, the client starts to download the expression at an arbitrary search point, and when playback starts after the target start time elapses, at each point in time, the client Provide a block structure and target download rate to ensure that playback continues seamless as long as the amount of downloaded data is at least the product of the target download rate and the time elapsed since the start of the download . Advantageously, the client has access to the target start time and target download rate, which is a means for determining when this will start playing the representation at the earliest point in time. Because it allows the client to continue playing the representation as long as the download satisfies the conditions described above. Thus, the methods described later can be used for the purposes described above as they provide a means for including the target start time and target download rate in the Media Presentation Description.

Media Presentation Data Model Figure 5 shows the content storage device shown in Figure 1, including segments and media presentation description ("MPD") files, as well as the breakdown, timing, and other structure of the segments in the MPD file The resulting structure is shown. Details of possible implementations of MPD structures or files will now be described. In many examples, the MPD is described as a file, but non-file structures can also be used.

  As shown in the figure, the content storage device 110 holds a plurality of source segments 510, an MPD 500, and a repair segment 512. The MPD may include a period record 501, which in turn may include a representation record 502 that includes segment information 503, such as a reference to an initialization segment 504 and a media segment 505.

  FIG. 9 (a) shows an exemplary metadata table 900, while FIG. 9 (b) shows how HTTP streaming client 902 obtains metadata table 900 and media block 904 through a connection to HTTP streaming server 906. Here is an example of how to do it.

  In the method described herein, a “Media Presentation Description” is provided that includes information about the representation of the media presentation that is available to the client. The representation may be an alternative in the sense that the client chooses one of the different alternatives, or the representation is from a set of alternatives, each possibly by the client, May be complementary in the sense that some of the representations are selected and presented together. Expressions may advantageously be assigned to groups, and clients may be programmed or configured to understand that the expressions in one group are each alternative to each other, while different groups The expression from is such that two or more expressions are presented together. In other words, if there are two or more expressions in a group, the client selects one expression from the group, selects one expression from the next group, etc. to form one expression.

  The information describing the representation advantageously includes details of the applied media codec, including the codec profile and level required to decode the representation, video frame rate, video resolution, and data rate. obtain. A client receiving a Media Presentation Description can use this information to determine in advance whether the representation is suitable for decoding or presentation. If the information to be distinguished is only included in the binary data of the representation, it is necessary to request binary data from all representations and analyze and extract the relevant information to find information about eligibility, The above is an advantage. Analysis and extraction of these multiple requests and data can take some time, which results in long start-up times and thus a bad user experience.

  In addition, the Media Presentation Description may include information that restricts client requests based on time. For example, for a live service, the client may be constrained to the requested portion of the presentation that is close to the “current broadcast time”. This is an advantage because in live broadcasts it may be desirable to remove data from the serving infrastructure for content that was broadcast beyond a given threshold prior to the current broadcast time. . This may be desirable for storage resource reuse within the serving infrastructure. This may also be desirable depending on the type of service provided, for example, in some cases the presentation is made available only live due to some subscription model of the receiving client device. While other media presentations may be made available live and on demand, other presentations are only live for first class client devices and second class clients It can be made available on demand only for devices and in a combination of either live or on demand for a third class of client devices. The method described in the media presentation data model (below) is such that clients can avoid making requests to users and adjusting offers for data that may not be available in the serving infrastructure. Allows the client to be informed of the above policy. As an alternative, for example, the client may present a notification to the user that this data is not available.

  In a further embodiment of the invention, the media segment is in an ISO-based media file format as described in ISO / IEC 14496-12 or in a derived specification (such as the 3GP file format described in 3GPP Technical Specification 26.244). Can be compliant. The Usage of 3GPP File Format section (above) describes a new improvement to the ISO-based media file format that allows efficient use of the data structure of this file format within a block request streaming system. As described in this reference, information can be provided in the file that allows a fast and efficient association between the time segment of the media presentation and the byte range in the file. The media data itself can be structured according to the construction of movie fragments as defined in ISO / IEC 14496-12. This information providing time and byte offsets can be structured hierarchically or as a single block of information. This information can be given at the beginning of the file. Preparing this information using efficient encoding as described in the Usage of 3GPP File Format section is useful when the file download protocol used by the block request streaming system is HTTP, for example HTTP partial. A GET request is used to allow the client to quickly retrieve the above information, which shortens startup, search, or stream switching times and thus provides an improved user experience.

  The representations in the media presentation are synchronized on a global timeline to ensure a seamless presentation across multiple representations, which are usually alternatives, and to ensure a synchronized presentation of two or more representations. Thus, the timing of media samples included in a representation within an adaptive HTTP streaming media presentation may be related to a continuous global timeline across multiple segments.

  Multiple types of media, eg, blocks of encoded media, including audio and video, may have different presentation end times for different types of media. In a block request streaming system, such media blocks can be played continuously in such a way that each media type is played continuously, so one type of media sample from one block It may be played before a media sample of a type of preceding block, referred to herein as a “continuous block join”. Alternatively, such media blocks may be played in such a way that the earliest sample of any type in one block is played after the earliest sample of any type in the preceding block, This is referred to herein as a “discontinuous block junction”. Continuous block concatenation may be appropriate if both blocks contain media from the same content item and the same representation, encoded in sequence, or otherwise. Usually, within one representation, continuous block joining can be applied when joining two blocks. This is advantageous because existing coding can be applied and segmentation can be performed without having to align the media tracks at the block boundaries. This is illustrated in FIG. 10, where the video stream 1000 includes block 1202 and other blocks with a RAP, such as RAP 1204.

Media Presentation Description
A media presentation can be viewed as a structured collection of files on an HTTP streaming server. The HTTP streaming client can download enough information to present the streaming service to the user. An alternative representation is one or more 3GP files that conform to the 3GPP file format, or at least conform to a well-defined set of data structures that can be easily converted to or from 3GP files. Or it can consist of part of a 3GP file.

  A media presentation can be described by a media presentation description. The Media Presentation Description (MPD) may include metadata that can be used by the client to construct an appropriate file request, eg, an HTTP GET request, to access the data at the appropriate time and provide a streaming service to the user. The media presentation description can provide enough information for the HTTP streaming client to select the appropriate 3GPP file and file fragment. The unit that is signaled to the client to be accessible is called a segment.

  The media presentation description can include, among other things, the following elements and attributes:

MediaPresentationDescription element An element that encapsulates metadata used by an HTTP streaming client to provide a streaming service to an end user. The MediaPresentationDescription element may include one or more of the following attributes and elements:

  Version: Number of protocol versions to ensure extensibility.

  PresentationIdentifier: Information that allows a presentation to be uniquely identified from other presentations. It may also contain private fields or names.

  UpdateFrequency: Update frequency of media presentation description. That is, how often the client can reload the actual media presentation description. If not present, the media presentation can be immutable. Updating the media presentation may mean that the media presentation cannot be cached.

  MediaPresentationDescriptionURI: URI to date the media presentation description.

  Stream: Describes the type of stream or media presentation: video, audio, or text. The video stream type may include audio and text.

  Service: Describes the service type with additional attributes. Service types can be live and on-demand. This can be used to inform the client that searching and access beyond some current time is not allowed.

  MaximumClientPreBufferTime: The maximum amount of time that the client can pre-buffer the media stream. This timing can differentiate streaming from progressive download if the client is constrained to download beyond this maximum pre-buffering time. There may not be a value indicating that constraints on pre-buffering cannot be applied.

  SafetyGuardIntervalLiveService: Information about the live service's maximum turnaround time on the server. Provide the client with an indication of which information is already accessible at the current time. This information may be necessary if the client and server are expected to operate in UTC time and are not strictly time synchronized.

  TimeShiftBufferDepth: Information about how far the live service can be moved back to the client for the current time. This extension of depth may allow time-shifted viewing and catch-up services without specific changes in service provisioning.

  LocalCachingPermitted: This flag indicates whether the HTTP client can cache the data locally after the downloaded data has been played.

  LivePresentationInterval: Contains the time interval during which the representation can be used by specifying StartTime and EndTime. StartTime indicates the start time of the service, and EndTime indicates the end time of the service. If EndTime is not specified, the end time is currently unknown, and UpdateFrequency may ensure that the client gets access to the end time before the actual end time of the service.

  OnDemandAvailabilityInterval: The presentation interval indicates the availability of the service on the network. Multiple presentation intervals may be provided. HTTP clients may be unable to access services outside any defined time frame. With provision of OnDemand Interval, additional time-shifted viewing can be defined. This attribute may also be present for live services. If present for a live service, the server can access the service as an on-demand service during all given available intervals. Therefore, LivePresentationInterval cannot overlap with any OnDemandAvailabilityInterval.

  MPDFileInfoDynamic: Describes the default dynamic structure of a file during media presentation. Further details are given below. The default specification for the MPD level may eliminate unnecessary repetition if the same rules are used for some or all alternative representations.

  MPDCodecDescription: Describes the main default codec during media presentation. Further details are given below. The default specification for the MPD level may eliminate unnecessary repetition if the same codec is used for some or all representations.

  MPDMoveBoxHeaderSizeDoesNotChange: A flag that indicates whether the MoveBox Header changes size between individual files in the entire media presentation. This flag may be used to optimize downloads and may only be present in certain segment formats, particularly if the segment includes a moov header.

  FileURIPattern: A pattern used by clients to generate request messages for files in a media presentation. Different attributes allow the generation of a unique URI for each of the files in the media presentation. The base URI can be an HTTP URI.

Alternative Representation: Describes a list of expressions.
Alternative Representation elements

  An XML element that encapsulates all metadata for a single representation. Alternative Representation elements may include the following attributes and elements:

  RepresentationID: A unique ID for this particular Alternative Representation within the media presentation.

  FilesInfoStatic: Provides an explicit list of all file start times and URIs in one alternative presentation. Immutable provisioning of a list of files may provide the advantage of describing the exact timing of a media presentation, but may not be as small, especially when alternative representations contain many files. The file name can have any name.

  FilesInfoDynamic: provides an implicit way to build a list of alternative presentation start times and URIs. Dynamic provisioning of a list of files can provide the advantage of a smaller representation. If only the start time order is given, the timing advantage remains the same here, but the file name will be built dynamically based on the FilePatternURI. If only the duration of each segment is given, the representation is small and may be suitable for use within a live service, but file generation may be governed by global timing.

  APMoveBoxHeaderSizeDoesNotChange: A flag that indicates whether the MoveBox Header changes size between individual files in the Alternative Description. This flag may be used to optimize downloads and may only be present in certain segment formats, particularly if the segment includes a moov header.

  APCodecDescription: Describes the main codec of the file in the alternative presentation.

Media Description elements
MediaDescription: An element that can encapsulate all metadata for the media contained in this representation. Specifically, it may include information about the recommended groupings of tracks, if possible, along with information about the tracks in this alternative presentation. The attributes of MediaDescription include the following attributes.

  TrackDescription: An XML attribute that encapsulates all metadata for the media contained in this representation. The attributes of TrackDescription include the following attributes.

  TrackID: The unique ID of the track in the alternative representation. This can be used when the track is part of a grouping description.

  Bitrate: The bit rate of the track.

  TrackCodecDescription: An XML attribute that contains all the attributes for the codec used in this track. The attributes of TrackCodecDescription include the following attributes.

  MediaName: An attribute that defines the media type. Media types include “audio”, “video”, “text”, “application”, and “message”.

  Codec: Codec type including profile and level.

  LanguageTag: Language tag when applicable.

  MaxWidth, MaxHeight: The height and width of the included video in pixels relative to the video.

  SamplingRate: Sampling rate for audio.

  GroupDescription: An attribute that provides recommendations to clients for proper grouping based on different parameters.

  GroupType: A type that allows the client to decide how to group tracks based on it.

  The information in the media presentation description is advantageously used by HTTP streaming clients to execute requests for files / segments or parts thereof at the appropriate time, eg access bandwidth, display capabilities, codec capabilities And so on, with respect to user preferences such as language, the segment is selected from an appropriate expression that matches its ability. In addition, since the Media Presentation description describes an expression that is aligned and mapped in time to a global timeline, the client can also initiate an ongoing media presentation to initiate the appropriate action to switch the expression. During that time, the information in the MPD can be used to present the representation together or explore the media presentation.

The signaling segment start time representation may be divided into multiple segments in time. There is an inter-track timing problem between the last fragment of a segment and the next fragment of the next segment. In addition, there is another timing issue when a constant duration segment is used.

  Using the same duration for each segment may have the advantage that the MPD is small and unchanged. However, each segment can still start with a random access point. Therefore, video encoding may be constrained to give random access points at these particular points, or the actual segment duration may not be exactly as specified in the MPD There is. The second option may be preferred because it may be desirable for the streaming system not to impose unnecessary constraints on the video encoding process.

  Specifically, if the file duration is defined in the MPD as d seconds, the nth file may start at a random access point at or immediately after time (n−1) d.

  In this approach, each file may contain information about the exact start time of the segment in units of global presentation time. Three possible ways to signal this include:

  (1) First, the start time of each segment is limited to the exact timing specified in the MPD. However, in doing so, the media encoder may not have any flexibility with regard to the placement of IDR frames and may require special encoding for file streaming.

  (2) Second, add a strict start time to the MPD for each segment. In the case of on-demand, the compactness of the MPD can be reduced. In the live case, this may require regular MPD updates, which can reduce scalability.

  (3) Third, add to the segment a global or exact start time relative to the announced start time of the expression or the announced start time of the segment in the MPD, in the sense that the segment contains this information. This can be added to a new box dedicated to adaptive streaming. This box may also contain information as provided by the “TIDX” or “SIDX” box. The result of this third approach is that when looking for a specific location near the beginning of one of the segments, the client can choose a subsequent segment of the segment containing the requested search point based on the MPD It is. A simple response in this case may be to move the search point before the start of the retrieved segment (ie, the next random access point after the search point). Random access points are usually provided at least every few seconds (and there is often little coding benefit from making random access points rarer), so even in the worst case the search points are more than specified. Can be moved after a few seconds. Alternatively, when retrieving the header information for a segment, the client can determine that the requested search point is actually in the previous segment and request that segment instead. This can sometimes increase the time required to perform the search operation.

List of accessible segments The media presentation includes a set of representations that each provide some different version of the encoding for the original media content. Advantageously, the representation itself contains information about the differential parameter of the representation when compared to other parameters. The representation also includes a list of accessible segments, either explicitly or implicitly.

  Segments can be distinguished in timeless segments that contain only metadata and media segments that contain primarily media data. The Media Presentation Description (“MPD”) advantageously identifies and assigns different attributes to each of the segments, either implicitly or explicitly. Attributes that are advantageously assigned to each segment include the resources and protocols through which the segment is accessible during the period that the segment is accessible. In addition, the media segment is advantageously assigned attributes such as the start time of the segment during the media presentation, the duration of the segment during the media presentation.

  If the media presentation is of an “on-demand” type, as advantageously indicated in an attribute in the media presentation description, such as OnDemandAvailabilityInterval, the media presentation description typically describes the entire segment and if the segment is accessible and Provide an indication of when the segment is not accessible. The start time of the segment is advantageously expressed relative to the start of the media presentation so that two clients that start playing the same media presentation at different times can use the same media segment with the same media presentation description. This advantageously improves the ability to cache segments.

  If the media presentation is of a “live” type, as advantageously indicated by an attribute in the media presentation description, such as the attribute Service, a segment that contains a media presentation that exceeds the actual time is generally fully described in the MPD. Is not generated or at least not accessible. However, by indicating that the media presentation service is of the “live” type, the client can access the segment with timing attributes for the real-time client internal time NOW based on the information contained in the MPD and the MPD download time. A list of can be generated. The server advantageously operates in the sense that the server makes the resource accessible so that a reference client operating on the MPD instant in real time NOW can access the resource.

  Specifically, the reference client generates a list of accessible segments together with timing attributes for real-time client internal time NOW based on information contained in the MPD and MPD download time. As time progresses, the client uses the same MPD to create a new accessible segment list that can be used to continuously play the media presentation. Thus, the server can announce the segments in the MPD before these segments are actually accessible. This is advantageous because it reduces frequent updates and downloads of the MPD.

  Assume that a list of segments each with a start time tS is described explicitly by a playlist in an element like FileInfoStatic or implicitly by using an element like FileInfoDynamic. An advantageous method for generating a segment list using FileInfoDynamic is described below. Based on this construction rule, the client has access to a list of URIs for each representation r, referred to herein as FileURI (r, i), and the start time tS (r, i) for each segment with index i. I have the right.

  Using the information in the MPD to create an accessible time frame for the segment can be performed using the following rules.

  For services of type “on demand”, as advantageously indicated by attributes such as Service, any current available real-time NOW at the client is available, as advantageously represented by an MPD element such as OnDemandAvailabilityInterval. If within range, all described segments of this on-demand presentation are accessible. If the current real-time NOW at the client is outside any available range, none of the described segments of this on-demand presentation is accessible.

  As advantageously indicated by attributes such as Service, for a “live” type service, the start time tS (r, i) advantageously represents the time available in real time. The available start time may be derived as a combination of the event's live service time and any turnaround time at the server for capture, encoding and publishing. For example, the time for this process may be defined in the MPD using, for example, a safety guard interval tG defined as SafetyGuardIntervalLiveService in the MPD. This minimizes the difference between UTC time and the availability of data on the HTTP streaming server. In another embodiment, the MPD explicitly defines the available time of a segment in the MPD as a difference between the live time of the event and the turnaround time without providing a turnaround time. In the following description, it is assumed that any global time is defined as available time. Those skilled in live media broadcasting can read this description and then derive this information from the appropriate information in the media presentation description.

  If the current real-time NOW at the client is outside any range of the presentation interval, as advantageously represented by an MPD element such as LivePresentationInterval, none of the described segments of this live presentation are accessible . If the current real-time NOW at the client is within the live presentation interval, at least some of the described segments of this live presentation may be accessible.

  Accessible segment constraints are governed by the following values:

  Real-time NOW (as available to clients).

  For example, the allowed time shift buffer depth tTSB specified as TimeShiftBufferDepth in the media presentation description.

The client at the relative event time t _{1 is} in the interval from (NOW-tTSB) to NOW, or includes the end time of the segment with duration d (NOW-tTSB-d) to NOW May be allowed only to request a segment with a start time tS (r, i).

Updating the MPD In some embodiments, the server may be, for example, when the server location changes, or when the media presentation includes some advertisement from a different server, or when the duration of the media presentation is unknown, or the server Does not know in advance the locator of the file or segment and the start time of the segment when it wants to obscure the locator of the next segment.

  In such an embodiment, the server may only describe segments that are already accessible or that will be accessible immediately after this instance of the MPD is published. Further, in some embodiments, the client advantageously consumes media that is close to the media described in the MPD so that the user experiences the included media program as soon as possible from the generation of the media content. . As soon as the client expects to reach the end of the described media segment in the MPD, the client advantageously requests a new instance of the MPD and the server publishes a new MPD describing the new media segment Continued playback based on the prediction that The server advantageously creates a new instance of the MPD and updates the MPD so that the client can use the procedure for continuous updates. The server can adapt the MPD update procedure with the generation of the segments, and the reference client procedure acting as a common client can operate.

  If a new instance of MPD only describes a short time ahead, the client needs to request a new instance of MPD frequently. This can lead to scalability issues and unnecessary uplink and downlink traffic due to unnecessary frequent requests.

  Thus, on the one hand, you can describe future segments as far as possible without making the segments still accessible, and on the other hand, you can represent new server locations and insert new content such as advertisements It makes sense to allow unpredictable updates of the MPD to provide or change codec parameters.

  Further, in some embodiments, the duration of the media segment may be short, such as in the range of a few seconds. Media segment duration is advantageously optimal for delivery or caching characteristics to compensate for end-to-end delays in live services, or other aspects of segment storage or delivery, or for other reasons It is flexible to be adjusted to an appropriate segment size that can be optimized. A large amount of media segment resources and start times need to be described in the media presentation description, especially if the segment is small compared to the media presentation duration. As a result, the size of the media presentation description can be large, which can adversely affect the download time of the media presentation description, which also affects the media presentation start delay and bandwidth usage on the access link Can give. Therefore, it is advantageous not only to allow the description of a list of media segments using playlists, but also to allow description by using templates or URL construction rules. Template and URL construction rules are used interchangeably in this description.

  In addition, the template can advantageously be used to describe segment locators that exceed the current time when live. In such a case, the MPD update itself is not necessary as a locator, and the segment list is described by a template. However, unforeseen events may still occur that require changes to the representation or segment description. When content from multiple different resources are spliced together, for example, when an advertisement is inserted, a change to the adaptive HTTP streaming media presentation description may be required. Content from different sources can differ in various ways. Another reason is that during a live presentation, the URL used for the content file may need to be changed to achieve failover from one original live server to another.

  In some embodiments, when the MPD is updated, it is advantageous that an update to the MPD is performed so that the updated MPD matches the previous MPD in the following sense. The above meaning is that the reference client, and thus any implemented client, would have generated from a previous instance of the MPD from an MPD that was updated at any time up to the validity time of the previous MPD This means that a list of accessible segments having the same function is generated. This requirement is (a) adapted to the old MPD before the update time, so that the client can start immediately using the new MPD without synchronizing with the old MPD, and (b) to the MPD Ensure that the time when the actual change occurs and the update time do not need to be synchronized. In other words, updates to the MPD may be announced in advance, and the server can replace the old instance of the MPD as new information becomes available without having to maintain a different version of the MPD.

  There may be two possibilities for media timing across MPD updates for a set of representations or all representations. (a) the existing global timeline follows the MPD update (referred to herein as `` continuous MPD update ''), or (b) the current timeline ends and the segment after the change Either a new time axis starts (referred to herein as “discontinuous MPD update”).

  The difference between these options may be obvious considering that the tracks of media fragments, and thus of segments, generally do not start and end simultaneously due to different sample granularities between tracks. During a normal presentation, samples from one track of a fragment can be rendered before some samples from another track of the previous fragment. That is, there may be no overlap within a single track, but there is some overlap between fragments.

  The difference between (a) and (b) is whether such duplication may be possible across MPD updates. If the MPD update is due to the joining of completely separate content, such duplication is generally realized because the new content requires a new encoding to be joined with the previous content. difficult. Therefore, it is advantageous to provide the ability for discontinuous updates of the media presentation by resuming the time axis for several segments and possibly also define a new set of representations after the update. . Also, if the content is independently encoded and segmented, it is also possible to avoid adjusting the time stamp so that it falls within the global time axis of the previous fragment of the content.

  Duplicate and continuous updates if the update is for a smaller reason, such as simply adding a new media segment to the list of described media segments, or if the URL location changes Can be allowed.

  For discontinuous MPD updates, the time axis of the last segment of the previous representation ends at the latest presentation end time for any sample in the segment. The timeline of the next representation (or more precisely, the first presentation time of the first media segment of the new part of the media presentation, also called the new time period) is normal, and advantageously, seamless and continuous playback Start at this moment, the same as the end of the last period presentation, to ensure.

  Two cases are shown in FIG.

  It is preferred and advantageous to constrain MPD updates to segment boundaries. The basis for constraining such changes or updates to segment boundaries is as follows. First, changes to binary metadata for each representation, usually Movie Header, can occur at least at segment boundaries. Second, the Media Presentation Description may include a pointer (URL) to the segment. In a sense, MPD is a “comprehensive” data structure that groups together all segment files associated with a media presentation. To maintain this containment relationship, each segment may be referenced by a single MPD, and when the MPD is updated, the MPD is advantageously updated only at segment boundaries.

  Segment boundaries generally do not need to be aligned, but for content spliced from different sources, and for discontinuous MPD updates, generally align segment boundaries (specifically, the last segment of each representation It makes sense that it may end in the same video frame and not include audio samples with a presentation start time later than that frame's presentation time). Discontinuous updates can then start a new set of representations at a common moment called a period. The start time at which this new set of representations becomes valid is given, for example, by the period start time. The relative start time of each representation is reset to 0, and the start time of the period places the set of representations in this new period on the global media presentation timeline.

  For continuous MPD updates, segment boundaries need not be aligned. Each segment of each alternative representation can be dominated by a single Media Presentation Description, so a new instance of Media Presentation Description is commonly caused by the prediction that no additional media segments will be described in the working MPD Update requests for can occur at different times depending on the consumed set of expressions, including the set of expressions that are expected to be consumed.

  In order to support updating the elements and attributes of the MPD in the more general case, any element, not just an expression or set of expressions, can be associated with a valid time. Thus, if some elements of the MPD need to be updated, for example, if the number of expressions changes, or if the URL construction rules change, these elements are each elements with unconnected lifetimes Can be updated individually at defined times by providing multiple copies of

  Effectiveness is advantageously associated with global media time so that the described elements associated with valid time are valid during the global time axis of the media presentation.

  As discussed above, in one embodiment, the validity time is added only to the complete set of representations. Each complete set then forms one period. And the effective time forms the start time of the period. In other words, in the specific case of using valid elements, the complete set of representations may be valid for a period of time, as indicated by the global valid time for the set of representations. The effective time of a set of expressions is called a duration. At the beginning of a new period, the validity of the previous set of expressions expires and the new set of expressions becomes valid. Note again that the validity period is preferably not linked.

  As mentioned above, changes to the Media Presentation Description occur at the segment boundary, so for each representation, changes to the element actually occur at the next segment boundary. The client can then form a valid MPD that includes a list of segments for each moment of time within the presentation time of the media.

  Discontinuous block joins may be appropriate when the blocks contain media data from different representations or from different content, eg, content segments and advertisements, or otherwise. It may be required in a block request streaming system that changes to presentation metadata occur only at block boundaries. This can be advantageous for implementation reasons, as updating the media decoder parameters within a block can be more complex than updating them only between blocks. In this case, the element is considered valid from the first block boundary not earlier than the start of the specified effective interval to the first block boundary not earlier than the end of the specified effective interval. It can be advantageously defined that the described effective interval can be interpreted as an approximation.

  The above exemplary embodiments describe a new improvement to the block request streaming system described in the section presented later, titled Changes to Media Presentation.

Segment Duration Signaling Discontinuous updates effectively divide the presentation into a series of unconnected intervals called periods. Each period has a unique time axis for media sample timing. The media timing of representations within a period can be advantageously indicated by defining a separate small list of segment durations for each period or for each representation during a period.

  For example, an attribute associated with an element in the MPD, called the period start time, can define the validity time of some elements within the media presentation time. This attribute can be added to any element of the MPD (attributes that can be assigned validity can be changed to elements).

  In discontinuous MPD update, all representation segments may end at discontinuities. This generally suggests at least that the last segment before the discontinuity has a different duration than the previous segment. Segment duration signaling may involve indicating that all segments have the same duration, or indicating a separate duration for each segment. It may be desirable to have a compact representation for a list of segment durations, which is efficient when many of the segment durations have the same duration.

  The duration of each segment in a representation or set of representations is advantageously a single interval from the start of the discontinuous update, i.e. the beginning of the period to the last media segment described in the MPD. Can be executed by a single string that defines all segment durations for. In one embodiment, the format of this element is a text string that conforms to a product containing a list of segment duration entries, where each entry has a duration attribute dur and an optional multiplier for the attribute mult. Including the first entry segment <mult> with the first entry duration <dur>, the second entry duration <dur> with the second entry segment <mult>, etc. Show.

  Each duration entry defines the duration of one or more segments. If the <dur> value is followed by the letter “*” and a number, this number defines the number of consecutive segments with this duration in seconds. If there is no multiplication symbol “*”, the number of segments is one. If “*” is present without a number that follows, all subsequent segments have a defined duration and there may be no further entries in the list. For example, the string “30 *” means that all segments have a duration of 30 seconds. The string “30 * 12 10.5” indicates that 12 segments with a duration of 30 seconds are followed by one segment with a duration of 10.5 seconds.

  If the segment duration is defined separately for each alternative representation, the total segment duration in each interval may be the same for each representation. In the case of a video track, the interval may end with the same frame in each alternative representation.

  Those of ordinary skill in the art, after reading this disclosure, may find similar or equivalent methods for compactly representing segment durations.

  In another embodiment, the duration of a segment is signaled by a single duration attribute <duration> as being constant for all segments in the representation except the last one. As long as the start point of the next discrete update or the start of a new period is provided, this implies that the duration of the last segment reaches the start of the next period, and the last before the discrete update The duration of the segments may be shorter.

Changes and updates to representation metadata
Indicating a change in binary-coded representation metadata, such as a change in Movie Header “moov”, can be accomplished in different ways. (A) There can be one moov box for all representations in a separate file referenced in the MPD, and (b) each in a separate file referenced in each Alternative Representation. There can be one moov box for alternative representations, (c) each segment can contain a moov box, so it is self-contained, (d) moov boxes for all representations in one 3GP file with MPD There can be.

  In the case of (a) and (b), a single “moov” advantageously means that more “moov” boxes can be referenced in the MPD unless the effective times of the “moov” boxes are connected. Can be combined with the concept of effectiveness described above. For example, by defining a period boundary, the validity of “moov” in the old period may expire due to the start of a new period.

  For option (a), a reference to a single moov box can be assigned a valid element. Multiple Presentation headers can be allowed, but only one can be valid at a time. In another embodiment, the effective time of the entire set of representations during a period, or the effective time of the entire period as defined above, is usually the effective time for this expression metadata provided as a moov header. Can be used.

  For option (b), a reference to the moov box of each representation can be assigned a valid element. Multiple Representation headers may be allowed, but only one may be valid at a time. In another embodiment, the lifetime of the entire representation, or the lifetime of the entire duration as defined above, can be used as the lifetime for this representation metadata, usually provided as a moov header.

  For option (c), signaling in the MPD may not be added, but additional signaling is added in the media stream to indicate whether the moov box will change for any of the upcoming segments. obtain. This is further described below in “Signaling of Updates in Segment Metadata”.

Signaling of updates in the segment metadata It is advantageous to signal any such updates with the media segment to avoid frequent updates of the media presentation description to gain knowledge about possible updates It is. An additional one that may indicate that updated metadata, such as media presentation description, is available and must be accessed within a certain amount of time in order to continue creating an accessible segment list Or multiple elements may be provided within the media segment itself. In addition, such elements can provide a file identifier, such as a URL, or information that can be used to construct a file identifier for the updated metadata file. The updated metadata file may include metadata that is equivalent to the metadata provided in the original metadata file for the presentation that has been modified to show the validity interval, with additional metadata that is also accompanied by a validity interval. Such an indication may be provided in the media segment of all available representations for the media presentation. Upon detecting such an indication in the media block, a client accessing the block request streaming system can retrieve the updated metadata file using a file download protocol or other means. Thus, the client is provided with information about the change in the media presentation description and information about when the change occurred or occurred. Advantageously, each client has an updated media presentation description only once when such a change occurs, rather than “polling” and receiving the file many times for possible updates or changes. Request.

  Examples of changes include changes to one or more representations, such as adding or removing representations, bit rate, resolution, aspect ratio, changes to included track or codec parameters, and changes to URL construction rules, such as advertising Including different origin servers. Some changes may only affect initialization segments, such as the Movie Header (“moov”) atom associated with the representation, while other changes may affect the Media Presentation Description (MPD).

  For on-demand content, these changes and timing may be known in advance and may be signaled in the Media Presentation Description.

  In live content, the change may not be known to the point where the change occurs. One solution is to allow the Media Presentation Description available at a specific URL to be dynamically updated and ask the client to periodically request this MPD to detect changes. That is. This solution has drawbacks in terms of scalability (original server load and cache efficiency). In situations where there are a large number of viewers, the cache makes many requests to the MPD after the previous version is no longer in the cache and before the new version is received and all of these can be transferred to the originating server. Can be received. The origin server may need to always process requests from the cache for each updated version of the MPD. Also, updates may not be easily aligned in time with media presentation changes.

  Since one of the benefits of HTTP streaming is the ability to utilize standard web infrastructure and services for scalability, the preferred solution involves only “immutable” (ie, cacheable) files. It does not rely on the client to “polling” the file to see if it has changed.

  Solutions for resolving metadata updates, including media presentation descriptions and binary representation metadata such as “moov” atoms in adaptive HTTP streaming media presentations, are discussed and proposed.

  For live content, it may not be known that the MPD or “moov” can change when the MPD is built. MPD frequent “polling” to check for updates should generally be avoided for bandwidth and scalability reasons, so updates to the MPD may be indicated as “in-band” in the segment file itself. That is, each media segment may have an option to indicate an update. Depending on the segment format (a) to (c) above, different updates may be signaled.

  In general, in a signal within a segment, the MPD is updated before requesting the following indication: the next segment in this representation, or any next segment that has a start time greater than the start time of the current segment. An obtaining indicator may be advantageously provided. The update may be announced in advance, indicating that the update should only occur in any segment that is later than the next segment. This MPD update can also be used to update binary representation metadata such as the Movie Header if the locator of the media segment is changed. Another signal may indicate that when this segment is complete, no more segments should be required to advance time.

  If the segment is formatted according to the segment format (c), i.e. each media segment may contain self-initializing metadata such as a movie header, then the subsequent segment shall contain an updated Movie Header (moov) Further additional signals can be added as shown. This advantageously allows the movie header to be included in the segment, but the Movie Header is when the previous segment indicates a Movie Header update, or when searching or random access when switching representations Only needs to be requested by the client. In other cases, the client can advantageously place a byte range request on the segment that excludes the movie header from the download, thus advantageously saving bandwidth.

  In yet another embodiment, when an MPD update indication is signaled, the signal may also include a locator such as a URL for the updated Media Presentation Description. The updated MPD can describe the representation both before and after the update, using valid attributes such as the new period and the old period in the case of discontinuous updates. This can advantageously be used to allow time-shifted viewing as described further below, but advantageously the MPD update can take place at any time before the changes that the MPD includes become effective. Can also be signaled. The client can immediately download a new MPD and apply it to the ongoing presentation.

  In certain implementations, any change to the media presentation description, moov header, or end-of-presentation signaling is formatted according to the segment format rules using the ISO-based media file format box structure, a streaming information box Can be included. This box may provide a unique signal for any of the different updates.

  Streaming Information Box

  Definition

Box type: "snif"
Container: None Required: None Quantity: 0 or 1

  The Streaming Information Box contains information about the streaming presentation that the file is part of.

  Syntax

aligned (8) class StreamingInformationBox
extends FullBox ('sinf') {
unsigned int (32) streaming_information_flags;

/// The following are optional fields
string mpd_location
}

  Semantics

  The streaming_information_flags includes zero or more of the following logical sums.

  0x00000001 Movie Header update follows

  0x00000002 Presentation Description update

  0x00000004 End presentation

  The mpd_location exists only when the Presentation Description update flag is set, and provides a Uniform Resource Locator for a new Media Presentation Description.

Example Use Case for MPD Updates for Live Services Assuming that a service provider wants to provide a live football event using the improved block request streaming described herein To do. In some cases, millions of users may wish to access the presentation of the event. Live events are sporadically interrupted by breaks when an interruption is declared or when other operations are paused, during which advertisements can be added. Usually there is no or little prior notification of the exact timing of the break.

  Service providers may need to provide redundant infrastructure (eg, encoders and servers) to allow for seamless switching in case of any failure of equipment during a live event.

  Assume that the user's Anna accesses the service from the bus with his mobile device and the service is immediately available. Sitting next to her is another user, Paul, who is watching the event on his laptop computer. The goal is recorded and both of them celebrate the event at the same time. Paul tells Anna that the first goal of the match was even more exciting, and Anna uses the service to see the event 30 minutes in time. After seeing the goal, she returns to the live event.

  To deal with its use cases, the service provider updates the MPD, signals the client that an updated MPD is available, and the client provides a streaming service so that the client can present data in near real time. It must be possible to allow access to

  The MPD update can be performed in a manner that is not synchronized with the distribution of the segments, as described elsewhere herein. The server can guarantee to the receiver that the MPD is not updated for some time. The server may use the current MPD. However, if the MPD is updated before some minimum update period, explicit signaling is not required.

  Fully synchronized playback is hardly achieved because the client may be operating on different MPD update instances and therefore the client may have drift. Using MPD updates, the server can communicate the changes and the client can be alerted of the changes even during the presentation. Since in-band signaling per segment can be used to indicate MPD updates, updates can be limited to segment boundaries, but it should be acceptable in many applications.

  An MPD element may be added that provides a real time publishing time of the MPD and an optional MPD update box that is added at the beginning of the segment to signal that an MPD update is required. Updates can be done hierarchically, like MPD.

  The MPD “Public Time” provides a unique identifier for the MPD and when the MPD was issued. It also provides an anchor for the update procedure.

  The MPD update box is found in the MPD after the “styp” box and may be defined by Box Type = “mupe”, does not require a container, is not required, and has an amount of 0 or 1. The MPD update box contains information about the media presentation that the segment is part of.

  An exemplary syntax is as follows.

  aligned (8) class MPDUpdateBox

  extends FullBox ('mupe') {

  unsigned int (3) mpd information flags;

  unsigned int (l) new-location flag;

  unsigned int (28) latest_mpd_update time;

  /// The following are optional fields

  string mpd_location

  }

  The semantics of various objects of class MPDUpdateBox can be as follows:

mpd_information_flags: OR of zero or more of the following
0x00 Media Presentation Description now updated

0x01 Media Presentation Description forward update
0x02 End of presentation

  0x03-0x07 Reserved

  When new_location flag is set to 1, a new Media Presentation Description can be used at a new location specified by mpd_location.

  latest_mpd_update time: Specifies the time (in ms) until MPD update is required for the MPD issue time of the latest MPD. The client may choose to update the MPD at any time between now.

  mpd_location exists only when new_location_flag is set, and when it exists, provides a Uniform Resource Locator for a new Media Presentation Description.

  If the bandwidth used by the update is an issue, the server can provide the MPD for some device capability so that only some of these are updated.

Timeshift viewing and network PVR
If time-shifted viewing is supported, it can happen that more than one MPD or Movie Header is valid for the duration of the session. In this case, a valid MPD can exist throughout the time frame by updating the MPD when needed, but by adding the concept of validity mechanism or duration. This means that the server can ensure that any MPD and Movie headers are announced for any period that is within a valid time frame for time-shifted viewing. It is up to the client to ensure that the available MPD and metadata for the current presentation are valid. Migration of live sessions to network PVR sessions using only small MPD updates may also be supported.

Special media segment
The problem when the ISO / IEC 14496-12 file format is used in a block request streaming system is that the media data for a single version of the presentation in multiple files, as described above, is continuous. It may be advantageous to store the data in a time segment. Furthermore, it may be advantageous to arrange each file to start with a random access point. Further, the location of the search point is selected during the video encoding process, and the presentation is segmented into a plurality of files, each starting at the search point, based on the search point selection made during the encoding process. It may be advantageous that each random access point may or may not be located at the beginning of the file, but each file starts with a random access point. In one embodiment with the characteristics described above, the presentation metadata, or Media Presentation Description, may include the exact duration of each file, such as the start time of the video media of the file and the following: It is understood to mean the difference in the start time of the video media of the file. Based on this information in the presentation metadata, the client can construct an association between a global timeline for media presentation and a local timeline for media in each file.

  In another embodiment, the size of the presentation metadata can be advantageously reduced instead by prescribing that each file or segment has the same duration. However, in this case, and if the media file is built according to the above method, the random access point may not exist at a point that is exactly at the specified duration from the start of the file, so the duration of each file May not be exactly equal to the duration specified in the media presentation description.

  In spite of the differences noted above, further embodiments of the present invention for realizing normal operation of a block request streaming system will now be described. In this method, the local time axis of the media in the file (the time axis starting at time stamp 0, for which the time stamp for decoding and combining the media samples in the file is defined according to ISO / IEC 14496-12, An element that defines the mapping of the intended) to the global presentation timeline may be provided in each file. This mapping information may include a single time stamp at the global presentation time, corresponding to time stamp 0 on the local file timeline. The mapping information was alternatively provided in the presentation metadata as the difference between the global presentation time corresponding to time stamp 0 on the local file timeline and the global presentation time corresponding to the start of the file. It may include an offset value that is defined according to the information.

  Examples of such boxes may be, for example, a track fragment decoding time (“tfdt”) box or a track fragment adjustment box (“tfad”) along with a track fragment media adjustment (“tfma”) box.

Exemplary Client Including Segment List Generation An exemplary client will now be described. This can be used as a reference client for the server to ensure proper generation and update of the MPD.

  HTTP streaming clients are guided by information provided in the MPD. The client is assumed to have access to the MPD received by the client at time T, ie, the time when it was possible to successfully receive the MPD. Determining that the reception was successful may include the client obtaining an updated MPD or the client confirming that the MPD has not been updated since the previous successful reception.

  An example client behavior is introduced. To provide a continuous streaming service to the user, the client first parses the MPD, possibly using playlists, or using URL construction rules, and the account segment list as described below Consider the generation procedure, create a list of accessible segments for each representation for client local time at the current system time. The client then selects one or more representations based on information in the representation attributes and other information, eg, available bandwidth and client capabilities. Depending on the grouping, the representation may be presented alone or together with other representations.

  For each representation, the client obtains binary metadata, such as a “moov” header for the representation, and the media segment of the selected representation, if any. A client accesses media content by requesting a segment or segment byte range, possibly using a segment list. The client can initially buffer the media before starting the presentation, and once the presentation starts, the client can continuously request a segment or part of a segment considering the MPD update procedure, Continue to consume media content.

  The client can switch the representation in view of updated MPD information and / or updated information from the environment, eg, changes in available bandwidth. Any request for a media segment that includes a random access point can cause the client to switch to a different representation. Moving forward, ie, the current system time (called “NOW time” to represent the time for the presentation), the client consumes an accessible segment. With each progression of NOW time, the client expands the list of accessible segments for each presentation, possibly in accordance with the procedures defined herein.

  Threshold at which the client predicts that the end of the media presentation has not yet been reached and the current playback time will run out of the media of the media described in the MPD for any consumed or consumed presentation In the case of entering, the client can request the MPD update by the new fetch time reception time T. Once received, the client then considers the possibly updated MPD and a new time T in generating the accessible segment list. FIG. 29 shows the procedure for live service at different times on the client.

Generate accessible segment list
Assume that an HTTP streaming client has access to the MPD and may wish to generate a segment list accessible to real-time NOW. Clients are synchronized to a global time base with some accuracy, but advantageously no direct synchronization to the HTTP streaming server is required.

  The accessible segment list for each representation is preferably defined as a list of segment start times and segment locator pairs, which can be defined as for the start of the representation without loss of generality. If this concept is applied, the start of the expression may be aligned with the start of the period. Otherwise, the start of the presentation can be at the beginning of the media presentation.

  The client uses URL construction rules and timing, for example, as further defined herein. Once the list of described segments is obtained, this list is further constrained to accessible segments, which may be a subset of the segments of the complete media presentation. The construction is governed by the current value of the clock at the client's NOW time. In general, a segment is only available for any time NOW in the set of available times. Segments are not available for time NOW outside this window. In addition, the live service assumes that some time checktime provides information about how far into the future the media will be described. The checktime is defined on the media time axis recorded by the MPD. When the client playback time reaches checktime, the client advantageously requests a new MPD.

  The segment list then further shows that the available media segment is the sum of the start time of the media segment and the start time of the expression, NOW minus timeShiftBufferDepth minus the last described segment duration and either checktime or NOW It is constrained by checktime together with the MPD attribute TimeShiftBufferDepth so that only the media segment falls within the interval with the other value.

Scalable blocks In some cases, one or more blocks that are currently received at the receiver can be used so that it is unlikely that they will be completely received in time to play without pausing the presentation Less bandwidth. The receiver can detect such a situation in advance. For example, the receiver receives a block that encodes 5 units of media every 6 unit hours and may determine that it has a buffer of 4 units of media, so after about 24 unit hours, the receiver It can be expected that the presentation must be stalled or paused. With sufficient notification, the receiver reacts to such a situation, for example, by stopping the current stream of blocks, and one or more blocks from different representations of content, for example, a smaller amount per unit playback time. May start requesting blocks that use the bandwidth of. For example, if the receiver switches to a block encoded representation of at least 20% longer video time for the same size block, the receiver will need to stall until the bandwidth situation improves. It may be possible to eliminate.

  However, it may be wasteful to have the receiver completely discard data that has already been received from the canceled representation. In the block streaming system embodiments described herein, the data in each block is used by several prefixes of the data in the block to continue the presentation without the rest of the block being received. Can be encoded and configured as is possible. For example, the well-known technique of scalable video coding can be used. Examples of such video encoding methods include temporal scalability of H.264 Scalable Video Coding (SVC) or H.264 Advanced Video Coding (AVC). Advantageously, this method allows the presentation to be part of the block being received, even if reception of one or more blocks may be interrupted, for example due to changes in available bandwidth. Allows to continue based on. Another advantage is that a single data file can be used as a source for multiple different representations of content. This is possible, for example, by utilizing an HTTP partial GET request that selects a subset of blocks corresponding to the requested representation.

  One improvement detailed herein is an improved segment, scalable segment map. A scalable segment map includes the location of different layers in the segment so that the client can access portions of the segment and extract the layers accordingly. In another embodiment, the media data in the segments is ordered so that the quality of the segments increases when data is downloaded gradually from the beginning of the segments. In another embodiment, a gradual increase in quality is applied to each block or fragment included in the segment so that the fragment request can be made to correspond to a scalable approach.

  FIG. 12 is a diagram illustrating an aspect of a scalable block. In the figure, the transmitter 1200 outputs metadata 1202, scalable layer 1 (1204), scalable layer 2 (1206), and scalable layer 3 (1208), and the one written behind is delayed. Receiver 1210 can then present media presentation 1212 using metadata 1202, scalable layer 1 (1204), and scalable layer 2 (1206).

Independent scalability layer As described above, if the receiver is unable to receive the requested block of a particular representation of media data in time for playback, it must stall Failure to do so is undesirable for block request streaming systems because it results in a bad user experience. Stalls are avoided by constraining the data rate of the chosen representation to be much less than the available bandwidth so that there is almost no chance that any given part of the presentation will not be received in time Although this strategy can be reduced, reduced or alleviated, this strategy has the disadvantage that the media quality is necessarily much lower than what can be supported in principle by the available bandwidth. A lower quality presentation than is possible can be interpreted as a bad user experience. Thus, block request streaming system designers have a data rate that is much lower than the available bandwidth that users can suffer from poor media quality in designing client procedures, client programming, or hardware configurations. Requesting a content version, or requesting a content version with a data rate close to the available bandwidth, where the user can experience a high probability of pauses during the presentation as the available bandwidth changes Either face the choice.

  In order to handle such situations, the block streaming system described herein provides multiple receivers so that receivers can make layered requests and transmitters can respond to layered requests. The scalability layers may be configured to be processed independently.

  In such embodiments, the encoded media data for each block may be partitioned into a plurality of unconnected fragments, referred to herein as “layers”, so that the layer combination includes the entire media data for the block. A client that receives several subsets of layers can perform decoding and presentation of the representation of the content. In this method, the order of data in the stream is such that the continuous range improves in quality, and the metadata reflects this.

  An example of a technique that can be used to generate a layer with the above properties is the technique of Scalable Video Coding, as described, for example, in ITU-T Standards H.264 / SVC. Another example of a technique that may be used to generate a layer with the above properties is a temporal scalability layer technique, such as provided in ITU-T Standards H.264 / SVC.

  In these embodiments, the metadata may be provided in the MPD or in the segment itself, which means that individual layers and / or combinations of layers and / or multiple blocks in any given block Allows construction of requirements for a given layer and / or a combination of layers of multiple blocks. For example, layers containing blocks may be stored in a single file, and metadata defining byte ranges in the file corresponding to individual layers may be provided.

  A file download protocol capable of defining byte ranges, eg HTTP 1.1, can be used to request individual layers or multiple layers. Further, as will be apparent to those skilled in the art who have reviewed the present disclosure, the techniques described above with respect to variable size block construction, requirements, and downloads, and variable combinations of blocks, may also be applied in this context. .

To achieve improved user experience and / or reduced serving infrastructure capability requirements compared to existing techniques through the use of media data partitioned into layers as described above Several embodiments are described herein that can be advantageously utilized by block request streaming clients.

  In the first embodiment, known techniques of a block request streaming system may be applied with modifications that in some cases different versions of content are replaced by different combinations of layers. That is, if an existing system can provide two separate representations of content, the improved system described herein can provide two layers, and one representation of content in an existing system can be If the first layer in the improved system is similar in terms of bit rate, quality and possibly other measures, the second representation of the content in the existing system is the two layers in the improved system. The same is true for the bit rate, quality, and possibly other measures. As a result, the storage capacity required in the improved system is reduced compared to that required in existing systems. In addition, existing system clients can issue requests for one representation or block of other representations, but improved system clients can request for either the first layer of the block or both layers. Can be issued. As a result, the user experience on the two systems is similar. Furthermore, improved caching is achieved because a common segment that is cached with higher probability is used for different qualities.

  In a second embodiment, a client in an improved block request streaming system that utilizes the layer method described herein maintains a separate data buffer for each of several layers of media encoding. obtain. As will be apparent to those skilled in the art of data management within a client device, these “separate” buffers can be obtained by allocating physically or logically separate memory areas to separate buffers or by buffered data. By other techniques in which data is stored in a single or multiple memory areas, and the separation of data from different layers is logically achieved through the use of data structures that include references to storage locations of data from separate layers In the following, it should be understood that the term “separate buffer” includes any way in which data of separate layers can be identified separately, as may be implemented. The client makes a request for an individual layer of each block based on the occupancy of each buffer; for example, a layer occupies any buffer for a lower priority layer when a request for data from one layer It can be ordered in order of preference so that it cannot be issued if the rate is lower than the threshold for that lower layer. In this way, the priority is lower priority so that only the lower layer is required if the available bandwidth falls below the required bandwidth to receive the higher priority layer. Given for receiving data from a layer. Further, the thresholds associated with different layers may be different, for example, so that the lower layer has a higher threshold. If the available bandwidth changes so that data for the higher layer cannot be received before the playback time of the block, the data for the lower layer will inevitably already be received, The presentation can continue on the lower layer alone. The buffer occupancy threshold may be defined in terms of bytes of data, the playback duration of data contained in the buffer, the number of blocks, or any other suitable measure.

  In the third embodiment, the method of the first and second embodiments is such that a plurality of media presentations each including a subset of layers is provided (as in the first embodiment) and the second Can be combined such that the embodiments apply to a subset of the layers in the representation.

  In the fourth embodiment, the method of the first, second, and / or third embodiment includes, for example, that at least one of the independent representations is the first, second, and / or third The embodiments can be combined with embodiments in which multiple independent representations of content are provided to include multiple layers to which the techniques of the embodiments are applied.

Advanced Buffer Manager In combination with the buffer monitor 126 (see FIG. 2), an advanced buffer manager can be used to optimize the client-side buffer. A block request streaming system may wish to ensure that media playback can start quickly and continue smoothly, while providing maximum media quality to the user or destination device simultaneously. This requires the client to request a block that has the highest media quality but can be quickly started to be played without forcing the presentation to be paused and then received in time. It can be.

  In embodiments that use an evolved buffer manager, the manager determines which blocks of media data are requested and when to make those requests. For example, an evolved buffer manager may be given a set of metadata for the content to be presented, which metadata includes a representation available for the content and a list of metadata for each representation. Metadata for the representation affects the data rate and other parameters of the representation, such as video, audio, or other codec and codec parameters, video resolution, decoding complexity, audio language, and representation choice at the client May include information about any other parameters that may provide

  The metadata for the representation may also include identifiers of previous blocks that the representation has been segmented into, and these identifiers provide the information needed for the client to request the block. For example, if the request protocol is HTTP, the identifier may be an HTTL URL, possibly with additional information identifying the byte range or time span in the file identified by the URL, The time span identifies a particular block within the file identified by the URL.

  In one particular implementation, the evolved buffer manager may determine when the receiver will make a request for a new block and itself handle the transmission of the request. In a new aspect, the evolved buffer manager makes a request for a new block according to a balancing ratio value that balances using too much bandwidth with running out of media during streaming playback.

Information received by the buffer monitor 126 from the block buffer 125 includes an indication of each event, when the media data is received, how much is received, when playback of the media data is started and stopped, And the speed of media playback. Based on this information, the buffer monitor 126 may calculate a variable B _current that represents the current buffer size. In these examples, B _current , represents the amount of media contained in one or more buffers of the client or other device, and one or more if no additional or partial blocks have been received such a will try time to all media playback represented by block or partially block is stored in the buffer B _current, as represented, B _current, can be evaluated in units of time. Therefore, B _current , which has not been reproduced yet, represents the “reproduction duration” of the media data available at the client at the normal reproduction speed.

As time passes, the value of B _current , decreases as the media is played and may increase each time new data for the block is received. In this description, it is assumed that a block is received when the entire data for that block is available at block requestor 124, but instead other measures may be considered, for example, to consider partial block reception. Note that it can be used. In practice, reception of a block can occur over a period of time.

FIG. 13 shows the variation over time of the value of B _current , as the media is played and the block is received. As shown in FIG. 13, the value of B _current is 0 at a time less than t ₀ , indicating that no data is received. In t _0, the first block is received, the value of B _current is increased to be equal to the playback duration of the received block. At this point, since the reproduction is not yet started, the value of B _current remains unchanged up to t _1, at t _1, the second block arrives, B _current is the size of the second block Increase by minutes. At this point, reproduction is started, the value of B _current begins to decrease linearly until t _2, at t ₂ is the third block arrives.

The progression of B _current continues in this “sawtooth” fashion, increasing in steps (at t ₂ , t ₃ , t ₄ , t ₅ and t ₆ ) each time a block is received, In between, it decreases smoothly as data is played back. In this example, playback proceeds at the normal playback rate for the content, so the slope of the curve during block reception is exactly -1 and 1 second of media data is taken for each elapsed 1 second of real time. It means to be played in between. When frame-based media is played for a given number of frames per second, for example 24 frames per second, a slope of -1 is a small step function that indicates the playback of each individual frame of data, for example each frame plays When approximated by a step of -1/24 of a second.

FIG. 14 shows another example of the development of B _current over time. In that example, the first block reaches t ₀ and playback starts immediately. Until time t ₃ when the value of B _current reaches zero, reaching and reproducing of the block is followed. When this happens, no additional media data is available for playback, and the media presentation is forced to pause. At time t _4, the received fourth block, playback may resume. Thus, this example shows a case where receiving the fourth block is slower than desired, resulting in a pause in playback and thus a bad user experience. Thus, the goal of the evolved buffer manager and other features is to simultaneously maintain high media quality while reducing the probability of this event.

The buffer monitor 126 can also calculate another measure B _ratio (t), which is the ratio of media received in a given period to the length of the period. More specifically, B _ratio (t) is equal to T _received / (T _now -t), where T _received is from any time t earlier than the current time to the current time T _now. The amount of media received (evaluated by its playback time).

B _ratio (t) can be used to evaluate the rate of change of B _current . B _ratio (t) = 0 is when no data has been received from time t, and assuming that the media is playing, B _current will decrease by (T _now -t) from that time. Let's go. B _ratio (t) = 1 is when the media is received for the time (T _now -t) by the same amount as the media is being played, and B _current is at time t _{now at} time t Will have the same value as the value. B _ratio (t)> 1 is an example in which more data than the data necessary for playback is received during time (T _now -t), and B _current is from time t to time T _now To increase.

The buffer monitor 126 further calculates the value State, which can take the number of individual values. Buffer monitor 126 further includes a function _{NewState (B current, B ratio)} , which is t <T when the value of the current value and B _ratio of B _current is given to _now, outputs a new State value As offered. _Whenever B _current and B _ratio cause this function to return a value different from the current value of State, a new value is assigned to State, and this new State value is indicated to block selector 123.

The function NewState may be evaluated by reference to the space of all possible values of the pair (B _current , B _ratio (T _now -T _x )), where T _x is a fixed (set) value it may be at, or, for example, may be derived by a configuration table that maps from the value of B _current to a value of T _x from B _current, or may depend on the previous value of State. The buffer monitor 126 is given one or more partitions of this space, each partition containing a set of unconnected regions, each region being annotated with a State value. The evaluation of the function NewState then includes the act of identifying the partition and determining the region into which the pair (B _current , B _ratio (T _now −T _x )) will fall. The return value is a note associated with the area. In simple cases, only one segment is given. In more complex cases, the partition may depend on the pair at the time of the previous evaluation of the NewState function (B _current , B _ratio (T _now −T _x )) or other factors.

In certain embodiments, the partition described above may be based on a configuration table that includes multiple B _current thresholds and multiple B _ratio thresholds. Specifically, the threshold of B _current is set as B _thresh (0) = 0, B _thresh (1), ..., B _thresh (n ₁ ), B _thresh (n ₁ +1) = ∞, where n ₁ Is the number of non-zero thresholds for B _current . The threshold of B _ratio is B _r-thresh (0) = 0, B _r-thresh (1), ..., B _r-thresh (n ₂ ), B _r-thresh (n ₂ +1) = ∞, where N ₂ is the number of threshold values of B _ratio . These thresholds define a partition containing a grid of (n ₁ +1) vs. (n ₂ +1) of cells, where the i th cell in the j th row is B _thresh (i-1) <= B _current <B _thresh (i) and B _r-thresh (j-1) <= B _ratio <B _r-thresh (j). Each cell of the grid described above is annotated with a state value, for example by being associated with a particular value stored in memory, and the function NewState has the values B _current and B _ratio (T _now -T _x Returns the state value associated with the cell indicated by).

In further embodiments, a hysteresis value may be associated with each threshold. In this improved method, the evaluation of the function NewState may be based on a temporal partition constructed using a set of temporally modified thresholds as follows: For each B _current threshold that is smaller than the range of B _current corresponding to the cell selected in the last evaluation of NewState, the threshold is reduced by subtracting the hysteresis value associated with that threshold. For each _Bcurrent threshold that is greater than the range of _Bcurrent corresponding to the cell selected in the last evaluation of NewState, the threshold is increased by converting the hysteresis value associated with that threshold. For each B _ratio threshold smaller than the range of B _ratio corresponding to the cell selected in the last evaluation of NewState, the threshold is reduced by subtracting the hysteresis value associated with that threshold. For each B _ratio threshold that is greater than the B _ratio range corresponding to the cell selected in the last evaluation of NewState, the threshold is increased by adding the hysteresis value associated with that threshold. The modified threshold is used to evaluate the value of NewState, and then the threshold is returned to the original value.

Other ways of defining the spatial divisions will be apparent to those skilled in the art having read this disclosure. For example, to define a half-space within the entire space and define each unconnected set as the intersection of many such half-spaces, the partition is an inequality based on a linear combination of B _ratio and B _current For example, for real values α0, α1, and α2, it may be defined by the use of a linear inequality threshold of the form α1 · B _ratio + α2 · B _current ≦ α0.

  The above description describes the basic process. An efficient implementation is possible, as will be apparent to those skilled in the art of real-time programming after reading this disclosure. For example, each time new information is provided to the buffer monitor 126, it is possible to calculate the future time when NewState transitions to a new value, for example if no further data for the block is received. The timer is then set to this time and if there is no further input, a new State value is sent to the block selector 123 upon expiration of this timer. As a result, calculations need not be performed continuously, but only when new information is provided to the buffer monitor 126 or when the timer expires.

  Appropriate values for State are “Low”, “Stable”, and “Full”. An example of a suitable set of thresholds and the resulting cell grid is shown in FIG.

In FIG. 15, the threshold of B _current is shown on the horizontal axis in milliseconds with a hysteresis value shown below as “+/− value”. The threshold for B _ratio is shown on the vertical axis in per-mil units (ie, multiplied by 1000), with the hysteresis value shown below as “+/− value”. State values are annotated to the grid cells as “L”, “S”, and “F” for “Low”, “Stable”, and “Full”, respectively.

  The block selector 123 receives a notification from the block requester 124 whenever there is an opportunity to request a new block. As explained above, the block selector 123 is provided with information regarding the available blocks and metadata for those blocks, including, for example, information about the media data rate of each block.

  Information about the media data rate of a block can be the actual media data rate of a particular block (i.e., the block size in bytes divided by the playback time in seconds), the average media data rate of the representation to which the block belongs, or , May include a measure of the available bandwidth needed to continuously play the representation to which the block belongs without pause, or a combination of the above.

  The block selector 123 selects a block based on the State value last indicated by the buffer monitor 126. If this State value is “Stable”, the block selector 123 selects a block from the same representation as the previously selected block. The selected block is the first block (in the order of playback) that contains media data for a period of time in a representation for which no media data has ever been requested.

  If the State value is “Low”, the block selector 123 selects a block from an expression that has a lower media data rate than the previously selected block. A number of factors can affect the exact choice of representation in this case. For example, the block selector 123 may be given an indication of the integration rate of incoming data and can select an expression with a media data rate lower than that value.

  If the State value is “Full”, the block selector 123 selects a block from an expression that has a higher media data rate than the previously selected block. A number of factors can affect the exact choice of representation in this case. For example, the block selector 123 may be given an indication of the rate of integration of incoming data and can select an expression with a media data rate not higher than that value.

  A number of additional factors can further affect the operation of the block selector 123. Specifically, even if the buffer monitor 126 continues to indicate the “Full” state, the frequency with which the media data rate of the selected block increases can be limited. In addition, block selector 123 receives an indication of a “Full” state, but a block with a higher media data rate is not available (for example, the last selected block is already the highest media data rate available). It may be). In this case, the block selector 123 can delay the selection of the next block by the selected time so that the total amount of media data buffered in the block buffer 125 is suppressed from above.

  Additional factors that are considered during the selection process can affect the set of blocks. For example, the available blocks may be limited to blocks from representations where the encoding resolution falls within a particular range given to the block selector 123.

  Block selector 123 may also receive input from other components that monitor other aspects of the system, such as the availability of computational resources for media decoding. When such resources become scarce, the block selector 123 determines that the block whose decoding is indicated as being lower in terms of computational complexity in the metadata (e.g., a representation with a lower resolution or frame rate generally has a lower decoding complexity). Lower).

The embodiment described above shows that the use of the value B _ratio in the evaluation of the NewState function in the buffer monitor 126 is a faster improvement in quality at the start of the presentation compared to a method that only considers B _current. Can be a significant benefit. Without considering the B _ratio , a large amount of buffered data can be accumulated before the system can select a block with a higher media data rate and hence higher quality. However, a large B _ratio value means that the available bandwidth is much larger than the media data rate of the previously received block and even if the buffered data is relatively small ( That is, a small value for B _current ) indicates that it is still safe to request a block with a higher media data rate and hence higher quality. Similarly, a low B _ratio value (<1, e.g.) means that the available bandwidth has fallen below the media data rate of the previously requested block and, therefore, the B _current is high. Also indicates, for example, that the system switches to a lower media data rate and thus lower quality in order to avoid reaching a point where B _current = 0 and stalling media playback. This improved behavior can be particularly important in environments where network conditions and thus the delivery rate can change very quickly and dynamically, for example, a user's streaming to a mobile device.

Another advantage is provided by using configuration data to define a spatial partition of the value of (B _current , B _ratio ). Such configuration data may be provided to the buffer monitor 126 as part of the presentation metadata or by other dynamic means. In actual deployments, the behavior of user network connections can vary greatly over time between users and even for a single user, so predicting a segment that works well for all users is Can be difficult. The ability to dynamically provide such configuration information to the user allows good configuration settings to evolve over time according to accumulated experience.

Variable request sizing If each request is for a single block and each block encodes a short media segment, frequent requests may be required. If the media block is short, video playback moves faster from block to block, which gives the receiver more frequent opportunities to adjust or change the selected data rate by changing the representation. Giving and increasing the probability that regeneration can continue without stalling. However, the drawback of frequent requests is that the capacity of certain networks where the available bandwidth to the server network is constrained at the client, such as the data link capacity from the client to the network, is limited, or Wireless WAN networks, such as 3G and 4G wireless WANs, that can become limited for short or long periods due to changes in radio conditions, may be unsustainable.

  High frequency demand also means high load on the serving infrastructure, which results in associated costs for capacity requirements. It is therefore desirable to have some of the advantages of high frequency requirements without all of the disadvantages.

  In some embodiments of block streaming systems, the flexibility of high frequency requests is combined with lower frequency requests. In this embodiment, the blocks may be constructed as described above and integrated into a segment that includes multiple blocks, also as described above. At the beginning of the presentation, each request references a single block, or multiple simultaneous requests are made to request a portion of a block, the process described above has a fast channel zapping time, And therefore applied to ensure a good user experience at the beginning of the presentation. Subsequently, if certain conditions described below are met, the client can make a request that includes multiple blocks in a single request. This is possible because blocks are integrated into a larger file or segment and can be requested using a range of bytes or time. Consecutive byte or time ranges may be combined into a single larger byte or time range, resulting in a single request for multiple blocks, and discontinuous blocks are also required in a single request. obtain.

  One basic configuration that can be done by determining whether to request a single block (or partial block) or multiple consecutive blocks is that the requested block is played back It has a configuration based on determination of whether or not there is a high possibility. For example, it is more preferable that the client make a request for a single block, i.e. a small amount of media data, if it is likely that it will soon need to be changed to another representation. One reason for this is that if a request for multiple blocks is made when switching to another representation may be imminent, the switch occurs before the last few blocks of the request are replayed. Because you get. Thus, the download of these last few blocks may delay the delivery of the media data in the destination representation, which can cause media playback stalls.

  However, requests for a single block result in frequent requests. On the other hand, if it is unlikely that you will need to change to a different representation soon, it is likely that all of these blocks will be played, so it may be preferable to make a request for multiple blocks. Can result in infrequent requests and can significantly reduce request overhead, especially when there are no immediate changes in the representation.

  In traditional block integration systems, the amount required for each request is not dynamically adjusted, i.e., usually each request is for the entire file, or each request is a file of approximately the same amount of representation (if Is evaluated in time, and in some cases is evaluated in bytes). Thus, if all requests are smaller, the request overhead is large, while if all requests are larger, this increases the probability of a media stall event and / or expresses faster as network conditions change. If a lower quality representation is chosen to avoid the need to change the, it results in lower quality media playback.

An example of a condition that, when satisfied, can cause subsequent requests to reference multiple blocks is the threshold for the buffer size _Bcurrent . If B _current is below the threshold, each issued request references a single block. If B _current is greater than or equal to the threshold, each issued request references multiple blocks. If a request is made that references multiple blocks, the number of blocks required in each single request can be determined in one of several possible ways. For example, the number can be a constant, such as 2. Alternatively, the number of blocks required in a single request may depend on the buffer state, specifically B _current . For example, the number of thresholds may be set, and the number of blocks required in a single request is derived from a plurality of threshold values below B _current .

  Another example of a condition that, when satisfied, can cause a request to reference multiple blocks is the value of the variable State described above. For example, when State is “Stable” or “Full”, requests can be issued for multiple blocks, but when State is “Low”, all requests can be for one block.

Another embodiment is shown in FIG. In this embodiment, if the next request is to be issued (determined at step 1300), the current State value and Bcurrent are used to determine the size of the next request. If the current State value is "Low", or if the current State value is "Full" and the current representation is not the highest available (as determined in step 1310, the answer is "Yes" ), The next request is chosen to be short, for example, to be for the next block (the block for which the request is made as determined in step 1320). The basis for this is that these are conditions when it is likely that there will be a change in expression soon. If the current State value is "Stable", or if the current State value is "Full" and the current representation is the highest available (as determined in step 1310, the answer is "No )), The duration of the continuous block required in the next request is chosen to be proportional to the α portion of B _current for some fixed α <1 (determined in step 1330, step 1340 For example, if B _current = 5 seconds for α = 0.4, then the next request may be for a block of about 2 seconds and if B _current = 10 seconds The next request may be for a block of about 4 seconds. One reason for this is that under these conditions, there is a low possibility that switching to a new expression will be performed for a time proportional to _Bcurrent .

Flexible Pipelining Block streaming systems may use a file request protocol with a specific underlying transport protocol, eg, TCP / IP. At the beginning of a TCP / IP or other transport protocol connection, it can take a considerable amount of time to achieve full utilization of available bandwidth. This can result in a “connection initiation penalty” each time a new connection is initiated. For example, in the case of TCP / IP, the connection initiation penalty is the time it takes for the initial TCP handshake to establish a connection and the time it takes for the congestion control protocol to achieve full use of available bandwidth. Generated by both.

  In this case, it may be desirable to issue multiple requests using a single connection to reduce the frequency at which connection startup penalties are triggered. However, some file transport protocols, such as HTTP, do not provide a mechanism to cancel a request other than disconnecting the entire transport layer connection, so when a new connection is established instead of an old connection Cause a connection startup penalty. The issued request may need to be canceled if it is determined that the available bandwidth changes and a different media data rate is required instead, ie there is a decision to switch to a different representation . Another reason for canceling an issued request is that the media presentation is terminated (possibly with the same content item at a different point in the presentation, or possibly with a new content item). It may be whether the user requested to be done.

  As is known, connection initiation penalties can be avoided by keeping the connection open and reusing the same connection for subsequent requests and is also known. As such, if multiple requests are issued simultaneously on the same connection, the connection can remain fully utilized (a technique known as “pipelining” in the HTTP context). However, the disadvantage of issuing multiple requests at the same time, or more generally in such a way that multiple requests are issued before completing through a connection with a previous request, It is spent on carrying a response, so that if the change to which a request is to be made becomes desirable, the connection can be broken if it becomes necessary to cancel an already issued request that is no longer desired possible.

  The probability that an issued request needs to be canceled can depend in part on the length of the time interval between the submission of the request and the playback time of the requested block in the following sense. The implication is that if this time interval is long, there is also a high probability that the issued request needs to be canceled (since the available bandwidth is likely to change during that interval).

  As is known, some file download protocols have the property that a single underlying transport layer connection can be advantageously used for multiple download requests. For example, HTTP has the above properties because the reuse of a single connection for multiple requests avoids the “connection initiation penalty” described above for the request other than the first one. However, the disadvantage of this approach is that the connection is spent transporting the data required in each issued request, so if one or more requests need to be canceled, the connection is disconnected, Either a connection startup penalty may be caused when an alternative connection is established, or the client may wait for receipt of data that is no longer needed and cause a delay in receiving subsequent data. That is.

  An embodiment will now be described that retains the benefits of connection reuse without causing this drawback, and further improves the frequency with which connections can be reused.

  Embodiments of the block streaming system described herein are configured to reuse connections for multiple requests without having to spend the initial connection on a specific set of requests. Basically, if a request already made on an existing connection is not yet complete, but close to completion, a new request is made on that connection. One reason for not waiting for existing requests to complete is that when the previous request completes, the connection speed may slow down, i.e. the underlying TCP session may become idle, or TCP This is because the cwnd variable can drop considerably, and therefore the initial download speed of new requests issued over that connection is considerably reduced. One reason for waiting close to completion before making additional requests is that if a new request is issued long before the previous request completes, the newly issued request will not even start for a significant period of time. This is because during this period before a newly issued request begins, the decision to make a new request may no longer be valid, for example, due to the decision to switch representations. . Thus, client embodiments that implement this technique make new requests on the connection as late as possible without slowing down the download capability of the connection.

  The method includes the steps of monitoring the number of bytes received on this connection in response to the most recent request issued on that connection and applying a test to this number. This can be done by configuring the receiver (or transmitter if possible) to monitor and test.

  Upon passing the test, further requests can be made on that connection. One example of a suitable test is whether the number of bytes received is larger than a fixed piece of the size of the requested data. For example, this fragment can be 80%. Another example of a suitable test is based on the following calculation, as shown in FIG. In the calculation, let R be an estimate of the data rate of the connection, T be an estimate of the round trip time (`` RTT ''), X be a factor that can be a constant set to a value between 0.5 and 2, for example, R and T estimates are updated periodically (updated at step 1410). Let S be the size of the data requested in the last request and B be the number of bytes of requested data received (calculated in step 1420).

  A suitable test is to have the receiver (or transmitter, if possible) execute a routine to evaluate the inequality (SB) <X · R · T (tested in step 1430). If “Yes”, the operation is performed. For example, a test may be performed to see if there is another request ready to be issued on the connection (tested in step 1440), and if yes, for that connection The request is made (step 1450), and if “No”, the process returns to step 1410 to continue updating and testing. If the test result in step 1430 is “No”, the process returns to step 1410 to continue updating and testing.

  The inequality test of step 1430 (e.g., performed by an appropriately programmed element) is performed when the amount of remaining data to be received is X and the current estimated receive rate in one RTT. Each subsequent request is issued when equal to the amount of data that can be done. Numerous methods for estimating the data rate R of step 1410 are known in the art. For example, the data rate may be estimated as Dt / t, where Dt is the number of bits received during the preceding t seconds, and t is, for example, 1 second or 0.5 seconds or some other time period possible. Another method is an exponentially weighted average or first order infinite impulse response (IIR) filter with an incoming data rate. Numerous methods for estimating RTT, T in step 1410 are known in the art.

  The test of step 1430 can be applied to the integration of all active connections on an interface, as described in more detail below.

  The method further includes building a list of candidate requests, associating each candidate request with an appropriate set of servers on which requests can be made, and arranging the list of candidate requests in priority order. Several entries in the list of candidate requests may have the same priority. The server in the list of appropriate servers associated with each candidate request is identified by the host name. Each host name corresponds to a set of internet protocol addresses that can be obtained from the domain name system, as is well known. Thus, each possible request for a list of candidate requests is associated with a set of internet protocol addresses, specifically a combination of a set of internet protocol addresses associated with a host name associated with a server associated with the candidate request. . Whenever the test described in step 1430 is met for a connection and no new request has been made for that connection, the highest on the list of candidate requests associated with the destination Internet Protocol address for that connection. Priority request is selected and this request is issued on that connection. The request is also removed from the list of candidate requests.

  Candidate requests may be removed (deleted) from the list of candidate requests, new requests may be added to the candidate list with a higher priority than existing requests on the candidate list, and existing requests on the candidate list may be added. Requests may be changed in priority. The dynamic nature of which requests are on the list of candidate requests and the dynamic nature of the priority of the requests on the candidate list will satisfy the type of test described in step 1430 Depending on the situation, it may change which request can be issued next.

  For example, if at some time t the answer to the test described in step 1430 is “Yes”, the next request made is request A, while the answer to the test described in step 1430 is some time t If 'Yes' until'> t, it is possible that the next request issued will be request B instead. Either request A was removed from the list of candidate requests during times t and t ', or request B was listed in the candidate requests with higher priority than request A during times t and t'. Or because request B was on the candidate list at time t but has a lower priority than request A, and the priority of request B is higher than the priority of request A between times t and t ' It was because it was done.

  FIG. 18 shows an example of a request list on the request candidate list. In this example, there are 3 connections, and there are 6 requests on the candidate list labeled A, B, C, D, E, and F. Each of the requests on the candidate list can be issued on a subset of connections as shown, for example, request A may be issued on connection 1 while request F is issued on connection 2 or connection 3 It is possible. The priority of each request is also labeled in FIG. 18, and a lower priority value indicates a higher priority for the request. Therefore, requests A and B with priority 0 are the highest priority requests, while request F with a priority value of 3 has the lowest priority among the requests on the candidate list.

  At time t, if connection 1 passes the test described in step 1430, either request A or request B is issued on connection 1. Instead, if connection 3 passes the test described in step 1430 at this time t, request D is issued on connection 3 because request D is the request with the highest priority that can be issued on connection 3.

  For all connections, the answer to the test described in step 1430 from time t to later time t ′ is “No”, and the priority of request A from time t and t ′ is 0 Assuming that it has changed to 5, request B is removed from the candidate list and a new request G with priority 0 is added to the candidate list. Then, at time t ′, the new candidate list can be as shown in FIG.

  At time t ′, if connection 1 passes the test described in step 1430, request C with priority 4 is request C because it is the highest priority on the candidate list that can be issued at this time for connection 1. Is issued on connection 1.

  In this same situation, instead suppose that request A was made on connection 1 at time t (request A is the choice of the two highest priorities for connection 1 at time t, as shown in FIG. One). For all connections, the answer to the test described in step 1430 from time t to later time t ′ is “No”, and connection 1 is still for requests made before time t Request A would not have been initiated at least until time t ′ since the data has been delivered at least until time t ′. Issuing request C at time t ′ is a better decision than issuing request A at time t because, after t ′, request C starts at the same time as request A is started. This is because, by that time, the request C has a higher priority than the request A.

  As another alternative, if a test of the type described in step 1430 is applied to active connection integration, the internet protocol address is associated with the first request on the list of candidate requests, or A connection with a destination associated with another request with the same priority as one request may be chosen.

A number of methods are possible for building a list of candidate requests. For example, the candidate list may include n requests that represent requests for the next n parts of the data in the current representation of the presentation in chronological order, where the request for the earliest part of the data is the highest. The request for the slowest part of the data has the lowest priority. In some cases, n may be 1. The value of n may depend on the buffer size B _current , or the variable State, or another measure of the client's buffer occupancy state. For example, multiple thresholds may be set for B _current and the value associated with each threshold, where the value of n is considered to be the value associated with the highest threshold less than B _current .

The embodiment described above ensures flexible allocation of requests for a connection and has the highest priority (because the destination IP address of the connection was not an IP address assigned to any of the host names associated with the request) Even if the request is not suitable for an existing connection, ensure that a preference is given to reusing that connection. The dependence of n on B _current or State or another measure of the state of the client's buffer occupancy needs to be urgent to issue and complete the request associated with the next piece of data to be played in chronological order. Ensure that such “out of priority” requests are not made when at the client.

  These methods can advantageously be combined with cooperative HTTP and FEC.

Consistent Server Selection As is well known, files to be downloaded using the file download protocol are generally identified by an identifier that includes a host name and a file name. For example, this is true for the HTTP protocol, where the identifier is a Uniform Resource Identifier (URI). A host name may correspond to multiple hosts identified by an Internet protocol address. For example, this is a common way to distribute the load of requests from multiple clients across multiple physical machines. Specifically, this approach is typically performed by a content distribution network (CDN). In this case, a request issued on a connection to any of the physical hosts is expected to succeed. Numerous methods are known that allow a client to select from among internet protocol addresses that are associated with a host name. For example, these addresses are typically provided to clients via the domain name system and in priority order. The client can then choose the highest priority (first) internet protocol address. However, in general, there is no cooperation between clients as to how this selection is made, so that different clients may request the same file from different servers. This may cause the same file to be stored in the caches of multiple nearby servers, which reduces the efficiency of the cache infrastructure.

  This can be handled by a system that advantageously increases the probability that two clients requesting the same block will request this block from the same server. The new method described here is in a manner determined by the identifier of the file to be requested, so that different clients presented with the same or similar selection of internet protocol address and file identifier make the same selection. And selecting from among available Internet protocol addresses.

A first embodiment of the method is described with reference to FIG. The client first obtains a set of internet protocol addresses IP ₁ , IP ₂ ,..., IP _n as shown in step 1710. If there is a file for which a request is to be made, as determined in step 1720, the client determines which internet protocol address to issue the request for that file, as determined in steps 1730-1770. To do. Given a set of internet protocol addresses and an identifier of a file to be requested, the method includes ordering the internet protocol addresses in a manner determined by the file identifier. For example, for each internet protocol address, a byte string is constructed that includes a concatenation of the internet protocol address and the file identifier, as shown in step 1730. As shown in step 1740, a hash function is applied to this byte string, and the resulting hash values are arranged according to a fixed order, for example, the order of increasing numbers, as shown in step 1750, and the Internet Produces a protocol address order. Since the same hash function can be used by all clients, it is guaranteed that the same result will be generated by the hash function for a given input by all clients. The hash function may be statistically configured to all clients in the set of clients, or all clients in the set of clients may have a portion of the hash function when the client obtains a list of internet protocol addresses. Or all clients in the set of clients may obtain a partial or complete description of the hash function when the client obtains the file identifier, or the hash The function may be determined by other means. The first internet protocol address in this order is chosen and this address is then used to establish a connection and issue a request for all or part of the file, as shown in steps 1760 and 1770.

  The above method can be applied when a new connection is established to request a file. This can also be applied where a large number of established connections are available and one of these can be chosen to issue a new request.

  Further, if the established connection is available and the request can be chosen from a set of candidate requests of equal priority, for example, the same method of hash values described above results in the order of candidate requests, The candidate request that appears first in this order is selected. Again, the method calculates the hash for each combination of connection and request, orders these hash values according to a fixed order, and selects the combination that appears first in the order produced by the set of request and connection combinations. Can be combined to select both connections and candidate requests from a set of equal priority connections and requests.

  This method is advantageous for the following reason. The usual approach performed by the block serving infrastructure, as shown in Figure 1 (BSI 101) or Figure 2 (BSI 101), especially the approach commonly performed by CDN, is to have multiple caching proxy servers that receive client requests. Will provide. A caching proxy server may not be given the requested file in a given request, in which case such a server typically forwards the request to another server, and usually a response containing the requested file From the server and forward the response to the client. The caching proxy server can also store (cache) the requested file so that it can immediately respond to subsequent requests for the file. The general approach described above has the property that the set of files stored on a given caching proxy server is highly dependent on the set of requests received by that caching proxy server.

  The method described above has the following advantages. If all clients in a set of clients are given the same list of internet protocol addresses, these clients use the same internet protocol for all requests made to the same file. If there are two different lists of internet protocol addresses and each client is given one of these two lists, the client will be at most two different for every request made to the same file Use an internet protocol address. In general, if the list of internet protocol addresses given to the client is similar, the client uses a small set of given internet protocol addresses for all requests made to the same file. Neighboring clients tend to be given a similar list of Internet protocol addresses, so neighboring clients are likely to make requests for files from only a small portion of the caching proxy server that they are available to . Thus, there are only a few caching proxy servers that cache files, which advantageously minimizes the amount of caching resources used to cache files.

  Preferably, the hash function is such that a very small portion of the various inputs are mapped to the same output, the various inputs are mapped to essentially random outputs, and for a given set of internet protocol addresses, the internet protocol The ratio of the first file in the sorted list in which the given one of the addresses is generated in step 1750 has the property that it is approximately the same for all Internet protocol addresses in the list. On the other hand, it is important that the hash function is deterministic in the sense that for a given input the output of the hash function is the same for all clients.

  Another advantage of the method described above is as follows. Assume that all clients in a set of clients are given the same list of internet protocol addresses. Due to the nature of the hash function described above, requests for different files from these clients are likely to be evenly distributed across a set of Internet protocol addresses, which in turn turns the request evenly across caching proxy servers. Means to be distributed. Thus, the caching resources used to store these files are evenly distributed across the caching proxy server, and requests for files are evenly distributed across the caching proxy server. Thus, the method achieves both storage balance and load balance across the caching infrastructure.

  Numerous variations on the techniques described above are known to those skilled in the art, and in many cases, these variations will result in the set of files stored on a given proxy being the set of requests received by the caching proxy server. Retains the property of being determined at least in part by In the common case where a given host name resolves to multiple physical caching proxy servers, it is common for all these servers to eventually store a copy of any given file that is frequently requested. Is. Such duplication may not be desirable because storage resources on the caching proxy server are limited and, as a result, files may sometimes be removed (purged) from the cache. The new method described here ensures that requests for a given file are directed to the caching proxy server in such a way that this duplication is reduced, thereby eliminating the need to remove the file from the cache. Reduce and increase the probability that any given file is in the proxy cache (ie, not purged from the proxy cache).

  If the file is in the proxy cache, the response sent to the client will be faster, which is the probability of delay in reaching the requested file, which can result in a pause in media playback and thus a bad user experience This is advantageous in reducing In addition, if the file is not in the proxy cache, the request may be sent to another server, causing additional load on both the serving infrastructure and the network connection between the servers. In many cases, the server to which the request is sent may be at a remote location, and returning a file from this server to the caching proxy server can incur transmission costs. Thus, the novel method described here results in a reduction in these transmission costs.

Probabilistic whole file request
A particular problem when the HTTP protocol is used with Range requests is the behavior of cache servers that are commonly used to provide scalability to the serving infrastructure. Although it is common for HTTP cache servers to support HTTP Range headers, the exact behavior of different HTTP cache servers varies from implementation to implementation. Many cache server implementations service Range requests from the cache when the file is available in the cache. A typical implementation of an HTTP cache server always forwards a downstream HTTP request containing a Range header to an upstream node, unless the cache server has a copy of the file (cache server or origin server). In some implementations, the upstream response to the Range request is the entire file, the entire file is cached, and the response to the downstream Range request is extracted from this file and sent. However, in at least one implementation, the upstream response to the Range request is simply the data bytes in the Range request itself, and these data bytes are not cached and are instead sent as a response to the downstream Range request. Just do. As a result, the use of the Range header by the client can result in the file itself never being cached and the undesirable scalability characteristics of the network being lost.

  Above, the operation of a caching proxy server has been described, and a method for requesting a block from a file that is the integration of multiple blocks has been described. For example, this can be accomplished by use of an HTTP Range request header. Such a request is referred to below as a “partial request”. Further embodiments will now be described that have advantages when the block serving infrastructure 101 does not provide full support for HTTP Range headers. In general, servers in a block serving infrastructure, such as a content distribution network, support partial requests, but cannot store responses to partial requests in local storage (cache). Such a server can satisfy a partial request by forwarding the request to another server unless the entire file is stored on the local storage device, and the entire file is stored on the local storage device. The response can be sent without forwarding the request to another server.

  If the block serving infrastructure shows this behavior, the block request streaming system that utilizes the new improvements in block integration described above may have poor performance, because all requests that are partial requests This is because it is forwarded to another server and the request is not serviced by the caching proxy server, making the goal of putting the caching proxy server first place meaningless. During the block request streaming process as described above, the client can request a block at the beginning of the file at some point.

  According to the novel method described herein, whenever a condition is met, such a request can be translated from a request for the first block in the file to a request for the entire file. When a request for the entire file is received by the caching proxy server, the proxy server typically stores the response. Thus, the use of these requests, whether subsequent to a complete file or partial request, can be serviced directly by the caching proxy server, brings the file to the local caching proxy server cache. It is. The above conditions are given at least for these requests between a set of requests associated with a given file, eg, a set of requests generated by a set of clients viewing the content item in question It can be filled against fragments.

  An example of a suitable condition is that a randomly chosen number is above a given threshold. This threshold is such that the conversion of a single block request to an entire file request, on average, for a given piece of request, for example, once in 10 (in this case, a random number is the interval [0 , 1], and the threshold may be 0.9). Another example of a suitable condition is that a hash function calculated for some information associated with the block and some information associated with the client takes one of the provided sets of values. Although this method has the advantage that for frequently requested files, the file is carried to the local proxy server cache, the operation of the block request streaming system is that each request is for a single block. There is no significant change from some standard behavior. In many cases, if a request change occurs from a single block request to an entire file request, the client procedure will continue to request other blocks in the file otherwise. If so, such a request may be suppressed because the block in question is received in either case as a result of the request for the entire file.

URL construction and segment list generation and exploration Segment list generation is specific client local to the start of the media presentation if on demand, or to a specific representation starting at some start time starttime, expressed in real time. It deals with the problem of how a client can generate a segment list from MPD at time NOW. The segment list may include a locator, eg, a URL for optional initial presentation metadata, along with a list of media segments. Each media segment may be assigned a starttime, duration, and locator. The starttime usually represents an approximate value for the media time of the media included in the segment, but does not necessarily represent the exact time of the sample. The starttime is used by HTTP streaming clients to issue download requests when appropriate. Generation of the segment list including each start time can be done in various ways. The URL may be provided as a playlist, or URL construction rules may be advantageously used for a compact representation of the segment list.

  A segment list based on URL construction may be performed, for example, when the MPD signals the segment list by specific attributes or elements, such as FileDynamicInfo or equivalent signals. A general method for creating a segment list from URL construction is given in the URL Construction Overview section below. Playlist-based construction may be signaled by different signals, for example. Searching the segment list and reaching the correct media time is also advantageously implemented in this situation.

URL Construction Overview As previously described, in one embodiment of the present invention, metadata may be provided that includes URL construction rules that allow a client device to construct a file identifier for a block of a presentation. Where available encodings such as changes to URL construction rules, changes to the number of available encodings, bit rate, aspect ratio, resolution, audio or video codec or codec parameters, or other parameters A further new improvement to the block request streaming system that provides changes to associated metadata is described.

  In this new improvement, additional data associated with each element of the metadata file can be provided that indicates the time intervals within the entire presentation. Within this time interval, the element may be considered valid and outside of that time interval, the element may be ignored. Furthermore, the syntax of metadata can be enhanced so that an element previously allowed to appear only once or at most once may appear multiple times. In this case, an additional constraint may be provided that specifies that the defined time intervals must be disjoint for such elements. Considering only the elements whose time interval includes a given time instant at any given time instant will result in a metadata file that is consistent with the original metadata syntax. Such a time interval is called an effective interval. This method thus implements signaling within a single metadata file change of the type described above. Advantageously, such a method may be used to provide a media presentation that supports the types of changes described at defined points in the presentation.

URL Constructor As described herein, the general features of a block request streaming system are required by a client to identify available media encodings and request blocks from those encodings. This means that it is necessary to provide “metadata” to the client that provides the information. For example, for HTTP, this information may include the URL of the file that contains the media block. A playlist file can be provided that lists the URLs of the blocks for a given encoding. Multiple playlist files, one for each encoded product, are provided, along with a master playlist of playlists that list playlists corresponding to different encoded products. The disadvantage of this system is that it can take some time for the metadata to be requested when the client starts the stream, since the metadata can be very large. A further disadvantage of this system is that live content is generated "on the fly" from a media stream in which a file corresponding to a media data block is being filmed in real time (live), eg, a live sports event or news program. It is obvious in the case. In this case, the playlist file may be updated each time a new block becomes available (eg, every few seconds). The client device can fetch the playlist file repeatedly, determine whether new blocks are available, and obtain their URLs. This can impose a heavy load on the serving infrastructure, specifically meaning that metadata files cannot be cached longer than the update interval, which is generally equal to a block size on the order of seconds. .

  One important aspect of the block request streaming system is the method used to inform the client of a file identifier, eg, a URL, to be used with the file download protocol to request a block. For example, for each presentation presentation, a playlist is provided that lists the URLs of files that contain blocks of media data. The disadvantage of this method is that in at least some of the playlists, the file itself needs to be downloaded before playback can begin, increasing channel zapping time and thus causing a bad user experience. For long media presentations with some or many representations, the list of file URLs can be large, so the playlist file can be large, further increasing channel zapping time.

  Another drawback of this method occurs in the case of live content. In this case, the complete list of URLs is not made available in advance, the playlist file is updated regularly as new blocks become available, and the client receives the updated version Request playlist files regularly. Because this file is updated frequently, it cannot be stored long in the caching proxy server. This means that very many requests for this file are forwarded to other servers and ultimately to the server that generates the file. In the case of popular media presentations this can result in a heavy load on this server and network, which in turn can lead to slow response times and thus long channel zapping times and bad user experiences. . In the worst case, the server is overloaded, which prevents some users from seeing the presentation.

  It is desirable in designing block request streaming systems to avoid imposing restrictions on the types of file identifiers that can be used. This is because a number of considerations can motivate the use of certain types of identifiers. For example, if the block serving infrastructure is a content delivery network, file names or storage agreements related to the desire to distribute storage or serving load across the network, or certain types of files that cannot be predicted when designing a system There may be other requirements leading to the identifier.

  Now, further embodiments will be described that retain the flexibility to choose an appropriate file identification information representation method while mitigating the drawbacks mentioned above. In this way, metadata can be provided for each representation of the media presentation, including file identifier construction rules. The file identifier construction rule may include a text string, for example. In order to determine the file identifier for a given block of the presentation, a method of interpretation of the file identifier construction rules may be provided, which includes determining input parameters and evaluating the file identification information construction rules along with the input parameters. including. The input parameters may include, for example, the index of the file to be identified, where the first file has index 0, the second file has index 1, and the third file has index 2. The same applies hereinafter. For example, if each file spans the same amount of time (or approximately the same amount of time), the index of the file associated with any given time in the presentation can be easily determined. Alternatively, the time within the presentation that each file spans can be provided in the presentation or version metadata.

  In one embodiment, the file identifier construction rule may include a text string that may include some spatial identifier corresponding to the input parameter. A method for evaluating a file identifier construction rule includes determining a position of a spatial identifier within a text string and replacing each such spatial identifier with a string representation of the value of the corresponding input parameter. .

  In another embodiment, the file identifier construction rules may include text strings that conform to the formula language. The expression language includes a definition of syntax that an expression in the language can conform to, and a set of rules for evaluating strings that conform to the syntax.

  Here, a specific example will be described with reference to FIG. An example of the syntax definition for an appropriate formula language defined in the extended Bacchus-Naur notation is as shown in FIG. An example rule for evaluating a string that conforms to the <expression> production in Figure 21 is to recurse a string that conforms to the <expression> production into a string that conforms to the <literal> production, as follows: Conversion.

  <expression> conforming to <literal> production is not changed.

  The <expression> that conforms to the <variable> production is replaced by the value of the variable identified by the <token> string in the <variable> production.

  An <expression> that conforms to a <function> production evaluates each of the arguments according to these rules, and applies transformations to these arguments according to the <token> element of the <function> production, as described below. Is evaluated.

  An <expression> that conforms to the last alternative form of an <expression> production evaluates two <expression> elements and becomes the <operator> element of the last alternative form of an <expression> production, as described below. It is evaluated by applying actions to these arguments accordingly.

  In the method described above, it is assumed that the evaluation is performed in a situation where multiple variables can be defined. A variable is a (name, value) pair, “name” is a string that conforms to <token> production, and “value” is a string that conforms to <literal> production. Some variables may be defined outside the evaluation process before the evaluation begins. Other variables can be defined within the evaluation process itself. All variables are "global" in the sense that only one variable exists with each possible "name".

  An example of a function is the “printf” function. This function accepts one or more arguments. The first argument may conform to <string> production (hereinafter “string”). The printf function is evaluated as a converted version of its first argument. The applied transformation is the same as the “printf” function in the C standard library, and the additional arguments included in the <function> production provide the additional arguments required by the C standard library printf function.

  Another example of a function is the “hash” function. This function accepts two arguments, the first argument may be a string, and the second argument may conform to <number> production (hereinafter “number”). The “hash” function applies a hash algorithm to the first argument and returns a result that is a non-negative integer less than the second argument. An example of a suitable hash function is given in the C function shown in FIG. 22, where the arguments of the C function are an input string (excluding the enclosing quotes) and a numerical input value. Other examples of hash functions are well known to those skilled in the art.

  Another example of a function is the “Subst” function that takes one, two, or three string arguments. If one argument is given, the result of the “Subst” function is the first argument. If two arguments are given, the result of the "Subst" function will eliminate any presence of the second argument (except the enclosing quotes) within the first argument, and the first so modified Calculated by returning an argument. If three arguments are given, the result of the "Subst" function is the presence of any second argument (excluding the enclosing quotes) in the first argument and the third argument (excluding the enclosing quotes) ) And return the first argument so modified.

  Some examples of operators are the add, subtract, divide, multiply, and modulo operators, identified by the <operator> production "+", "-", "/", "*", "%" respectively. It is. These operators require the <expression> production on either side of the <operator> production to be evaluated as a number. Operator evaluation applies the appropriate arithmetic operations (addition, subtraction, division, multiplication, and modulo, respectively) to these two numbers in the usual way and returns the result in a format that conforms to <number> production. including.

  Another example of an operator is an assignment operator identified by the <operator> production “=”. This operator requires that the left argument be evaluated as a string whose contents conform to the <token> production. The content of the string is defined as the string within the enclosing quotes. The equals operator causes a variable whose name is <token> equal to the contents of the left argument to be assigned a value equal to the result of evaluating the right argument. This value is also the result of evaluating the operator expression.

  Another example of an operator is the ordinal operator identified by the <operator> production “;”. The result of the evaluation of this operator is the right argument. Note that, like all operators, both arguments are evaluated and the left argument is evaluated first.

  In one embodiment of the present invention, the file identifier may be obtained by evaluating a file identifier construction rule according to the above rules with a specific set of input variables identifying the requested file. An example of an input variable is a variable with the name “index” and a value equal to the numerical index of the file in the presentation. Another example of an input variable is a variable with the name “bit rate” and a value equal to the average bit rate of the requested version of the presentation.

  FIG. 23 shows some examples of file identifier construction rules, where the input variables are “id” giving the identifier of the desired presentation representation and “seq” giving the sequence number of the file .

  Numerous variations of the above method are possible, as will be apparent to those skilled in the art after reading this disclosure. For example, not all of the functions and operators described above may be provided, or additional functions or operators may be provided.

  URL construction rules and timing

  This section provides basic URI construction rules for assigning file or segment URIs, as well as start times for each segment, in presentations and media presentations.

  In this section, the availability of media presentation description at the client is assumed.

  Suppose an HTTP streaming client is playing media that is downloaded within a media presentation. The actual presentation time of the HTTP client can be defined with respect to where the presentation time is relative to the beginning of the presentation. At initialization, a presentation time t = 0 can be assumed.

  At any point t, the HTTP client can accept any data with a playback time tP (also for the beginning of the presentation) that is at most MaximumClientPreBufferTime before the actual presentation time t and user interaction, eg search, fast forward Any data required can be downloaded. In some embodiments, MaximumClientPreBufferTime may not even be defined in the sense that the client can download data before the current playback time tP without constraints.

  An HTTP client can avoid downloading unnecessary data, for example, any segment from a representation that is not expected to be played typically will not be downloaded.

  The basic process in providing a streaming service is by generating an appropriate request to download the entire file / segment or a subset of files / segments, for example by using an HTTP GET request or an HTTP partial GET request It can be a data download. Although this description deals with how to access data for a particular playback time tP, in general, clients may download data for playback times in a longer time range to avoid inefficient requests. it can. HTTP clients can minimize the number / frequency of HTTP requests when providing streaming services.

  In order to access media data at a playback time tP, or at least close to the playback time tP, in a particular representation, the client determines the URL for the file containing this playback time and in addition to this playback time Determine the byte range in the file to access.

  In Media Presentation Description, for example, the expression id r can be assigned to each expression by using the RepresentationID attribute. In other words, when the MPD content is written by the capture system or read by the client, it is interpreted as having an assignment. In order to download data for a particular representation with id r for a particular playback time tP, the client can construct an appropriate URI for the file.

  Media Presentation Description can assign the following attributes to each file or segment of each expression r.

  (a) i = 1,2, ..., Nr, the order number i of the file in the expression r, and the expression id r and the file index i defined for the presentation time defined as (b) ts (r, i) The relative start time of the accompanying file, (c) File URI of the file / segment with representation id r and file index i, denoted as FileURI (r, i).

  In one embodiment, the file start time and file URI may be explicitly provided for the representation. In another embodiment, a list of file URIs may be explicitly provided, where each file URI is uniquely assigned according to its position in the list and the segment start time is from 1 to i-1. Is derived as the sum of the durations of all segments for that segment. The duration of each segment can be provided according to any of the rules discussed above. For example, a person skilled in basic mathematics can use other methods to derive a method to easily derive the start time from a single element or attribute and the location / index of the file URI in the representation. can do.

  If dynamic URI construction rules are provided in the MPD, the start time of each file and the URI of each file are added in the construction rules, the index of the requested file, and possibly any media presentation description Can be built dynamically by using Information can be provided in MPD attributes and elements such as FileURIPattern and FileInfoDynamic, for example. FileURIPattern provides information on how to construct a URI based on file index sequence number i and expression ID r. FileURIFormat is constructed as follows:

  FileURIFormat = sprintf ("% s% s% s% s% s.% S", BaseURI, BaseFileName,

  RepresentationIDFormat, SeparatorFormat,

  FileSequenceIDFormat, FileExtension);

  FileURI (r, i) is constructed as follows.

  FileURI (r, i) = sprintf (FileURIFormat, r, i);

The relative start time ts (r, i) for each file / segment may be derived by some attribute included in the MPD that describes the duration of the segment in this representation, eg, the FileInfoDynamic attribute. The MPD may also include an order of FileInfoDynamic attributes that is global for all presentations in the media presentation, or at least for all representations over a period of time, in the same manner as defined above. If media data for a particular playback time tP in the expression r is requested, the playback time of this index is ts (r, i (r, tP)) and ts (r, i (r, tP) +1) As in between, the corresponding index i (r, tP) can be derived as i (r, t _p ). Access to a segment may be further restricted by the above case, for example, the segment is not accessible.

Accessing the exact playback time tP after the corresponding segment index and URI are obtained depends on the actual segment format. In this example, assume that the media segment has a local time axis that starts at 0 without loss of generality. To access and present data at playback time tP, the client supports local time from a file / segment that can be accessed through URI FileURI (r, i) where i = i (r, t _p ) Data to download.

  In general, the client can download the entire file and then access the playback time tP. However, not all information in a 3GP file need be downloaded, because the 3GP file provides a structure for associating local timing with byte ranges. Thus, as long as sufficient random access information is available, only a specific byte range for accessing the playback time tP may be sufficient to play the media. Sufficient information about the structure, byte range mapping, and media segment local timing may also be provided in the first part of the segment using, for example, a segment index. By having access to the first, eg 1200 bytes, of the segment, the client may have enough information to directly access the required byte range for the playback time tP.

  In a further example, assume that a segment index, sometimes defined as a “tidx” box, may be used to identify the byte offset of the requested fragment or fragments, as follows. A Partial GET request may be formed for the requested fragment or fragments. Other alternatives exist, for example, the client can issue a standard request for the file and cancel it when the first “tidx” box is received.

Search The client may attempt to search for a specific presentation time tp in the representation. Based on the MPD, the client has access to the media segment start time and media segment URL for each segment in the representation. The client uses the segment index segment_index of the segment most likely to include a media sample for the presentation time tp as the largest segment index i whose start time tS (r, i) is less than or equal to the presentation time tp, that is, segment_index = max { i | tS (r, i) <= tp} can be obtained. The segment URL is acquired as FileURI (r, i).

  Note that the timing information in the MPD may be approximate due to issues related to random access point placement, media track alignment, and media timing drift. As a result, the segment identified by the above procedure may start slightly later than tp, and the media data for the presentation time tp may be in the previous media segment. In the case of searching, the search time may be updated to be equal to the first sample time of the retrieved file, or the preceding file may be retrieved instead. However, media data for the time between tp and the beginning of the retrieved segment is still available during continuous playback, including when there is a switch between alternative representations / versions. Please note that.

  For an accurate search for the presentation time tp, the HTTP streaming client needs to access a random access point (RAP). In the case of 3GPP-adaptive HTTP streaming, to determine a random access point in the media segment, the client uses, for example, information in the “tidx” or “sidx” box to identify the random access point, if present. , A corresponding presentation time during the media presentation can be located. If the segment is a 3GPP movie fragment, the client uses the information in the "moof" and "mdat" boxes to locate the RAP, for example, the required presentation time from the information in the movie fragment, and the MPD It is also possible to obtain the segment start time derived from If a RAP with a presentation time prior to the requested presentation time tp is not available, the client can access the previous segment or use only the first random access point as a result of the search be able to. These procedures are simple if the media segment starts with a RAP.

  Note also that not all information in the media segment need necessarily be downloaded to access the presentation time tp. The client may first request a “tidx” or “sidx” box from the beginning of the media segment, for example, using a byte range request. By using the “tidx” or “sidx” box, the timing of the segment can be associated with the byte range of the segment. By using partial HTTP requests continuously, only the relevant part of the media segment can be accessed for improved user experience and low start-up delay.

Segment List Generation A simple HTTP that uses the information provided by the MPD to create a list of segments for a representation with a signaled approximate segment duration, dur, as described herein. It should be clear how a streaming client should be implemented. In some embodiments, the client can assign media segments within the continuous index i = 1,2,3, ... of the representation, ie, the first media segment is assigned index i = 1. , The second media segment is assigned index i = 2, and so on. The list of media segments with segment index i is then assigned startTime [i], and URL [i] is generated, for example: First, index i is set to 1. The start time of the first media segment is acquired as 0, and startTime [1] = 0. The URL and URL [i] of the media segment i are acquired as FileURI (r, i). This process continues for all described media segments with index i, the starttime [i] of media segment i is obtained as (i-1) * dur, and URL [i] is FileURI (r, i ).

Simultaneous HTTP / TCP Requests One challenge in block request streaming systems is the requirement to always request the highest quality blocks that can be completely received in time for playback. However, since the data arrival rate may not be known in advance, it may happen that the requested block does not arrive in time to be played. This should cause media playback to pause, resulting in a bad user experience. This problem is due to requiring lower quality (and hence smaller size) blocks that are more likely to be received in time, even if the data arrival rate drops during the reception of the block. Can be mitigated by a client algorithm that employs a conservative approach to the selection of blocks to request. However, this conservative approach has the disadvantage that it may deliver lower quality playback, which is also a bad user experience, to the user or destination device. The problem can be magnified when multiple HTTP connections are used simultaneously to download different blocks as described below, which means that available network resources are shared across multiple connections, thus This is because they are used simultaneously for blocks with different playback times.

  It may be advantageous for a client to make requests for multiple blocks simultaneously, and in this context, “simultaneously” means that responses to requests are made in overlapping time intervals, and the requests are not necessarily exact Or even not at the same time. In the case of the HTTP protocol, this approach can improve the utilization of available bandwidth due to the behavior of the TCP protocol (as is well known). This can be particularly important for improving content zapping time. That is, when new content is requested for the first time, the corresponding HTTP / TCP connection through which data for the block is requested may be slow to start, so at this point some HTTP / TCP connections This is because the data delivery time of the first block can be dramatically increased by using. However, requesting different blocks or fragments over different HTTP / TCP connections can also lead to performance degradation. It can be that the request for the block to be played first conflicts with the request for the subsequent block, and the competing HTTP / TCP downloads vary greatly in terms of delivery speed, and therefore the request completion time can vary greatly, It's generally impossible to control which HTTP / TCP downloads complete quickly and which are slower, so at least some of the first few blocks of HTTP / TCP downloads are complete This is likely to lead to long and variable channel zapping times.

  Each block or fragment of the segment is downloaded over a separate HTTP / TCO connection, the number of parallel connections is n, the playback duration of each block is t seconds, and the streaming rate of the content associated with the segment is S Assume that there is. When the client first starts streaming content, a request may be issued for the first n blocks representing n * t seconds of media data.

  As is known to those skilled in the art, there are significant variations in the data rate of a TCP connection. However, to simplify this discussion, ideally all connections proceed in parallel so that the first block is received almost simultaneously with the other n-1 blocks required. Assume. To further simplify the discussion, assume that the total bandwidth used by n download connections is fixed at the value B for the entire download duration, and that the streaming rate S is constant throughout the representation. To do. Further, the media data structure assumes that block playback can occur when the entire block is available at the client, i.e., block playback is, for example, the structure of the underlying video encoding Because of encryption, or because encryption is used to encrypt each fragment or block separately, and therefore the entire fragment or block needs to be received before it can be decrypted, It can only start after it is received. Thus, to simplify the following discussion, assume that the entire block needs to be received before any of the blocks can be played. Then, the time required for the first block to arrive and be replayed is about n * t * S / B.

  Since it is desirable to minimize content zapping time, it is desirable to minimize n * t * S / B. The value of t can be determined by factors such as the underlying video coding structure and how the capture method is utilized, so t can be quite small, but a very small value of t May lead to overly complex segment maps, and in some cases may not match them if efficient video encoding and decoding is used. The value of n can also affect the value of B. That is, B may be larger for a larger number n of connections, and thus reducing the number n of connections may reduce the amount of available bandwidth B used. Negative side effects, and therefore may not be effective in achieving the goal of reducing content zapping time. The value of S depends on which representation is chosen for download and playback, and ideally S is as much as possible to maximize media playback quality for a given network condition. Must be close to B. Therefore, to simplify this discussion, assume that S is approximately equal to B. Then, the channel zapping time is proportional to n * t. Therefore, using more connections to download different fragments, as is usually the case, if the total bandwidth utilized by a connection is quasi-linearly proportional to the number of connections, the channel zapping time is reduced. Can deteriorate.

  As an example, assume t = 1 second, if n = 1, B value = 500 Kbps, if n = 2, B value = 700 Kbps, and if n = 3, assume B value = 800 Kbps . Suppose an expression with S = 700 Kbps is chosen. Then, for n = 1, the download time for the first block is 1 * 700/500 = 1.4 seconds, for n = 2, the download time for the first block is 2 * 700/700 = 1 second, and n = In 3, the download time of the first block is 3 * 700/800 = 2.625 seconds. In addition, as the number of connections increases, it is likely that the individual download speed variation of the connection will increase (although there is likely to be some significant variation in a single connection). Thus, in this example, channel zapping time and channel zapping time variations increase as the number of connections increases. Intuitively, the blocks being delivered have different priorities, i.e. the first block has the earliest delivery deadline, the second block has the second earliest deadline, etc. Download connections through which content is distributed compete for network resources during delivery, so blocks with the earliest deadlines become more delayed as more competing blocks are requested. On the other hand, even in this case, the final use of two or more download connections makes it possible to support a higher streaming rate sustainably, for example, with three connections, this can lead to a streaming rate of up to 800 Kbps. Although it can be supported in the example, only one stream of 500 Kbps can be supported per connection.

  In practice, as stated above, the data rate of a connection can vary greatly over time and between connections within the same connection, resulting in n requested blocks being It generally does not complete in the same time, and in practice it can generally happen that one block can be completed in half the time of another. This effect can cause the first block to complete much earlier than other blocks in some cases and the first block to complete much later than other blocks. As a result, the onset of playback can occur relatively early in some cases, and can occur late in other cases, resulting in unpredictable behavior. This unpredictable behavior can be annoying to the user and can be considered a bad user experience.

  Therefore, what is needed is a method that supports multiple possible high quality streaming rates while multiple TCP connections can be utilized to improve channel zapping time and channel zapping time variability . What is also needed is the portion of available bandwidth allocated to each block so that, if necessary, a larger portion of available bandwidth can be allocated to the block with the closest playback time. Is a method that allows the block playback time to be adjusted as it approaches.

Cooperative HTTP / TCP Requests Here, a method for using simultaneous HTTP / TCP requests in a cooperative manner is described. The receiver can utilize multiple simultaneous cooperative HTTP / TCP requests, for example using multiple HTTP byte range requests, each such request being one of the fragments in the source segment. Or all of the fragments of the source segment, or part of the repair fragment of the repair segment, or all of the repair fragments of the repair segment.

  The benefits of coordinated HTTP / TCP requests with the use of FEC repair data can be particularly important in order to stably provide fast channel zapping time. For example, at channel zapping time, it is likely that a TCP connection has just been initiated or has been idle for a period of time, in which case the congestion window cwnd is at the minimum value for that connection and therefore The delivery speed of these TCP connections takes some round trip time (RTT) to increase, and during this rise time there is a large variation in the delivery speed on different TCP connections.

  An overview of non-FEC methods is described here, which is a cooperative HTTP / TCP request method, where only source block media data is requested using multiple simultaneous HTTP / TCP connections, i.e. FEC repair data is not required. In the non-FEC method, for example, each HTTP byte range request is indicated in the segment map for a fragment because the same fragment portion is requested through a different connection, for example using an HTTP connection byte range request for the fragment portion. For the portion of the byte range to be Bandwidth is available because individual HTTP / TCP requests can increase their delivery speed to fully utilize available bandwidth through several RTTs (round trip times). If there is a relatively long period of time that is less than the width, and therefore a single HTTP / TCP connection is used to download the first fragment of content to be played, for example, the channel zapping time may be large is there. Downloading different parts of the same fragment over different HTTP / TCP connections using non-FEC methods can greatly reduce the channel zapping time.

  An overview of the FEC method is presented here, which is a cooperative HTTP / TCO request method, where the source segment media data and the FEC repair data generated from the media data use multiple concurrent HTTP / TCP connections As required. In the FEC method, the same fragment part and the FEC repair data generated from that fragment are requested through different connections using HTTP connection byte range requests for the fragment part, for example, each HTTP byte range request is , For the portion of the byte range indicated in the segment map for the fragment. Bandwidth is available because individual HTTP / TCP requests can increase their delivery speed to fully utilize available bandwidth through several RTTs (round trip times). If there is a relatively long period of time that is less than the width, and therefore a single HTTP / TCP connection is used to download the first fragment of content to be played, for example, the channel zapping time may be large is there. Using the FEC method has the same advantages as the non-FEC method and does not require all of the requested data to arrive before the fragment can be recovered, further reducing channel zapping time and channel zapping time variation It has the additional advantage of. By making a request over a different TCP connection and making another request by also requesting FEC repair data on at least one of the connections, for example, the first requested fragment that allows media playback to start The time taken to deliver a sufficient amount of data to recover is greatly reduced and can be made much more stable than if cooperative TCP connection and FEC repair data were not used.

  FIGS. 24 (a) to 24 (e) show the distribution rate variation of five TCP connections over the same link to the same client from the same HTTP web server in the emulated evolution data optimized (EVDO) network. In Figures 24 (a) to 24 (e), the X-axis represents time in seconds and the Y-axis represents the client through each of the five TCP connections, measured over an interval of 1 second for each connection. Indicates the rate at which bits are received. In this particular simulation, there were a total of 12 TCP connections over this link, so the network was relatively busy during the time shown, which means that two or more clients can be May be typical when streaming in the same cell. Note that the delivery rates are somewhat correlated over time, but at many points in time there are significant differences in the delivery rates of the five connections.

  Figure 25 shows a possible request structure for a fragment that is 250,000 bits in size (approximately 31.25 kilobytes), with four HTTP byte range requests made in parallel for different parts of the fragment, i.e. the first HTTP The connection requires the first 50,000 bits, the second HTTP connection requires the next 50,000 bits, the third HTTP connection requires the next 50,000 bits, and the fourth HTTP connection requires the next 50,000 bits To do. If FEC is not used, i.e. for non-FEC methods, these are all four requests for the fragment in this example. If FEC is used, that is, in the FEC method, in this example, there is one additional HTTP connection that requests FEC repair data for an additional 50,000-bit repair segment generated from the fragment.

  FIG. 26 is an enlargement of the first few seconds of the five TCP connections shown in FIGS. 24 (a) to 24 (e), where in FIG. 26 the X axis shows time at 100 millisecond intervals and the Y axis Indicates the rate at which bits are received at the client on each of the five TCP connections measured over an interval of 100 milliseconds. One line is the bit received at the client for the fragment from the first 4 HTTP connections (excluding the HTTP connection through which FEC data is requested), i.e. the sum of what arrives using the non-FEC method The amount of Another line is the sum of the bits received at the client for the fragment from all five HTTP connections (including the HTTP connection through which FEC data is requested), i.e. the total that arrives using the FEC method. Indicates the amount. In the FEC method, it is assumed that the fragment can be FEC decoded from the reception of any 200,000 bits out of the requested 250,000 bits, which can be realized if, for example, a Reed-Solomon FEC code is used. For example, if a RaptorQ code described in Luby IV is used, it can be basically realized. For the FEC method, in this example, enough data is received after 1 second to recover the fragment using FEC decoding, and data for subsequent fragments is requested before the first fragment is fully played. Channel zapping time of 1 second (assuming that it can be received and received). In the non-FEC method, in this example, all data for four requests needs to be received before the fragment can be recovered, and fragment recovery occurs after 1.7 seconds, resulting in a channel zapping time of 1.7 seconds . Thus, in the example shown in FIG. 26, the non-FEC method is 70% worse in terms of channel zapping time than the FEC method. One of the reasons for the advantages demonstrated by the FEC method in this example is that in the FEC method, any 80% reception of the requested data allows fragment recovery, whereas in the non-FEC method the requested data 100% of the reception is required. Therefore, the non-FEC method has to wait for the slowest TCP connection to finish delivery, and due to natural variations in TCP delivery rate, the delivery speed of the slowest TCP connection compared to the average TCP connection Tend to fluctuate significantly. Due to the FEC method, in this example, one slow TCP connection does not determine when a fragment can be recovered. Instead, with the FEC method, the delivery of sufficient data is much more dependent on the average TCP delivery rate than the worst case TCP delivery rate.

  There are many variations of the non-FEC method and the FEC method described above. For example, a coordinated HTTP / TCP request may be used for only the first few fragments after a channel zap occurs, after which only a single HTTP / TCP request may be used for further fragments, multiple fragments, or Used to download the entire segment. As another example, the number of cooperative HTTP / TCP connections used depends on the urgency of the requested fragment, i.e. how imminent the playback time of these segments is, and the current network conditions It may be in accordance with both.

  In some variations, multiple HTTP connections may be used to request repair data from the repair segment. In other variations, different amounts of data may be requested over different HTTP connections, for example depending on the current size of the media buffer and the data reception rate at the client. In another variation, the source representations are not independent of each other and instead represent layered media coding, where, for example, the improved source representation may depend on the base source representation. In this case, there may be a repair expression corresponding to the basic source expression, and another repair expression corresponds to a combination of the basic source expression and the improved source expression.

  The additional overall elements increase the benefits that can be realized by the methods disclosed above. For example, the number of HTTP connections used may vary depending on the current amount of media in the media buffer and / or the receiving rate into the media buffer. Cooperative HTTP requests that use FEC, ie, the FEC method described above and variations of that method, may be actively used when the media buffer is relatively free, eg, more cooperative HTTP requests are made in parallel to different parts of the first fragment, requesting all of the source fragments, and a relatively large part of repair data from the corresponding repair fragment, and then fewer as the media buffer grows Migrate to simultaneous HTTP requests, request more part of media data per request, request smaller pieces of repair data, for example, migrate to one, two, or three simultaneous HTTP requests Move to making requests for the whole fragment or multiple consecutive fragments for each request, and do not request repair data. And move on.

  As another example, the amount of FEC repair data may vary depending on the media buffer size, i.e., if the media buffer is small, more FEC repair data may be requested, and as the media buffer increases, the request The amount of FEC repair data that is played may be reduced, and at some point when the media buffer becomes large enough, repair data may not be required, only data from the source segment of the source presentation may be required. The advantage of such improved techniques is that it enables faster and more stable channel zapping time and greater resilience to possible media jams or stalls, while at the same time request message traffic and FEC repair data. By reducing both the amount of additional bandwidth used beyond the amount that would only be consumed by delivering media in the source segment, and at the same time given network conditions It is possible to support the highest possible media rate for.

Additional improvements when using simultaneous HTTP connections
An HTTP / TCP request may be discarded when appropriate conditions are met, and another HTTP / TCP connection may be made to download data that can replace the requested data in the discarded request, The second HTTP / TCP request is exactly the same data as in the original request, eg source data, or duplicate data, eg part of the same source data and repair data not requested in the first request Or completely disjoint data, for example repair data that was not requested in the first request. Examples of suitable conditions are no response from the Block Server Infrastructure (BSI) within a given time, or failure to establish a transport connection to the BSI, or explicit failure from the server A request fails due to receipt of a message or another failure condition.

  Another example of a suitable condition is an estimate of the expected connection speed or the connection speed required to receive a response before the playback time of the contained media data or another time depending on that time According to the comparison of the connection speed measure with the value (data arrival rate in response to the request in question), data reception is proceeding abnormally slowly.

  This approach has advantages when the BSI fails or its performance may be poor. In this case, the above approach increases the probability that the client can continue to reliably play the media data despite failure or poor performance within the BSI. In some cases, it may be advantageous to design the BSI in such a way as to indicate such failure or bad performance from time to time, for example, such a design may exhibit such failure or bad performance. Costs may be lower than alternative designs that do not show or show less. In this case, the methods described herein have the further advantage of allowing the use of such lower cost designs for BSI without the natural degradation of the user experience.

  In another embodiment, the number of requests made for data corresponding to a given block may depend on whether the appropriate conditions for the block are met. If the condition is not met, the client is constrained to make further requests for the block if successful completion of all currently outstanding data requests for the block allows the block to be restored with high probability. obtain. If the condition is met, more requests for the block may be issued, i.e., the above constraints do not apply. An example of a suitable condition is that the time until the scheduled playback time of the block, or another time depending on that time, falls below a given threshold. This method is advantageous because when the reception of the block is more urgent due to the near playback time of the media data containing the block, an additional request for data for the block is issued. In the case of a common transport protocol such as HTTP / TCP, these additional requests have the effect of increasing the occupancy of available bandwidth dedicated to the data that contributes to receiving the block in question. This reduces the time required to receive enough data to restore and complete the block, thus reducing the probability that the block cannot be restored by the scheduled playback time of the media data containing the block. As described above, the method described herein is advantageous because if the block cannot be restored by the scheduled playback time of the media data containing the block, playback may pause and result in a bad user experience. In particular, it reduces the probability of this bad user experience.

  Throughout this specification, references to the scheduled playback time of a block may be the first time that encoded media data containing the block is available at the client to achieve playback of the presentation without pausing. Please understand that it refers to time. As will be apparent to those skilled in the art of media presentation systems, this time is actually the actual time of appearance of the media, including the blocks in the physical transducers (screens, speakers, etc.) used for playback. It is slightly earlier than that, in order to make the actual playback of the block, some conversion functions may need to be applied to the media data containing the block, and these functions are completed This is because some time may be required. For example, media data is generally transported in a compressed form and decompression conversion can be applied.

A method for generating a file structure that supports a collaborative HTTP / FEC method An embodiment for generating a file structure that can be advantageously used by clients utilizing a collaborative HTTP / FEC method is described herein. Is done. In this embodiment, for each source segment, there is a corresponding repair segment generated as follows. The parameter R indicates on average how much FEC repair data is generated for the source data in the source segment. For example, R = 0.33 indicates that if a source segment contains 1,000 kilobytes of data, the corresponding repair segment contains about 330 kilobytes of repair data. The parameter S indicates the symbol size used for FEC encoding and decoding in bytes. For example, S = 64 indicates that the source data and the repair data include symbols each having a size of 64 bytes for FEC encoding and decoding purposes.

  A repair segment may be generated for the source segment as follows. Since each fragment of the source segment is considered as a source block for the purposes of FEC encoding, each fragment is treated as a source symbol sequence of the source block from which the repair symbol is generated. The total number of repair symbols generated for the first i fragments is calculated as TNRS (i) = ceiling (R * B (i) / S), where ceiling (x) is at least x A function that outputs the smallest integer having a value. Therefore, the number of repair symbols generated for fragment i is NRS (i) = TNRS (i) −TNRS (i−1).

  A repair segment includes a concatenation of repair symbols for fragments, and the order of repair symbols within the repair segment is the order of the fragments from which the repair symbols were generated, and within the fragments, the repair symbols are encoded symbol identifiers (ESI). ) Order. A repair segment structure corresponding to the source segment structure, including the repair segment generator 2700, is shown in FIG.

  By defining the number of repair symbols for a fragment as described above, the total number of repair symbols for all previous fragments, and thus the byte index in the repair segment, is R, S, B (i- Note that it depends only on 1), and B (i), not on any previous or subsequent structure of the fragment in the source segment. This means that the client can quickly calculate the starting position of the repair block in the repair segment, and can quickly calculate the number of repair symbols in the repair block. This is advantageous because it allows only local information about the structure of the fragment to be used. Thus, if the client decides to start downloading and playing a fragment from the middle of the source segment, the client can also quickly generate and access the corresponding repair block from within the corresponding repair segment.

  The number of source symbols in the source block corresponding to fragment i is calculated as NSS (i) = ceiling ((B (i) −B (i−1)) / S). If B (i) -B (i-1) is not a multiple of S, the last source symbol is padded with 0 bytes for FEC encoding and decoding, i.e., the last source symbol is FEC Although the last source symbol is padded to be S bytes in size for encoding and decoding, these zero padding bytes are not stored as part of the source segment. In this embodiment, the ESI of the source symbol is 0, 1,..., NSS (i) -1, and the ESI of the repair symbol is NSS (i),..., NSS (i) + NRS (i) -1. .

  The URL of the repair segment in this embodiment may be generated from the URL of the corresponding source segment, for example, by simply adding the extension “.repair” to the source segment URL.

  As described herein, repair indexing information and repair segment FEC information are implicitly defined by the indexing information for the corresponding source segment and from the values of R and S. The fragment structure including the time offset and repair segment is determined by the time offset and structure of the corresponding source segment. The byte offset for the end of the repair symbol in the repair segment corresponding to fragment i may be calculated as RB (i) = S * ceiling (R * B (i) / S). Then, since the number of bytes in the repair segment corresponding to fragment i is RB (i) -RB (i-1), the number of repair symbols corresponding to fragment i is NRS (i) = (RB (i) -RB Calculated as (i-1)) / S. The number of source symbols corresponding to fragment i may be calculated as NSS (i) = ceiling ((B (i) −B (i−1)) / S). Thus, in this embodiment, repair indexing information for repair blocks in the repair segment, and corresponding FEC information, can be implicitly derived from indexing information for R, S, and corresponding fragments of the corresponding source segment.

  As an example, consider the example shown in FIG. 28 showing fragment 2 starting at byte offset B (1) = 6,410 and ending at byte offset B (2) = 6,770. In this example, the symbol size is S = 64 bytes, and the vertical dotted line indicates the byte offset in the source segment corresponding to a multiple of S. The size of the entire repair segment as a fragment of the source segment size is set to R = 0.5 in this example. The number of source symbols in the source block for fragment 2 is calculated as NSS (2) = ceiling ((6,770-6,410) / 64) = ceil (5.625) = 6, these 6 source symbols are ESIs 0, ..., 5 and the first source symbol is the first 64 bytes of fragment 2 starting at byte index 6,410 in the source segment, and the second source symbol starts at byte index 6,474 in the source segment The next 64 bytes of fragment 2, and so on. The end byte offset of the repair block corresponding to fragment 2 is calculated as RB (2) = 64 * ceiling (0.5 * 6,770 / 64) = 64 * ceiling (52.89…) = 64 * 53 = 3,392, corresponding to fragment 2 The starting byte offset of the repair block to be calculated is calculated as RB (1) = 64 * ceiling (0.5 * 6,410 / 64) = 64 * ceiling (50.07…) = 64 * 51 = 3,264. In the repair block corresponding to fragment 2 with 7 and 7, there are two repair symbols, starting at byte offset 3,264 and ending at byte offset 3,392, respectively, in the repair segment.

  In the example shown in FIG. 28, even if R = 0.5, there are 6 source symbols corresponding to fragment 2, and the number of repair symbols is calculated simply by using the number of source symbols. Note that it is calculated as 2 instead of 3 which can be predicted in some cases, instead according to the method described herein. In contrast to simply using the number of source symbols in a fragment to determine the number of repair symbols, the embodiment described above is based only on index information associated with the corresponding source block of the corresponding source segment. Allows the calculation of the placement of repair blocks within a repair segment. Furthermore, as the number K of source symbols in the source block increases, the number of repair symbols KR in the corresponding repair block is approximated by K * R. That is because, in general, KR is at most ceil (K * R) and KR is at least floor ((K-1) * R), where floor (x) is at most the largest of x It is an integer.

As those skilled in the art will appreciate, there are many variations of the above embodiment for generating a file structure that can be advantageously used by clients utilizing cooperative HTTP / FEC methods. As an example of an alternative embodiment, the original segment for the representation may be partitioned into N> 1 parallel segments, where i = 1,... Fragment F _i is included in the i th parallel segment, and for i = 1,..., N, the sum of F _i is equal to 1. In this embodiment, there is one master segment map used to derive the segment map for all of the parallel segments, similar to the way the repair segment map is derived from the source segment map in the embodiment described above. possible. For example, the master segment map can show the fragment structure if all of the source media data is not partitioned into parallel segments but instead contained in one original segment, then the segment map for the i th parallel segment If the amount of media data in the first prefix of the original segment fragment is L bytes, the total number of bytes in this prefix is ceil (L * G _i ) across the first i parallel segments. By calculating something, it can be derived from the master segment map, where G _i is the sum of F _j over j = 1,. As another example of an alternative embodiment, a segment may consist of a combination of the original source media data for each fragment, immediately followed by repair data for each fragment, from the source media data and its source media data Resulting in a segment containing a mixture of repair data generated using FEC codes. As another example of an alternative embodiment, a segment that includes a mixture of source media data and repair data may be partitioned into multiple parallel segments that include a mixture of source media data and repair data.

Methods for Handling Low Latency Streaming In some deployment scenarios, low latency streaming for live services may be desirable. For local in-situ distribution of events, such as sporting events or concerts, for example, it is desirable that the delay between the live action and the presentation of the live service on the client is as short as possible. For example, a delay of up to 1 second may be desirable.

  As described above, it may be advantageous to arrange each file storing a segment of a media presentation to start at a random access point (RAP). Some profiles, specifically live profiles in ISO-based media file formats, require each media segment to start with a RAP.

  However, in environments where low end-to-end latency delivery is required, the duration of each segment should be short to minimize the delay between live action and live event presentation on the client. I must. It is desirable to avoid inserting a RAP in each segment that should be used for low latency streaming. For example, RAP in video is usually realized by IDR frames. Encoding efficiency can be improved by avoiding the use of IDR frames within the short segments desirable for low latency streaming.

  According to an embodiment, a live profile compliant representation and a low latency representation of the media presentation are generated. Expressions that conform to the live profile have a relatively long media segment duration. Each media segment of the representation that conforms to the live profile has a RAP at the beginning of the media segment. The low latency representation has relatively short segments (which may be referred to as “media fragments”) that may not include RAP. Clients that support low-latency streaming can receive media fragments that are generated for the low-latency representation of the media presentation, but clients that do not support low-latency streaming can use a representation that conforms to the media presentation's live profile. It may be possible to receive media segments generated for the purpose.

  FIG. 30 shows the relationship between media fragments and media fragments for low latency streaming. Media segment 3002 generated for live profile streaming includes RAP 3004 at the beginning of the media data ("mdat"). In contrast, of media fragments 3004, 3006, and 3008 generated for low latency streaming, only media fragment 3004 contains a RAP.

  Media fragments are generated on the fly and can be downloaded by the client via HTTP. Media fragments can be stored in media segments that conform to a live profile of an ISO-based media file format without any modification to the required media fragments. For example, media fragments can be concatenated into media segments.

  Both media segments and media fragments can be created using the same encoding process. In this way, media can be efficiently encoded for consumption by clients operating in environments that require low latency between terminals and clients using protocols that require RAP in each segment. .

  In some embodiments, a segment index (SIDX) is generated for each media fragment. SIDX may include the presentation time range within the media segment and the corresponding byte range of the media segment that is dedicated by the media fragment. In some embodiments, SIDX indicates whether RAP is present in the fragment. In FIG. 30, the contains_RAP field of the SIDX box of the media fragment 3004 is set to 1, indicating that the media fragment 3004 includes a RAP. The contains_RAP field of the SIDX box for media fragments 3006 and 3008 is set to 0, indicating that media fragments 3006 and 3008 do not contain a RAP. SIDX may further indicate the presentation time of the first RAP in the fragment.

  According to an embodiment, the media server can generate fragments for low latency streaming and push the fragments to the cache. The cache can concatenate the fragments to generate live profile compatible media segments. After the media segment is created, the cache can purge the concatenated media segments to create the media segment.

  A single media presentation description (MPD) may store information about a first representation having a live profile conforming media segment of the media presentation and a second representation having a media fragment of a low latency stream. Time-shifted viewing can be achieved using media segments for time-shifting buffering and media fragments for processing viewing at the end of the streaming near live. The client can switch to these representations by moving closer to the live end, for example by starting with a time shift buffer and skipping a section of the media presentation. Each representation of the MPD may be assigned an attribute to represent an array of representations available for a single media presentation.

  In an MPD that stores information about a first representation with media segments and a second representation with media fragments, it is advantageous to provide information indicating which media fragments of the second representation start with a RAP. possible. For example, the MPD may include an attribute to indicate the frequency of occurrence of RAP within multiple media fragments. In one embodiment, the MPD includes an attribute that indicates the frequency with respect to the number of fragments (ie, one media fragment contains a RAP for every x media fragments). In another embodiment, the attribute indicates the frequency with respect to the temporal distance between adjacent RAPs.

  Alternatively, information about the media fragment may be stored in the first MPD and information about the media segment may be stored in the second MPD.

  In some embodiments, the MPD may signal certain parameters applicable to a particular representation, such as the maximum duration of a presentation media segment or media fragment.

  After reading this disclosure, one of ordinary skill in the art will recognize additional embodiments. In other embodiments, advantageously, combinations or sub-combinations of the inventions disclosed above may be made. It should be understood that exemplary configurations of components are shown for illustrative purposes, and that combinations, additions, reconfigurations, etc. are contemplated in alternative embodiments of the invention. Thus, although the invention has been described with respect to exemplary embodiments, those skilled in the art will recognize that many modifications are possible.

  For example, the processes described herein may be implemented using hardware components, software components, and / or any combination thereof. In some cases, software components may be provided on a tangible non-transitory medium for execution on hardware that is provided in or separate from the media. The specification and drawings are accordingly to be regarded as illustrative rather than restrictive. However, various modifications and changes may be made without departing from the broader spirit and scope of the invention as set forth in the claims below, and the invention is intended to be within the scope of the following claims. It will be apparent that modifications and equivalents are intended to be included.

100 block streaming system
101 block serving infrastructure
102 content
103 Content preparation (media capture system)
104 HTTP streaming server
106 HTTP cache
108 HTTP streaming client
110 Content storage device
112 request
114 request
122 network
123 block selector
124 block requester
125 block buffer
126 Buffer monitor
127 Media decoder
128 media transducer
300 buses
302 uptake system
304 memory
306 Disk storage device
308 video display
310 Alphanumeric input device
312 Network interface
400 bus
402 Client processor
404 memory
406 Disk storage device
408 video display
410 Alphanumeric input device
412 Network interface
500 MPD
501 period recording
502 expression record
503 segment information
504 Initialization segment
505 Media segment
510 Source segment
512 repair segment
700 simple index
712 Hierarchical index
900 metadata table
902 HTTP streaming client
904 blocks
906 HTTP streaming server
1000 video streams
1200 transmitter
1202 metadata
1204 Scalable Layer 1
1206 Scalable Layer 2
1208 Scalable Layer 3 (not fully received)
1210 receiver
1212 Media presentation
3002 Media segment
3004 Media fragment
3006 Media fragment
3008 Media fragment

Claims (14)

A method for structuring content data to be provided using a media server, comprising:
Obtaining the content to be provided;
Generating a plurality of media segments that are encoded according to an encoding protocol that represents the content and includes one or more frames of a media presentation encoded into each media segment, the random access point being Generating steps available in each media segment;
Generating a plurality of media fragments encoded according to the encoding protocol, wherein the media segment includes the plurality of media fragments, at least some of the plurality of media fragments include random access points, and at least Some do not include a random access point, and the random access point includes a position in the segment where a decoder can decode a fragment that follows the random access point independently of the previous fragment of the random access point; and
Generating a segment index for the media fragment;
Including
Whether the segment index is a presentation time range for each media fragment in the media segment, a corresponding byte range in the media segment that each media fragment occupies, and whether a random access point is present in each media fragment; show, a random access point exists indicator seen including,
The media segment is a live profile conforming media segment generated by concatenating the plurality of media fragments for low latency streaming;
The method further stores information regarding a first representation of the media representation having the plurality of live profile conforming media segments and a second representation of the media representation having the plurality of media fragments of a low latency stream. Including generating a single media presentation description (MPD) file,
Method. Generating the media segment in a cache, wherein after the media segment is generated in the cache, the plurality of media fragments used to generate the media segment are from the cache; The method of claim 1 , wherein the method is purged. The MPD file includes an attribute to indicate the frequency of occurrence of the random access point in the second representation method according to claim 1. 4. The method of claim 3 , wherein the frequency is a period. 4. The method of claim 3 , wherein the frequency is the number of media fragments.   The method further comprises: determining a position of the plurality of media fragments relative to the random access point, wherein the random access point is disposed at a variable non-fixed position of the plurality of media fragments. The method described in 1. A media server,
Get the content to be provided,
Generating a plurality of media segments representing the content and encoded according to an encoding protocol that includes one or more frames of the media presentation encoded into each media segment;
Generating a plurality of media fragments encoded according to the encoding protocol;
Generating a segment index for the media fragment;
With a processor configured to
A random access point is available in each media segment,
The media segment includes the plurality of media fragments, at least some of the plurality of media fragments include random access points, at least some do not include random access points, and a random access point precedes the random access point. A position in the segment where a decoder can decode a fragment following the random access point independently of the fragment of
Whether the segment index is a presentation time range for each media fragment in the media segment, a corresponding byte range in the media segment that each media fragment occupies, and whether a random access point is present in each media fragment; show, a random access point exists indicator seen including,
The media segment is a live profile conforming media segment generated by concatenating the plurality of media fragments for low latency streaming;
The processor further stores information regarding a first representation of the media representation having the plurality of live profile conforming media segments and a second representation of the media representation having the plurality of media fragments of a low latency stream. Configured to generate a single media presentation description (MPD) file,
Media server. The processor is further configured to generate the media segment in a cache, and the plurality of media used to generate the media segment after the media segment is generated in the cache The media server of claim 7 , wherein fragments are purged from the cache. 8. The media server according to claim 7 , wherein the MPD file includes an attribute for indicating a frequency of occurrence of random access points in the second representation. 10. The media server according to claim 9 , wherein the frequency is a period. 11. A media server according to claim 10 , wherein the frequency is the number of media fragments. The processor is further configured to determine a position of the plurality of media fragments with respect to the random access point, the random access point further comprising a step of being located at a variable non-fixed position of the plurality of media fragments. The media server according to claim 7 . When executed, one or more computing devices
Get the content to be provided,
Generating a plurality of media segments representing the content and encoded according to an encoding protocol that includes one or more frames of the media presentation encoded into each media segment;
Generating a plurality of media fragments encoded according to the encoding protocol;
Generating a segment index for the media fragment;
One or more non-transitory computer-readable storage media storing computer-executable instructions comprising:
A random access point is available in each media segment,
The media segment includes the plurality of media fragments, at least some of the plurality of media fragments include random access points, at least some do not include random access points, and a random access point precedes the random access point. A position in the segment where a decoder can decode a fragment following the random access point independently of the fragment of
Whether the segment index is a presentation time range for each media fragment in the media segment, a corresponding byte range in the media segment that each media fragment occupies, and whether a random access point is present in each media fragment; show, a random access point exists indicator seen including,
The media segment is a live profile conforming media segment generated by concatenating the plurality of media fragments for low latency streaming;
The instructions further relate to the device a first representation of the media representation having the plurality of live profile conforming media segments and a second representation of the media representation having the plurality of media fragments of a low latency stream. Generate a single media presentation description (MPD) file to store the information,
Non-transitory computer readable storage medium. When executed, the computer-executable instructions cause the one or more computing devices to determine a position of the plurality of media fragments relative to the random access point, the random access point being the plurality of media fragments. 14. The non-transitory computer readable storage medium of claim 13 , further comprising the step of: being located at a variable non-fixed location.

Images