JP6059820B2 - Low latency streaming - Google Patents

Description

  The present invention relates to low latency streaming, and in particular, a method and system that enables low latency streaming of segments to a client, a configuration that enables low latency streaming to a client, and a client for use in such a system And, without limitation, database structures and computer program products for using such methods.

Conventional technology

  As Internet video and Internet TV become increasingly popular, there is an increasing need for adaptive streaming solutions that enable continuous playback and best user experience under changing network conditions. The concept of adaptive streaming is based on the philosophy of adapting the bandwidth required by a video stream to the bandwidth available on the network path between the streaming source and the client. Bandwidth is adapted by changing the bit rate (ie quality).

  Currently, a number of HTTP-based adaptive streaming (HAS) protocols have been developed, and the best practices of these solutions are condensed into the ISO standard ISO / IEC 23001-6, which is currently evolving. This is called MPEG Dynamic Adaptive Streaming over HTTP (MPEG DASH: MPEG HTTP Dynamic Adaptive Streaming). In the HAS solution, the content stream is typically partitioned into segments (also called “chunks”), and each of these segments can be encoded with a different quality level (representation). A content distribution network (CDN) is typically used to efficiently distribute segments to the majority of content processing devices.

  The segments and their different representations are described in a so-called manifest file. The manifest file may contain information about the segments in the stream (segment identifier, location, play-out time, etc.) and information about the temporal relationship between different segments in the stream. A client at the content processing device can use the manifest file to request a segment from the network and process the segment for playout. The client can be configured to switch between different representations depending on network conditions.

  MPEG DASH and other adaptive streaming solutions have been developed for delivery over unmanaged (best effort) networks and the Internet. In order to deal with unexpected jitter and congestion, and to reduce the risk of buffer underruns, the buffering performed at the client is the total end-to-end delay between the content processing device source and playout. Compared to the amount.

  A HAS client typically uses a playout buffer (preconfigured) for at least three complete segments before playout begins. This buffer grows linearly with the segment size and thus easily reaches over 30 seconds. In addition, to reduce the risk that not enough segments are available to fill the playout buffer, the client will be required by the client (when subscribing to a streaming event) It is configured to be a segment made available by the streaming source earlier (typically 3 segments earlier) than the segment made available by the streaming source when subscribing to the streaming event.

  Therefore, between the playout of live streaming events by the HAS client and the playout of live streams by other content processing devices based on other transport mechanisms such as traditional broadcast or multicast streaming. There is considerable latency (delay).

  Even for the distribution of content on a management network (eg TV channel), in particular live content, the content is received by different transport mechanisms (eg DVB-S, DVB-C, radio and MPEG DASH) It is desirable to have a playout time of approximately the same content on different content processing devices (eg, TV, STB, tablet, smart phone, etc.) that can be.

  Typically, synchronized playout between different content processing devices can be achieved by inter-destination media synchronization (IDMS) techniques. IDMS is based on delaying the playout of all receivers to the most delayed receiver so that synchronous playout is achieved.

  The problem with delaying all other receivers to match the HAS streaming device is that in the current HAS implementation, the playout delay can be on the order of 10 seconds. Thus, achieving synchronous playout means that all other devices must be delayed by the same amount. However, delaying media signals by 10 seconds in a wide variety of devices can cause unpredictable problems. For example, a regular set top box (STB) configured to receive and decode DVB signals may not have enough memory to store a DVB stream for as long as 30 seconds. Thus, an IDMS synchronized social TV service (eg, “view and watch together”) may not be feasible between a legitimate STB and a HAS client.

  In addition, in live streaming applications and streaming applications that allow user interaction, it is desirable to have the maximum allowable end-to-end delay for the delivery and presentation of media signals, as may be prescribed by law. There is even a thing.

  From the above, the playout delay introduced by HAS streaming results in a significant degradation of the user experience and hinders large-scale commercial use of HAS streaming.

  Accordingly, there is a need in the art for an improved method and system for low latency adaptive streaming that provides an optimal low latency playout while providing a high user experience.

  In particular, there is a need in the art for methods and systems that enable low latency adaptive streaming of content for heterogeneous devices and delivery schemes.

The object of the present invention is to reduce or eliminate at least one of the disadvantages known in the prior art. In one aspect, the present invention may relate to a method for enabling low latency streaming of segments at a client processing device over a network to a client, preferably a HAS (HTTP-based adaptive streaming) client. The client is configured to request and receive a segment from the server system based on the manifest file. The method includes: a monitoring system in the network collecting quality metrics associated with one or more paths between the client and one or more streaming servers in the server system; Storing in a quality database in the network, preferably a client access-line quality database (CAQD), and / or at least a part of the stored quality metric, or the stored quality metric in the content processing device. Providing quality of service information determined based on at least a portion of and / or one or more configuration parameters;
And / or based on at least part of the quality metric or the quality of service information or the configuration parameters, a configuration module in the content processing device is configured to provide a buffer in the content processing device, preferably a playout buffer, and / or Or configuring a segment request function in the content processing device.

  The present invention includes a client (included in a content processing device) containing quality metrics associated with one or more streaming paths between the client and one or more streaming servers in the server system. Allows access to quality database. Quality metrics are measured network parameters, including home or residential network parameters, and measured device parameters collected by the monitoring system and stored in a quality database. The quality metrics can be directly accessed by the content processing device (eg, supplied / sent to the content processing device) and used by the configuration module. Alternatively, the expected QoS level can preferably be determined based on the history of (at least part of) the collected quality metrics. This QoS level can also be determined in the network, preferably by a monitoring system or other network entity that has access to quality metrics. Subsequently, this QoS level can be sent to the content processing device. Alternatively, the quality metric may be supplied (transmitted) from the network to the client, in which case this QoS level may be determined by the client based on the received quality metric. This information (eg, QoS level) is used by the HA client (via the configuration module) to configure the buffer and / or segment request functions when the HA client wants to subscribe to a (live) streaming event. May be. Alternatively, the quality metric may be used directly by the configuration module to configure the buffer or segment request function without going through an intermediate step to determine the expected QoS level.

  When content is streamed through a managed network (eg, bandwidth is guaranteed), there is less congestion and presence compared to situations where content is streamed through an unmanaged network (such as the public Internet) The variation in latency is also reduced. When streaming content over a managed network, large buffer size settings to avoid buffer underruns are therefore no longer needed. The QoS level indicates the quality (eg, stability) of the streaming path used to stream segmented content.

  Thus, when the quality metric of the streaming path indicates that a certain QoS level can be expected, the buffer size at the content processing device can be reduced (or increased) and the HAS client can stream for playout. Can request the first segment that is relatively close (or relatively far) to the segment that was available at the time of joining the session. In this way, the playout delay difference between content processing devices using HAS clients and other streaming clients, eg content processing devices using DVB in high QoS level networks, is greatly reduced. be able to.

  In one embodiment, the method may include determining the one or more configuration parameters for the configuration module based on the at least part of the quality metric.

  In one embodiment, the one or more configuration parameters may include at least one buffer size parameter that determines a size of the buffer before data playout in the buffer is initiated. In one embodiment, the minBufferTime parameter as defined in the MPEG DASH standard can be used to determine the size of the buffer.

  In one embodiment, the one or more configuration parameters are selected from segments identified in the manifest file, and at least one segment that determines a first segment that the segment request function requests from the streaming server It may include a request parameter, preferably a segmentStartOffset parameter.

  The quality metric associated with the streaming path may represent (or indicate) some QoS level of the streaming path, depending on the configuration module at the client device (ie, content processing device), the buffer size and / or segment. It can also be used to determine (configuration) parameters that can be used to configure the request function. Since the minBufferTime parameter is a known parameter in MPEG DASH, the present invention can be easily realized based on a conventional HAS client (device).

  In one embodiment, the method includes determining quality of service information based on the quality metric, wherein the quality of service information defines one or more QoS levels, and the QoS level is (( Associated with one or more predetermined configuration parameters (used by In an embodiment, the one or more QoS levels are at least a low latency level associated with one or more (preconfigured) configuration parameters for configuring the HAS client in a low latency mode (ie, Small buffer size, small segment offset start) and one or more (pre-configured) to configure the client in high latency ["normal"] mode (ie, large buffer size, large segment offset start) And a high latency level (also referred to as “normal” mode) associated with the configuration parameter. Thus, the quality of service information may be associated with different sets of configuration parameters that may be preconfigured at the client. In this way, certain (low, medium, high) latency modes of the HAS client can be selected by sending a message containing the expected QoS level (mode) to the configuration module.

  In one embodiment, at least some of the one or more configuration parameters may be determined by the monitoring system and stored in the quality database.

  In another embodiment, the quality of service information may be determined by the monitoring system and stored in the quality database. In other embodiments, the quality metric, the one or more configuration parameters, and / or the at least part of the quality of service information may be sent to the content processing device. In these embodiments, the monitoring system may be configured to determine one or more configuration parameters or quality of service information based on the quality metric. Alternatively, other network entities that may be remote entities from the monitoring system (eg, streaming servers, network nodes that include request derivation functions, generate or update manifest files, and / or content The network node responsible for setting up the HAS control channel to the processing device) may determine these parameters based on quality metrics stored in the database. These configuration parameters can be sent to the client. Processing time on the client side can be saved by saving processing power on the client side by processing (eg, a monitoring server) of (part of) the quality metrics in the network.

  In one embodiment, the method includes the client requesting the server system for at least a portion of a manifest file or a manifest file update; the server system comprising the quality metric, the one or more Retrieving the configuration parameters and / or the at least part of the quality of service information from the quality database; the server system comprising the quality metrics and / or the one or more configuration parameters; and / or It may also include at least one of sending at least a portion of a manifest file that includes the at least a portion of quality of service information to the client. Thus, quality metrics, QoS information, and / or configuration parameters can be sent to the client in a manifest file. Thus, when the HAS client needs to be configured, i.e., when the client wants to join a streaming session, this information is sent to the client.

  In one embodiment, the quality metric, the one or more configuration parameters, and / or at least a portion of the quality of service information are sent to a client or configuration module over another communication channel. In an embodiment, a communication channel may be established between monitoring systems (or a quality database associated with a monitoring system) based on location information associated with the monitoring system in a manifest file, eg, a URL or URI. . Alternatively, such a communication channel may be established between the content processing device and other network entities (nodes) configured to communicate with the content processing device.

  In one embodiment, the communication channel may be a (HAS) streaming control channel, preferably a Websocket based streaming control channel. In an embodiment, the method provides the client with channel setting information for setting a (bidirectional) streaming control channel between the server system and the client, wherein the streaming control channel is Preferably, a Websocket streaming control channel may be included, and the (bidirectional) streaming control channel may be established based on the channel setting information. An HTTP-based (Websocket) streaming channel allows servers in the network to send HAS client messages, eg, manifest update requests or quality of service requests, over the streaming path during segment streaming. Further, in an embodiment, the HAS streaming control channel may be configured based on channel setting information sent to the client via a manifest file.

  In one embodiment, the method includes at least a first monitoring agent in the content processing device collecting a first metric associated with the content processing device, and / or at least a second monitoring agent in the network, Collecting a second metric associated with at least a portion of the network, and based on the first and / or second metric, the monitoring system includes the one or more streaming servers in the server system Determining a quality metric associated with one or more paths between the client and the client, and storing the quality metric in the quality database.

  Thus, a point-to-point monitoring system configured to collect real time QoS (Quality of Service) and QoE (Experience Quality) metrics (hereinafter referred to as quality metrics) from clients and networks can be used. When the monitoring system is deployed in an Internet Service Provider (ISP) network, quality is determined to determine quality of service information associated with one or more streaming paths between the client and one or more streaming servers. Metrics can be used.

  The monitoring system distinguishes between the different devices used in the home network, thereby losing source packets in this home network, home network load fluctuations, terminal capabilities, bandwidth available in the home network. It can be configured such that the width can be taken into account. In this way, different HAS streaming devices connected to the same home gateway, for example wireless mobile devices such as televisions and electronic tablets, can be configured based on different quality metrics.

  In one embodiment, the manifest file may also include information regarding one or more latency modes (eg, low, medium, and / or high, or alternatively “low” and “normal”). This information is available to clients. In one embodiment, each latency mode may be associated with a set of configuration parameters.

In yet another aspect, the present invention can relate to a content delivery system that enables low latency streaming of a segment to a content processing device over at least one network. The system is a content processing device including a client, preferably a HAS client, wherein the client is configured to request and receive segments from one or more streaming servers based on a manifest file. , Content processing devices,
The content processing device, preferably a client at the client processing device, is further based on at least a portion of the stored quality metric, or at least a portion of the stored quality metric and / or one or more configuration parameters. Configured to provide service quality information determined by
A monitoring system configured to collect quality metrics associated with one or more paths between the client and the one or more streaming servers and store the quality metrics in a quality database in the network; ,
Configuring a buffer, preferably a playout buffer, at the content processing device based on at least a portion of the quality metric and / or the quality of service information and / or the configuration parameters; and A configuration module that configures a segment request function in the content processing device;
Can be included.

  In yet another aspect, the invention may relate to a configuration module for use in a content processing device, wherein the configuration module provides low latency streaming to a client, preferably a HAS client, in the content processing device. Configured to enable, and wherein the client is configured to request and receive segments from one or more streaming servers in the server system based on a manifest file. The configuration module is further based on quality of service information determined based on a quality metric and / or based on at least a portion of the stored quality metric and / or based on one or more configuration parameters. Configuring a buffer, preferably a playout buffer, and / or a segment request function in the content processing device, wherein the quality metric is one or more in the client and the server system. Associated with one or more paths to and from a streaming server, the quality metrics are configured to be collected by a monitoring system in the network and stored in a quality database in the network.

  In one embodiment, the buffer is configured to determine the size of the buffer before data playout is initiated in the buffer based on one or more configuration parameters, preferably the minBufferTime parameter. And / or the segment request function is selected from the segments identified in the manifest file based on one or more configuration parameters, preferably a segmentStartOffset parameter, and the segment request function is the streaming • configured to determine the first segment to request from the server;

  In one aspect, the present invention can relate to a monitoring system that monitors quality metrics associated with a streaming path in a network between a HAS client and a streaming server in a content processing device. The monitoring system collects device metrics associated with the content processing device and network metrics associated with at least a portion of the network from one or more monitoring agents; the device; and Means for determining a quality metric associated with a streaming path based on a network metric; and determining one or more configuration parameters for a configuration module in the content processing device based on the at least part of the quality metric. Preferably, the one or more configuration parameters include at least one buffer size parameter for determining a size of the buffer before data playout in the buffer is initiated. Is selected from the minBufferTime parameter and / or the segment identified in the manifest file, and preferably at least one segment request parameter for determining a first segment that the segment request function requests from the streaming server, preferably Includes means including a segmentStartOffset parameter.

In one aspect, the invention can relate to a data structure, preferably at least a portion of a manifest file for use by a client at a content processing device, wherein the client is based on at least one based on the manifest file. Configured to request and receive segments from one server, the data structure enabling low latency streaming to the client, the data structure comprising one or more segment identifiers and segment playout information; The data structure further comprises:
Quality of service information including one or more QoS levels associated with a quality metric and / or a streaming path between the client and a streaming server, the quality metric and / or the quality of service information Allowing a configuration module associated with a client to determine one or more configuration parameters for a buffer, preferably a playout buffer and / or a segment request function in the content processing device, and / or the data structure Further includes one or more configuration parameters associated with a streaming path between the client and the streaming server, wherein the one or more configuration parameters are at the content processing device, Serial configuration module associated with the client, the buffer preferably allows configuring the playout buffer and / or segments required functions.

  The present invention may also relate to a program product, wherein the computer program product is configured to perform method steps as described above when executed in a computer memory. Including parts. The invention will be further illustrated with reference to the accompanying drawings, which schematically show embodiments according to the invention. It should be understood that the present invention is not limited to these specific embodiments.

FIG. 1 shows a schematic diagram of a content distribution system according to an embodiment of the present invention. FIG. 2 shows a schematic diagram of a manifest file according to an embodiment of the present invention. FIG. 3 illustrates a process for updating a manifest file according to an embodiment of the present invention. FIG. 4 illustrates a process for delivering segments to clients according to one embodiment of the present invention. FIG. 5 shows a schematic diagram of a content distribution system according to another embodiment of the present invention. FIG. 6 shows a schematic diagram of a manifest file according to still another embodiment of the present invention. FIG. 7 illustrates a process for delivering segments to clients according to another embodiment of the present invention. FIG. 8 illustrates a process for collecting metrics according to an embodiment of the present invention. FIG. 9 illustrates a protocol flow between a content processing device and a server according to an embodiment of the present invention. FIG. 10 shows a protocol format for reporting metrics according to an embodiment of the present invention. FIG. 11 illustrates another protocol format for reporting metrics according to another embodiment of the present invention. FIG. 12 illustrates yet another format for reporting metrics according to an embodiment of the present invention. FIG. 13 shows a schematic diagram of a manifest file containing quality of service information according to yet another embodiment of the present invention. FIG. 14 illustrates a protocol flow for requesting quality of service information based on the ALTO protocol according to an embodiment of the present invention.

  As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention are generally hardware embodiments, generally software embodiments (including firmware, resident software, microcode, etc.), or embodiments that combine software and hardware aspects. Which can all be collectively referred to herein as “circuitry”, “module”, or “system”, “node”. Further, aspects of the invention may take the form of a computer program product embodied in one or more computer readable media embodying computer readable program code.

  Many of the functional units described in this specification have been referred to as modules, to more particularly emphasize their implementation independence. For example, a module can be implemented as a hardware circuit that includes custom VLSI circuits or gate arrays, semiconductors that can be bought at any time, such as logic chips, transistors, or other discrete components. Modules can also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, and the like.

  Modules can also be implemented in software for execution by various types of processors. An identified module of computer readable program code can include, for example, one or more physical or logical blocks of computer instructions that can be organized, for example, as objects, procedures, or functions. However, the identified module executables do not have to be physically located together, they can also contain foreign instructions stored in different locations, and when they are logically collected together, Configure a module and achieve the stated purpose for that module.

  Indeed, a module of computer readable program code may be one instruction or many instructions, even distributed across different memory segments, across different programs, between different programs, and across different memory devices. it can. Similarly, operational data can be identified and illustrated herein within a module, embodied in any suitable form, and organized in any suitable type of data structure. The operational data can be collected as a single data collection, or can be distributed across different locations including different storage devices, or can exist at least in part as a mere electronic signal on the system or network. If the module or part of the module is implemented in software, the computer readable program code may be stored and / or propagated in one or more computer readable medium (s).

  The computer readable medium may be a tangible computer readable storage medium that stores computer readable program code. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor system, apparatus, or device, or any suitable combination of the above, It is not limited to.

  More specific examples of computer readable media include portable computer diskettes, hard disks, random access memory (“RAM”), read only memory (“ROM”), erasable programmable read. Only memory ("EPROM" or "flash memory"), portable compact disk read only memory ("CD-ROM"), digital versatile disk ("DVD"), optical storage device, Magnetic storage devices, holographic storage media, micromechanical storage devices, or any suitable combination of the above can be included, but are not limited to. In the context of this document, a computer readable storage medium may contain and / or store computer readable program code for use and / or associated with an instruction execution system, apparatus, or device. Any tangible medium may be used.

  The computer readable medium may be a computer readable signal medium. A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms including, but not limited to, electronic, electromagnetic, magnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, and communicates, propagates, or transports computer-readable program code for use by or in connection with an instruction execution system, apparatus, or device Any computer-readable medium that can be used. Computer readable program code embodied on a computer readable signal medium includes wireless, wireline, fiber optic cable, radio frequency (“RF”), etc., or any suitable combination of the foregoing. Any suitable medium can be used for transmission, without limitation.

  In one embodiment, the computer readable medium may include a combination of one or more computer readable storage media and one or more computer readable signal media. For example, computer readable program code can be propagated as an electromagnetic signal through a fiber optic cable for execution by a processor and stored in a RAM storage device for execution by the processor.

  Throughout this specification "one embodiment," "an embodiment," or similar language, when referring to a particular feature, specific features described in connection with that embodiment, A structure, or characteristic, is meant to be included in at least one embodiment. That is, throughout this specification, when the phrase “in one embodiment”, “in an embodiment”, or similar language appears, all may refer to the same embodiment, but not necessarily explicitly. Otherwise, it means “one or more embodiments, but not all”. The terms “including”, “comprising”, “having” and variations thereof mean “including but not limited to”, unless expressly specified otherwise. A list that lists items does not imply that any or all of these items are mutually exclusive unless explicitly specified otherwise. The terms “a”, “an”, and “the” also refer to “one or more”, unless expressly specified otherwise.

  Furthermore, the described features, structures, or characteristics of the embodiments may be combined in any suitable manner. In the following description, programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to obtain a full understanding of the embodiments A number of specific details are provided, such as However, one of ordinary skill in the art will appreciate that embodiments may be practiced without one or more of these specific details or using other methods, components, materials, and the like. On the other hand, well-known structures, materials, or operations have not been shown or described in detail to avoid obscuring aspects of the embodiments.

  Aspects of embodiments are described below with reference to schematic flow diagrams and / or schematic block diagrams of methods, apparatus, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the schematic flowchart and / or schematic block diagram, and combinations of blocks in the schematic flowchart and / or schematic block diagram, can be implemented by computer readable program code. These computer readable program codes can be provided to a processor of a general purpose computer, special purpose computer, sequencer, or other programmable data processing device to generate a machine, and the instructions are transmitted to a computer or other computer When executed by a processor of a programmable data processing apparatus, means are formed for performing functions / acts specified in one or more blocks of the schematic flow diagram and / or schematic block diagram. .

  FIG. 1 shows a schematic diagram of a content distribution system according to an embodiment of the present invention. Specifically, FIG. 1 shows a schematic diagram of a content distribution system including a server system 102. Server system 102 is configured to stream content to at least one client 104 at content processing device 106. Server system 102 may include one or more streaming servers. The streaming server is configured to receive one or more content signals 108, eg, (live) video and / or multimedia signals, and process the content signals to generate a segmented content stream. .

  As described in further detail below, the generation of a segmented content stream may include dividing the content signal into separate segments of a predetermined size, and each segment can be encoded into a number of different representations. Here, the representation may relate to different variants of the content signal, including 2D and 3D formats, different video and / or audio qualities (eg SD / HD, bit rate, etc.), different spatial resolutions, etc. it can.

  Generate one or more manifest files (also known as media presentation description or MPD in MPEG-DASH, also known as M3U8 playlist in Apple HTTP live stream) during segmented content stream generation can do. Here, the term “manifest file” can generally refer to a specialized data structure that can include a segment identifier (descriptor) that identifies a content item, eg, a segment that forms a video title. In addition, this may also include location information of the (set of) network node (s), eg, (media) streaming server (s). The streaming server can be configured to deliver the segment to the client or to provide information to the client and retrieve the segment there. The manifest file may further include segment playout information for determining a (core) relationship between the segment and the media metadata. Media metadata includes information about media characteristics, such as video resolution, bit rate, and the like.

  Manifest files and segments associated with different representations can be stored in place in a server system, eg, segment storage node 110 and manifest storage node 112. In an embodiment, the location of the manifest file can be stored as a URL or a URI. When requesting a particular content item, the server system can supply the client with the manifest file URL or URI.

  When a user subscribes to a (live) streaming event or wants to start a streaming service, the event or service can be selected from a web page. After selection, the (HTTP) streaming function 116 at the server system can send a manifest file associated with the streaming event or service to the client over the network connection 114. In an embodiment, the streaming function can be implemented on an HTTP server in the server system.

  The client parses the manifest file and uses the information in the manifest file to request a segmented stream from the server stream's streaming server and configure it to receive segments sent to the client over the network Can do. During streaming, packets associated with a segment can follow one or more paths in the network. Such a route may be hereinafter referred to as a “streaming path”. The streaming path between the client and the streaming server can be established based on the HTTP / TCP protocol.

  To that end, the server system can include an HTTP server. The HTTP server is configured to send the segment to the client based on the HTTP Adaptive Streaming (HAS) protocol. In such a scheme, a client (HAS client) can send an HTTP request for a segment to the HTTP server, which responds by sending one or more segments of a particular representation in the HTTP response message. Send to client. During streaming of the segment to the client, the segment is buffered in the client receive buffer 118. Segment data (frames) encoded and encapsulated in segments can be unpacked, decoded, and arranged in playout buffer 120. The data in the playout buffer can be arranged according to the presentation timeline of the content stream. Subsequently, the media engine 122 can display the data in the playout buffer to the user.

  Examples of adaptive streaming protocols include Apple HTTP Live Streaming (http://tools.ietf.org/html/draft-pantos-http-live-streaming-07], Microsoft Smooth Streaming ( (Microsoft Smooth Streaming) [http://www.iis.net/download/ SmoothStreaming], Adobe HTTP Dynamic Streaming (http://www.adobe.com/products/httpdynamicstreaming), 3GPP -DASH [TS26.247 Transparent end-to-end Packet-switched Streaming Service (PSS), Progressive Download and Dynamic Adaptive Streaming over HTTP (Progressive download and dynamic adaptation over HTTP Streaming), and MPEG Dynamic Adaptive Streaming over HTTP [MPEG DASH ISO / IEC 23001-6]. HTTP enables an efficient, firewall-friendly and scalable method for delivering segments to clients.

  Content processing devices are typically (mobile) content playout devices such as electronic tablets, smart phones, notebooks, media players, home gateways, or DASH enabled devices such as DASH enabled HbbTV display devices. It only has to relate to the device. Alternatively, the content processing device is a set top box configured to process and temporarily store the content for future consumption by a content playout device that can access the stored content Alternatively, it may be a content storage device.

  Similarly, a server system can include one or more streaming servers, including one or more HTTP streaming servers. Alternatively, the server system may include (part of) one or more content distribution networks (CDNs). A CDN (not shown) may include one or more edge nodes (also called distribution nodes or surrogate nodes) and at least one central CDN node. A central CDN node is a content origin function (COF) that controls the collection of content from an external source (eg, content provider or other CDN) to the CDN, and one or more copies of the segment And a CDN control function (CDNCF) (a process also known as request routing) that controls the distribution of the client to the appropriate distribution node and redirects the client to the appropriate distribution node. A content location database can also be used to store information about where segments are stored in the CDN (eg, which distribution node and which folder in the distribution node).

  In one embodiment, the CDN may include a content processing function configured to process a (live) media stream after being collected by the CDN. The content processing function may include an encoder that generates different (real-time) versions (representations) of the content signal. The content signal can then be partitioned into segments of a predetermined size and a specific representation can be associated with each segment. For example, one representation of a content signal can include segments of a predetermined quality (eg, high, average, and low bit rate). These segments can be encapsulated according to a suitable transport protocol format, eg, a scheme based on MPEG. In an embodiment, the segment may be encrypted and / or scrambled before being segmented. Segments associated with live streams may be temporarily stored (buffered) at one or more distribution node servers of the CDN before content consuming devices can access them.

  In response to changing network conditions (eg, variable delay occurring in data (segments) sent to the client through the streaming path), the streaming protocol follows an adaptation algorithm that can use the buffer status as an adaptation parameter. , Dynamic adaptation of the representation (e.g. switching from a high quality representation to a low quality representation or vice versa) during streaming of the segment to the client can be addressed. If the playout buffer approaches an underrun status, the client can change (adapt) the representation from the first representation to the second (lower quality) representation.

  In this case, the segment request function 125 at the client can select segments associated with the requested representation from the manifest file and continue the streaming process based on these segments. In this way, the client can adapt the streaming process in response to changing network conditions (as reflected in the buffer status). This can be very important when content is streamed over an unmanaged network such as the Internet.

  In conventional HAS client configurations, the size of the playout buffer is pre-configured and is typically relatively large (eg, about three or more segments in size). In order to deal with unexpected jitter and congestion, and to reduce the chances of buffer underruns, the buffering performed at the client is the end-to-end between the source and the content processing device playout. It is considerably larger than the end delay. In this way, conventional HAS clients are configured to handle the changing situations that can occur in unmanaged networks such as the Internet.

  End-to-end delay is the delay associated with the creation and encoding of the segment, the delay associated with the distribution of the segment to the content distribution network (CDN), and the delay associated with the internal distribution of the segment to its edge nodes within the CDN, It may include delays associated with requesting segments to the CDN and delivering the segments to the client (via the network and home network), and delays associated with processing the segments at the content processing device. These delays may include, for example, delays due to buffering and decoding processes.

  In order to reduce the risk that there are not enough segments available to fill the buffer, traditional clients are further requested by the client (when subscribing to a streaming event) when the first segment is It is configured to be a segment made available by the streaming source earlier (typically 3 segments earlier) than the segment made available by the streaming source when subscribing to the streaming event.

  Thus, when starting streaming, this requires a “go back” segment, and before the decoding and playout of data begins in the playout buffer, the playout buffer is segmented to the configured size. Filled with. Buffer and segment request function configuration allows playout of live streaming events by HAS clients and other transports such as traditional broadcast or multicast streaming, even if a large playout buffer is not required There is significant latency between the live stream playout by other content processing devices based on the mechanism. These delays cannot be solved by known IDMS techniques.

  For example, in some cases, streaming events and services may be distributed (at least partially distributed) over a managed network, such as a network operator's IP network. This is managed according to a predetermined quality of service (QoS). Such delay variations in the network are considered to be much smaller and much more predictable and stable compared to the network delay of the unmanaged network.

  To address these undesirable effects, in one embodiment, the content processing module can include a configuration module 126. The configuration module 126 is configured to adapt the size of the playout buffer based on a quality metric associated with the streaming path 114 between the server system and the content processing device. In yet another embodiment, the configuration module can configure the segment request function 125 at the client based on the quality metric so that when the client subscribes to a (live) streaming event, Allows the segment request function to determine what is first requested by this client.

  The quality metric of the streaming path can be determined by the monitoring system 128. The monitoring system 128 can receive quality metrics from one or more monitoring agents 124, 130. In an embodiment, the monitoring system may receive device quality metrics from the monitoring agent 124 at the content processing device. The device monitoring agent 124 can be configured to collect device quality metrics (ie, receive buffer and / or playout buffer segment reception time, buffer overload and underrun, segment playout time, etc.) Measurement parameters associated with segment extraction and playout processes performed on content processing devices). The device monitoring agent pulls these parameters from the client 104, receive buffer 118, playout buffer 120, and media engine 122 and sends these device quality metrics to the monitoring system 128 associated with the server system. be able to. The process of collecting and processing quality metrics will be described in more detail below with reference to FIG.

  In other embodiments, the monitoring system may also receive network quality metrics from one or more network monitoring agents 130 in the network. The network monitoring agent measures and collects network real-time quality of service (QoS) and / or experience quality (QoE) metrics (network quality metrics) from the network and collects at least some of these network quality metrics, It can be configured to correlate with one or more streaming paths between the client and one or more streaming servers of the server system.

  The monitoring agent can collect metrics on an end-to-end basis. That is, collecting and including metrics associated with the home network and the types of devices connected to the home network, thereby including the loss of source packets in the home network, home-network load fluctuations, terminals Capacity, and bandwidth available in the home-network. The monitoring agent collects quality metrics based on IP address and / or network identifier so that different devices connected to the server system by different access networks, eg, WLAN or Wi-Fi networks, can be distinguished be able to. In one embodiment, the monitoring agent can communicate with the monitoring system through the HTTP-based communication channel 131.

  During the streaming of content to the client, the monitoring system can collect quality metrics associated with the streaming path between the client and server system and store the quality metrics in the quality database 132. In an embodiment, quality metrics can be stored in a QoS client profile. Once enough data has been collected, the QoS client profile in the quality database can include a sufficiently reliable quality metric, which is the first by the client when the HA client subscribes to a (live) streaming event. Can be used to configure the playout buffer and / or segment request function to determine which segments are required.

  When a client requests the server system to subscribe to a stream event, for example, when requesting or receiving a manifest associated with this streaming event, the requesting client's QoS client profile is accessed and a quality metric And can be processed so that it can be used by the configuration module 126. Various methods of processing and instructing the configuration module are described in further detail below.

  In one embodiment, at least a portion of the quality metric is sent to the configuration module of the content processing device, where the content processing device determines the size of the buffer before data playout in the buffer is initiated. One or more configuration parameters 134, 135 may be determined to determine the first segment from the segment identified in the manifest file that the segment request function requests from the streaming server. Alternatively, the monitoring system can determine the one or more configuration parameters and send these parameters to the configuration module.

  Alternatively and / or additionally, in other embodiments, the monitoring system may determine QoS information and / or one or more configuration parameters based on the quality metric and store this information in the QoS client.

  The QoS information can define one or more QoS levels, which can be associated with one or more predetermined configuration parameters for the configuration module.

  For example, one or more QoS levels may include at least one (preconfigured) configuration parameter for configuring the client in a low latency mode (ie, small buffer size, small segment offset start). A QoS (low latency) level that defines a QoS and one or more (pre-configured) configuration parameters for configuring the client in a high latency mode (ie, large buffer size, large segment offset start) High latency) levels can be included. A high latency mode or level, sometimes referred to as a “regular” latency level or mode, is typically used when extracting segmented content through an unmanaged network such as the Internet (eg, , “Regularly”) can be associated with a default level or mode. Thus, the quality of service information may be associated with a different set of configuration parameters that may be pre-configured in the monitoring system or client. In this way, a certain (low, medium, high) latency mode of the HA client is selected by sending a message containing QoS information associated with the expected QoS level (low, medium, high) to the configuration module. be able to. Based on the QoS information, the configuration module can select a pre-configured set of configuration parameters and use these to configure the client.

  In this way, playout delay due to an excessively large playout buffer (made with excessive dimensions) can be reduced. In some embodiments, the server system may process the QoS information before it is sent to the configuration module. A more detailed example of QoS information and processing of the WoS information will be described below.

  During streaming, the monitoring system can collect quality metrics for the streaming path and update the QoS client profile in the quality database. These metrics can be collected continuously, periodically, or at a given time.

  If the QoS level of the streaming path drops or rises by a predetermined amount, the client can be reconfigured by the configuration module based on the new quality metric. To this end, in one embodiment, the client can be configured to regularly (periodically) send quality metric update requests to a server system or monitoring system. In one embodiment, the update request may relate to a request for a manifest file, and the manifest file may include configuration parameters associated with the current quality metric, QoS information, and / or streaming path.

  In other embodiments, the monitoring system can also monitor changes in the QoS level of the streaming path. If such a change is detected, the monitoring system triggers an update of the quality metric, and the quality metric, QoS information, and / or configuration parameters associated with the streaming path are sent to the content processing device to another communication channel ( For example, it can be sent over an HTTP-based communication channel). In one embodiment, this communication channel may relate to the (bidirectional) HAS control channel associated with the streaming path. Details of the HAS control channel will be described in more detail below with reference to FIG. In this way, the buffer and / or segment request functions can be adjusted dynamically in response to changing network conditions (and associated QoS level changes).

  In order for the configuration module to configure the playout buffer, in one embodiment, QoS information can be sent to the client in a manifest file. In this case, preferably the configuration module may be part of the HAS client. When a client requests a server system to subscribe to a (live) streaming event, a streaming path can be established, and in response to this request, a manifest containing QoS information associated with this streaming path. • Send files to clients. In other embodiments, QoS information can also be sent separately from the manifest file to the client over an HTTP / TCP connection.

  FIG. 2 illustrates a process for collecting metrics according to an embodiment of the present invention. Specifically, FIG. 2 illustrates at least part of the process of streaming a segment to a HAS client, where the client uses a manifest file to retrieve the segment from the server system and uses one or more monitors An agent can collect quality metrics associated with the streaming path between this HAS client and the streaming server. This streaming process can include the step where the HAS client sends a (HTTP) request for a given segment (in this case segment_high-1.mp4) to the server system (step 202), where the server system This request can be answered by sending a (HTTP) response containing (part of) the requested segment to the client (step 204), and the client processes the received data and Before being decoded and presented to the user, the data can be sent to a playout buffer associated with the media player (step 206).

  During this process, the monitoring agent at the content processing device collects device quality metrics from the HAS client, playout buffer, and media agent (step 208) and sends this information to the monitoring server in the QoS report (step 210). The monitoring server can use this report to update the QoS client profile in the quality database (step 211). This process allows the monitoring server to collect and build a streaming path QoS client profile that includes (historical) quality metrics that can be used to determine the buffer and segment request function settings in the content processing device. As can be done, it can be repeated throughout the streaming process (see, eg, steps 212-224). In one embodiment, the monitoring agent can collect device quality metrics for a number of requested segments before sending the device metrics to the monitoring system.

  Monitoring agents can collect and measure different types of metrics, and these metrics are stored in the quality database for device type, data type, protocol / codec type, network type, access technology type, etc. Metadata that can be used by the surveillance system to classify can also be included. In an embodiment, quality metrics may include metrics associated with the adaptive streaming process, such as, for example, manifest file number, content profile, content name, content description, specified playout period, initial Profile, Originating Server Manifest, Origin Server Segment, Number of Redirects, Bit Rate Increase Change, Bit Rate Decrease Change, Number of Buffered Segments, Received Segments, Requested Segments, Buffer Buffer Underrun, segment received byte, segment profile bit rate, segment read bit rate, segment nr read bit rate lower than profile, number of segment timeouts, segment The number of defects, the number of segments in the receive buffer and / or playout buffer are increased.

  In other embodiments, the quality metric includes information associated with the access technology, such as the type of access technology (Cable, DSL, fiber, etc.), the type of access line (VDSL, ADSL, etc.), and these access lines. Can include typical delays and / or bandwidths associated with.

  In still other embodiments, the quality metric may include information associated with the content processing device, such as a serial number, the content processing device manufacturer, and / or a chipset identifier. In other embodiments, the metrics may also include information associated with hardware utilization at the content processing device, such as available free memory, CPU utilization, and / or signal strength. In yet other embodiments, the metric may include an identifier for identifying the access network and / or platform.

  In one embodiment, the quality metric can include information associated with the network, throughput associated with load due to the number of users sharing the same network resource, missing packets, bit errors, queues or congestion. Latency, jitter, and out-of-order delay due to the fact that different packets may reach their destinations in different orders.

  In yet another embodiment, the monitoring agent monitors a device quality metric associated with the streaming process (at least in part) according to a DASH (quality) metric as defined in MPEG DASH ISO / IEC 23001-6. Can also be collected. Appendix D, entitled “DASH Metrics” in this document, specifies various DASH (quality) metrics: TCP connection, HTTP request / response transaction, representation switching event, buffer level, and play list.

  For example, in one embodiment, at the input of the HAS client, the monitoring agent may receive multiple sets of TCP connections (IP address, start, connect, and close time), one or more sent HTTP segment requests (each of which is a send time, The content and the one or more received HTTP responses containing the requested segment (determined by the response message reception time, the response header content, and the reception time of each byte of the response body, respectively) ) Can be monitored.

  In other embodiments, the monitoring agent may also monitor one or more encoded (MPEG) frames at the output of the HAS client (the encoded frame may include media type, identifier, decoding time, presentation Time and / or delivery time). Similarly, at the media engine, the monitoring agent can have one or more decoded (MPEG) frames (each of which is the media type, the presentation time stamp of the frame, and the actual of the decoded frame (or part thereof)). (Determined by the presentation time).

  Once the metrics are collected by the monitoring agent, a reporting process occurs and the monitoring agent sends the collected metrics to the server.

  In an embodiment of the invention, these collected metrics are inserted as parameters within the URI of an HTTP GET or HTTP POST request. HTTP GET or HTTP POST requests can be used to report collected metrics. FIG. 10 shows an example of such requests 1001, 1002 for buffer level metrics.

  In an alternative embodiment according to the present invention, the metric can also be reported by the monitoring agent in an HTTP POST or HTTP PUT request. FIG. 11 shows an example of such a request 1103, 1104 for a buffer level metric.

  In one embodiment, the JSON format is utilized to describe the metric. In yet another embodiment, the metrics are written according to an XML format. FIG. 12 shows an example format for such a request 1205, 1206 that contains a buffer level metric.

  In the request example of FIG. 11, the URI in the first line of this request represents the name of a file created on the server. Thus, for example, according to the format detailed in FIG. 12, the appropriate extension, .xml or Json is given.

  In other embodiments, the collected metrics are returned and reported via WebSocket. Once a connection is established between the monitoring agent and the server via WebSocket, the collected metrics are sent to the server by the monitoring agent.

  In other embodiments, the collected metrics are written to a file by the monitoring agent and then sent to the server. The file can be transferred by FTP, FTPS, peer-to-peer (for example, Bittorent), SFTP, and SCP, for example. The file can be written in XML or JSON format as shown in FIG. 12, for example. If WebSocket is used, the file can be encoded in base64 and inserted into the body of the HTTP message. The HTTP message can be sent to the server (system) by the monitoring agent via a WebSocket connection.

  FIG. 3 shows a schematic diagram of at least a portion of a manifest file according to an embodiment of the present invention. Specifically, FIG. 3 shows at least a portion of a manifest file (in this example, a portion of an MPEG DASH MPD) that includes QoS information and configuration parameters, and the playout buffer by the configuration module in the content processing device. And can be used to configure the segment request function.

  In this particular embodiment, the manifest file may include QoS information that includes a first QoS level (eg, “high latency” or “normal” mode) 310 and a second QoS level (“low latency” mode) 312. it can. The first QoS level may define a normal mode 310 that includes configuration parameters associated with conventional settings of the HAS client that are suitable when streaming through an unmanaged network. The second QoS level may define a low latency mode 312 that includes configuration parameters associated with HA client low latency settings suitable when streaming through a managed network.

  A manifest file containing QoS information and associated configuration parameters can be sent to the HAS client in the manifest file, which can parse this manifest file and send QoS information and configuration parameters to the configuration module. . The configuration module stores this information and parameters.

  Based on the current quality metric of the streaming path, the monitoring system or configuration module can select a normal mode or a low latency mode, for example.

  It should be noted that many variations on the use of quality metrics, QoS information, and / or configuration parameters are possible without departing from the invention. For example, in one embodiment, the manifest file may include only configuration parameters associated with the current quality metric, which are used by the configuration module to configure the buffer and / or segment request functions. In other embodiments, the manifest file may include only QoS information for reporting (expected) QoS levels without any configuration parameters (eg, normal mode or low latency mode). The QoS level can be used by the configuration module to select a pre-configured set of configuration parameters, which are stored locally in the content processing device memory. In yet other embodiments, the manifest file may include a quality metric associated with the streaming path.

The quality metric is used by the configuration module to determine the QoS level and select a preconfigured configuration parameter based on the QoS level, or to directly determine one or more configuration parameters based on the quality metric. be able to. In an embodiment of the present invention, such a quality metric may be one or more parameters related to service quality and is also referred to as a QoS parameter. Such QoS parameters can relate to, for example, but not limited to, guaranteed bandwidth, packet loss rate, delay, and jitter. FIG. 13 shows an example of MPD including quality metrics in the form of QoS parameters. Examples of these QoS parameters in MPD are as follows.
MinGuaranteedBandwidth: The minimum bandwidth that the client can expect (in this example, in bits per second).
MaxGuaranteedBandwidth: The maximum bandwidth that the client can expect (in this example, in bits per second).
PacketLossRatelnPercent: Percentage of lost packets based on traffic history.
Delay: Latency between the client and the content pulling node (eg streaming server / cache). In this example, it is given in milliseconds.
Jitter: Jitter between the client and the content pulling node (eg, streaming server / cache). In this example, it is given in milliseconds.

  In one embodiment, the configuration parameters can include MDP parameters or combinations of MDP parameters as defined in the ISO standard ISO / IEC 23001-6 to achieve the desired playout buffer configuration. For example, the minimum buffer size parameter “minBufferTime” 314, 318 can be used as a configuration parameter to set the minimum size of the buffer before playout of data in the buffer is initiated (eg, normal mode Is 5 seconds, and low latency mode is 1 second).

  In other embodiments, the configuration parameters may include suggested presentation delay parameters “suggestedPresentationDelay” 316, 320. This parameter can be used to configure the HAS client to introduce a playout delay in addition to the playout time of a given segment indicated in the manifest file. For example, if the information in the manifest file determines that the segment needs to be played out at 12:10 and the parameter suggestedPresentationDelay is set to 00:10, the segment will be 10 seconds later, ie at 12:20 Play out.

  In yet another embodiment, the configuration parameters include segment start parameters “segmentStartOffset” 324, 326 for determining the first segment from the segment identified in the manifest file that the segment request function requests from the streaming server. be able to.

  This parameter can define an offset for the current segment made available by the content source when the client subscribes to a live streaming event. For example, a streaming server in the server system may have created the current segment 1000. However, if the client initiates playout based on the current segment 1000, it is impossible to build a sufficiently sized buffer because segment 1001 is not yet available. For this reason, a problem may occur in a network having a low QoS level or an intermediate network. In this case, it cannot be guaranteed that the client receives the segment 1001 in a timely manner.

  To address this issue, the “segmentStartOffset” parameter can be set. If this parameter is set to 1, 2, or 3, respectively, the segment request function is available at the source based on segments 999, 998, or 997, ie when the user subscribes to a live streaming event. It is determined that playout is started based on a segment generated earlier than the current segment 1000. For example, in FIG. 3, the normal mode is associated with the first segment. The first segment is 3 segments behind the segment made available by the streaming server at the time the HAS client joins the streaming session. The low latency mode is associated with the first segment. The first segment is one segment behind the segment made available by the streaming source when the HAS client subscribes to the streaming event. Selecting normal mode to reduce the risk that sufficient data (segments) cannot be buffered by the playout buffer when the quality metric indicates that the streaming path is associated with a low QoS (unmanaged) network Can do. When the quality metric indicates that the streaming path is associated with a high QoS (managed) network, a low latency mode can be selected to reduce the latency in streaming as much as possible.

  Instead of supplying quality metrics, QoS information and / or configuration parameters in the manifest file to the client, at least part of this information is supplied to the client by another communication channel instead of part of the manifest file. You can also. This embodiment will be described in more detail later with reference to FIGS.

  The manifest file can also contain other information. For example, the manifest file can include a manifest type indicator 302. Manifest type indicator 302 indicates whether the manifest file is a static manifest file that does not change over time or a dynamic manifest file that changes over time. For example, in one embodiment, the client may receive a newly updated manifest file or a manifest file update to update the manifest file at the client at a predetermined time, eg, periodically. it can. The manifest file update includes newly updated information associated with the segment path. In this way, the HAS client can be reconfigured in response to QoS level changes in the stream path.

  In addition, the manifest file includes a segment presentation duration parameter “mediaPresentationDuriation” 304, ie, the length of the content stream (in seconds), segment location information 322, eg, one or more URLs indicating where the content can be retrieved. And profile parameters 308 indicating media metadata, eg, HAS file type, eg, MPEG-DASH, and / or service type, eg, 2011 version of the VoD service.

  FIG. 4 illustrates a process for updating a manifest file according to an embodiment of the present invention. This process can begin when a customer makes a payment to subscribe to a live streaming event through the content provider's website. After gaining access, the customer can instruct the client at some point through the user interface (eg, by pressing a play button) to subscribe to a live streaming event. In this case, the client can request a manifest file from the server system. For example, a request message for a manifest file associated with a streaming event, such as an HTTP GET message, can be sent to the server system (step 402). In this case, the request message is used to identify the client requesting the streaming service, for example, the name of the manifest file and the client identifier (eg, IP address and / or device identifier and / or HTTP, TCP, or Any other information from the IP header) can be used.

  After receiving the request message, the server system can query the quality database for the QoS information associated with the streaming path based on the client identifier (step 404). The database can look for QoS client profiles (step 406), and if a profile is found, it can determine QoS information based on quality metrics and send this information to the server system (step 406). 408). The server system can then modify this by pulling the manifest file associated with the (live) streaming event from the manifest storage and inserting the QoS information into the manifest file. Thus, a client specific manifest file is created, which is specific to this client and includes QoS information that defines the optimal configuration of the content processing device (step 410).

  As already described in detail with reference to FIGS. 1-3, various types of information (quality metrics, QoS information, quality information, QoS information, in order to configure a content processing device based on quality metrics associated with a streaming path). And / or configuration parameters) can be sent to the content processing device. Thus, although the embodiments described with reference to FIGS. 4-9 are described based on the use of QoS information, these examples also include the use of quality metrics, QoS information, and / or configuration parameters. I offer you that.

  A manifest file containing QoS information can be sent to the client (step 412). In one embodiment, the manifest file can be sent in an HTTP response message to the client, and the client parses the information in the manifest file (step 414) and sends QoS information to the configuration module (which is part of the client). (Step 416). In addition, the manifest file can also include configuration parameters, but this is not necessarily so. Configuration parameters can also be transferred to the configuration module (eg, if the configuration module does not yet have them). The configuration module can use the QoS information to configure the playout buffer (step 418). Thus, client specific QoS information in the streaming path and / or content and / or manifest file can be used to set the playout buffer to a size that minimizes startup delay.

  In this way, HAS clients that subscribe to (live) streaming events can be synchronized more quickly with other content processing devices in the streaming event.

  FIG. 5 illustrates a protocol flow that enables low latency streaming of segments to clients in accordance with one embodiment of the present invention. This process can begin in the same manner as described with reference to FIG. If the client is instructed to subscribe to a live streaming event, a manifest file can be requested from the server system (step 502). Upon receipt of the request, the server system may create a client specific manifest file that includes QoS information associated with the streaming path (step 504). The manifest file can be created according to a process as described with reference to FIG.

  A client specific manifest file that includes QoS information can be sent to the client (step 506), and the client includes information in the manifest file (eg, QoS information, segment identifier, segment location information, and segment playout information). Can be analyzed (step 508).

  Based on the QoS information, the configuration module (which may be part of the client) can configure the content processing device to minimize playout delay. To this end, the configuration module can configure a segment request function at the client based on QoS information that can be associated with a predetermined segmentStartOffset parameter (step 510). In this way, the segment request function can determine the first segment to request when starting streaming. Alternatively and / or additionally, the configuration module may use QoS information that can be associated with a predetermined minBufferTime parameter (configuration parameter) to configure the size of the playout buffer. This parameter can be sent to the buffer (step 512) so that the buffer can be configured accordingly (step 514).

  In the meantime, the client can initiate the process of requesting a segment from the server system. Specifically, after the client has identified the first segment (eg, segment_high-2.mp4 that needs to be retrieved), the location information, eg, the URL associated with the first segment, can be retrieved and the HTTP request A message, eg, an HTTP GET message, can be sent to that location (eg, the network address of the HTTP server) (step 516). The HTTP server can respond to the request by sending an HTTP response message containing the requested segment to the client (step 518). The client can unpack and decode the segment data and transfer the decoded data to a play buffer associated with the media engine (step 520).

  FIG. 6 shows a schematic diagram of a content distribution system according to another embodiment of the present invention. This distribution system can include components similar to the content distribution system described with reference to FIG. That is, server system 602 is configured to stream content to at least one (HAS) client 604 at content processing device 606. The server system receives one or more content signals 608, eg, (live) video and / or multimedia signals, and processes the content signals to generate a segmented content stream and associated manifest file. It can be constituted as follows. The segmented content stream and associated manifest file can be stored in segment storage 610 and manifest storage 612, respectively.

  When subscribing to a (live) streaming event, the (HTTP) streaming function in the server system can send a manifest file associated with this streaming event to the client via the (HTTP) network connection 614. . The manifest file can be used by the client to request a segment from the HTTP streaming function, which sends the segment to the client via an (HTTP) network connection (streaming path). The HTTP streaming function is configured to deliver segments to the client based on the HAS protocol. During streaming, the segments are buffered in the client receive buffer 618, which unpacks the encapsulated segments (frames) and decodes and plays the unpacked (MPEG encoded) data by the media engine 622. It can be arranged in the playout buffer 620 for out.

  In response to changing network conditions (eg, delays that occur in data (segments) sent to the client through the streaming path), the streaming protocol addresses the dynamic adaptation of the representation while streaming the segment to the client. Can be allowed.

  The content processing device can include a configuration module 626. The configuration module 626 is configured to determine one or more configuration parameters 634, 635 based on the QoS information. In one embodiment, one or more configuration parameters can be used to adapt the size of the playout buffer. In this way, the buffer size can be controlled based on the quality metric associated with the streaming path 616. Further, in other embodiments, one or more configuration parameters 635 are used by segment request function 625 at the client to determine the first segment that needs to be requested when subscribing to a live streaming event. be able to.

  The monitoring system 628 can receive quality metrics from one or more monitoring agents. The device monitoring agent 624 collects device quality metrics from the client 604, receive buffer 618, playout buffer 520, and media engine 622 and sends these device quality metrics to the monitoring system 628 associated with the server system. Can be configured to send.

  The network monitoring agent can be configured to collect quality metrics from the network and correlate these metrics with specific streaming paths. Based on the quality metric, the monitoring system can determine the quality of service information associated with the streaming path between the client and the streaming server and store the quality metric in the quality database 632 (see FIG. 1). The same as described above). When sufficient data has been collected, it is considered that reliable quality metrics have been collected in the QoS client profile in the database.

  Quality metrics in the QoS client profile are used by the configuration module in the content processing device to set the playout buffer to a size and / or to configure the segment request function to reduce playout delay. be able to.

  In this particular embodiment, no QoS information (and / or quality metrics and / or configuration parameters) is sent to the client based on the manifest file (as described in detail with reference to FIGS. 3-5). Instead, a separate (bidirectional) communication channel 636 is established between the quality database (or the monitoring system associated with the quality database) and the configuration module (or client associated with the configuration module), and at least the QoS information A portion can be used to send to the configuration module. In one embodiment, this communication channel may also be established when a client subscribes to a specific (live) streaming event.

  In one embodiment, when subscribing to a (live) streaming event, the client includes location information in a quality database, eg, Client Access-Line Quality Database (CAQD), eg, URL, caqd.example.com/ A manifest file can be received. The client access-line quality database may include a QoS client profile that includes quality metrics, QoS information, and / or configuration parameters associated with the streaming path.

  An embodiment of such a manifest file is shown in FIG. Here, at least a portion of the manifest file may contain information similar to that described with reference to FIG. 3, and the manifest file may or may not change over time. A manifest type indicator 702 that indicates which of the dynamic manifest files to perform, a segment presentation period parameter “mediaPresentationDuriation” 704, segment location information 720, and media metadata, eg, profile parameters 708.

  In addition, the manifest file may include quality database location information, eg, URL, caqd.example.com / </ CAQDLocation>. This location information can be used by the configuration module in the content processing device to request QoS information and / or quality metrics and / or configuration parameters from the quality database.

  In yet another embodiment, the manifest file may include channel configuration information 724, 726 for configuring a communication channel associated with the streaming path, in particular, a (bidirectional) HAS control channel. In one embodiment, the channel setup information may include a channel target parameter 724 that provides a reference to a network node that includes a streaming control function. Further, in other embodiments, the channel setup information may include channel parameters 1402, ie parameters used by the streaming control function / control channel server function. For example, in the case of WebSocket, these parameters can refer to the use of the WebSocket sub-protocol, WebSocket version, etc. The setting and use of the HAS control channel will be described in more detail with reference to FIG.

  FIG. 8 illustrates a process for delivering segments to clients according to another embodiment of the present invention. Specifically, FIG. 8 shows a process for delivering segments to clients using a content delivery system as described with reference to FIGS. This process can begin when a client subscribes to a (live) streaming event, and the client requests a manifest file from the server system and responds, for example, using an HTTP request message. For example, a manifest file may be received in an HTTP response message (steps 802 and 804).

  The client can then parse the manifest file and extract location information associated with the quality database from the manifest file (step 806). The location information and URL of the quality database can be transferred to the configuration module (step 808). The configuration module can use the location information to set up a (bidirectional) communication channel between the quality database and the configuration module (or the HAS client associated with the configuration module). After establishing the communication channel, the configuration module can query the quality database for QoS information, specifically information regarding the expected QoS of the streaming path (step 810). The configuration module receives QoS information in response and determines a configuration parameter, eg, a pre-configured configuration parameter, in the configuration module based on a predetermined QoS level, eg, QoS information including a low latency mode. This QoS information can be used.

  The configuration module may send one or more configuration parameters to configure the playout buffer associated with the media engine (step 814). In addition, the configuration module can send one or more configuration parameters to the segment request function at the client (step 815), which uses the parameters to determine which ones when subscribing to a live streaming event. It can be determined whether a (first) segment should be requested (step 816).

  The HAS client then requests a (first) segment that is determined based on one or more first configuration parameters, and receives the requested segment in response (steps 818 and 820, respectively), thereby streaming the The process can be started.

  FIG. 9 shows a protocol flow between a content processing device 930 and a server system 932 for setting up a bidirectional HAS control channel according to an embodiment of the present invention. The content processing device may include a HAS client 948 and a media engine 946 (similar or identical to that described with reference to FIGS. 1 and 6). Similarly, the server system may stream content to a HAS client and monitoring system 940 connected to a quality database 941 that includes a client QoS profile (similar or identical to that described with reference to FIGS. 1 and 6). HTTP streaming functionality 942 can be included.

  The content processing device and server system can further include a control channel client function (CCCF) 944 and a control channel server function (CCSF), eg, a HAS control channel server function 934, respectively. These are configured to establish a streaming control channel 936 between the CCSF and the CCCF 944. Here, the streaming control channel can be used to exchange streaming control information between the client and the server. In particular, the streaming control channel can be used to send streaming control information originating from the server system towards the client during streaming of the segmented content 938 to the client. For example, in one embodiment, the streaming control channel sends an update manifest trigger from the server system (or monitoring system) to the CCCF so that the client sends a request for a manifest file (or manifest update) to the server system. Can be used for.

  Here, the process may begin, for example, when a user subscribes to a live streaming event, as described above with reference to other processes (step 900). The client can send an HTTP GET request to obtain a manifest file from the server system, and the server system can respond to this request by sending a manifest file to the client (step 902). 904).

  The CCSF at the server is configured to insert channel configuration information into the manifest file. This information allows the CCCF at the client and the CCSF at the server to set up the streaming control channel. Accordingly, during analysis of the manifest file (step 906), channel setting information is extracted from the manifest file (see, for example, FIG. 7), and a channel setting request for setting a server-client streaming control channel is issued at the server. Can be used by the CCCF at the content processing device to send to the CCSF (step 908).

  In one embodiment, the CCCF and CCCSF may include an HTTP WebSocket API configured to use the WebSocket protocol and channel configuration information for setting up a streaming control channel between the client and the server. WebSocket connections typically use standard HTTP ports 80 and 443 so that data can easily be transferred through the firewall and NAT, but other ports may be used.

  Use of the WebSocket protocol is within the context of CDN and HAS, such as low message overhead for scalability, use of HTTP for protocol convergence and firewall traversal, and the possibility of tunneling other protocols. There are various advantages. In other embodiments, Session Initiation Protocol (SIP) (http://tools.ietf.org/html/rfc3261) can also be used, in which case the client can include a SIP user agent and the server Is a SIP application server.

  In yet another embodiment, extensible messaging and presence protocol (XMPP) (http://www.ietf.org/rfc/rfc3920.txt) is used, in which case the client can include an XMPP client. The server includes an XMPP server. Both SIP and XMPP protocol messages can be tunneled over WebSocket according to draft-ibc-rtcweb-sip-websocket-00 and draft-moffitt-xmpp-over-websocket-00.

  Channel parameters can be exchanged between the CCCF and the CCSF during setup of the streaming control channel (step 910). In addition, to handle messages originating from clients, the CCSF can create a dedicated channel processing process (thread) (step 912). Once the streaming control channel is established (914), the client can begin the process of streaming the segments identified in the manifest file. The streaming process may begin with an HTTP GET request that includes the URL associated with the first segment segment_low-1.avi based on the HAS-type streaming protocol (step 916). Once delivery of the first segment has been confirmed by an HTTP 200 OK response (step 918), the client can request a subsequent segment segment_high-2.avi (steps 920, 922).

  The CCSF in the server system can then determine that the client needs to update its manifest file. For example, the CCSF may have received a message from the monitoring system that the playout buffer may be resized because the QoS level of the streaming path has changed significantly. Accordingly, a manifest update signal can be sent over the streaming control channel (step 924). In one embodiment, the update signal may include a URL that points to a new manifest file that includes the new QoS information. Upon receiving the manifest file update signal, the CCCF can request a new manifest file. Upon receipt of a new manifest file containing new QoS information, the client can send QoS information to the configuration module, which then reconfigures the playout buffer based on the received QoS information. To do. Streaming the segment to the client can continue based on the reconstructed playout buffer. Similarly, the segment request function can also be reconfigured based on QoS information.

  In other embodiments, instead of sending QoS information to the client in the manifest file, the QoS information (at least a portion thereof) may be sent to the client through the HAS control channel.

  In one embodiment, instead of transferring the channel setting information in the manifest file, the channel setting information may be pre-installed in the terminal or may be pulled from other (network) sources through another communication channel. . In that case, when the client receives the manifest file, it triggers the streaming control channel client function to establish a streaming control channel as described with reference to steps 908-914 of FIG. Pull configuration information.

  In other embodiments, the server system can also be configured to stream segments to multiple clients, each client performing network initiation, eg, server initiation control, as described with reference to FIG. To enable, it is associated with its own streaming control channel. In this way, the server system can control the streaming of segmented content to multiple clients based on quality metrics stored in the quality database. When the monitoring system 940 associated with the server system detects a QoS change in the network, the CCSF can be notified to initiate a manifest file update, and (at least a part of) the HAS client can be notified of the new QoS. A new manifest file containing the information is provided and the new QoS information can be used to reconfigure the playout buffer to the appropriate size.

  It is noted that the figures depict non-limiting examples of the invention and that the embodiments described with reference to these figures may be combined with each other without departing from the invention. For example, in an embodiment, a manifest file is used to send a first portion of QoS information, quality metrics, and / or configuration parameters to the content processing device (as described with reference to FIGS. 1-4). , QoS information, quality metrics, and / or other portions of configuration parameters may be sent over another communication channel (as described with reference to FIGS. 6-9). Thus, in this case, the manifest file is a quality that includes QoS information, quality metrics, and / or configuration parameters (as described with reference to FIG. 3), location information, eg, QoS client profile for the streaming path. Both the URL associated with the database and / or channel setting information (as described with reference to FIG. 7) for setting the HAS control channel can be included.

  FIG. 14 shows that the configuration module of the HAS client, preferably the HAS client (device), includes ALTO client functionality and uses the ALTO protocol to query the ALTO server / database in the network for QoS information (1410) implementation. A form is demonstrated. The ALTO server / database containing the ALTO server functionality can then provide a response (1412) containing the QoS information. Based on the received QoS information, the playout buffer (parameter) can be (re) configured.

  Furthermore, although the embodiments in the previous figures have been described with reference to the playout buffer configuration, the present invention can be applied to any type of buffer that contributes to playout delay in a content processing device. (E.g., receive buffer, decoding buffer, etc.).

  It should be noted that any feature described with respect to any one embodiment can be used alone or in combination with other features described, and further, one or more features of any of the other embodiments described above. It should be understood that any combination and combination of, or any other of the above-described embodiments can be used. One embodiment of the invention can be implemented as a program product for use by a computer system.

  The program product program (s) define the functionality of the above-described embodiments (including the methods described herein) and can be stored on various computer-readable storage media. Exemplary computer readable storage media include: (i) a non-writable storage medium in which information is permanently stored (eg, a read-only internal computer such as a CD-ROM disc readable by a CD-ROM drive). A memory device, flash memory, ROM chip, or any type of solid state non-volatile semiconductor memory), and (ii) a writable storage medium in which variable information is stored (eg, in a diskette drive) Including, but not limited to, floppy disks, or hard disk drives, or any type of solid state random access semiconductor memory). The invention is not limited to the embodiments described above, but can be varied within the scope of the appended claims.

Claims (15)

A method for enabling low-latency streaming of segments over a network to a client (104) for HTTP-based adaptive streaming at a content processing device (106) , wherein the client allocates a segment to a server based on a manifest file. Configured to request and receive from the system, the method comprising:
A monitoring system (128) in the network,
Collecting quality metrics associated with one or more streaming paths between the client and one or more streaming servers in the server system, and storing the quality metrics in a quality database (132) in the network Storing, and
Quality of service information supplied to the content processing device at least a portion of the stored quality metric or determined based on at least a portion of the stored quality metric and / or one or more configuration parameters Wherein the quality of service information defines the one or more QoS levels, the one or more QoS levels comprising one or more configuration parameters for configuring the client in a low latency mode. At least a low latency level associated with and a high latency level associated with one or more configuration parameters for configuring the client to a high latency mode ;
Wherein the quality metric or at least a portion of the service quality information, or based on the configuration parameters, configuration module in the content processing device, the content processing device in contact Keru buffer segment requests in the US and / or the contents processing device Configuring the function. The method of claim 1, comprising:
Based in part on the at no less of the quality metric comprises determining one or more configuration parameters to the configuration module, the one or more configuration parameters,
To determine the size of the buffer before playout of the data is started in the buffer, at least one buffer size parameter is selected from the segment identified in the Contact and / or the manifest file, the segment request At least one segment request parameter that determines the first segment that a function requests from the streaming server ;
Including, method. The method of claim 2 , wherein the segment request parameter is a segmentStartOffset parameter .   4. The method according to any one of claims 1 to 3, wherein at least part of the one or more configuration parameters and / or the quality of service information is determined by the monitoring system and stored in the quality database. ,Method.   5. A method according to any one of the preceding claims, wherein the providing step comprises the content processing with the quality metric, the one or more configuration parameters and / or the at least part of the quality of service information. A method comprising sending to a device. 6. The method of claim 5, wherein the supplying step.
The client requesting the server system for at least a portion of a manifest file or manifest file update;
The server system extracts the quality metric, the one or more configuration parameters, and / or the at least a portion of the quality of service information from the quality database, the quality metric, the one or more configuration parameters; And / or sending a manifest file containing the at least part of the quality of service information to the client;
Including the method. The method according to claim 5 or 6, wherein at least part of the quality metric, the one or more configuration parameters, and / or the quality of service information is sent to the client through a WebSocket based HAS streaming control channel. Method. The method of claim 7, comprising:
Bi lateral Kosu streaming control channel setting information for setting a channel comprising the steps supplied to the client, before Symbol streaming control channel WebSocket based HAS streaming control channel between the server and the client system Is a step,
A step of based on said channel setting information, to establish a pre-Kiso side Kosu streaming control channel,
Including the method. A method according to any one of claims 1 to 8, comprising:
At least a first monitoring agent in the content processing device collects a first metric associated with the content processing device; and / or at least a second monitoring agent in the network is associated with at least a portion of the network. Collecting a second metric,
Based on the first and / or second metrics, the quality of the monitoring system associated with one or more streaming paths between the one or more streaming servers and the client in the server system Determining a metric;
Storing the quality metric in the quality database;
Including the method. A content delivery system that enables low latency streaming to a content processing device of a segment through at least one network comprising:
A content processing device including a client HTTP-based adaptive streaming, the client, based on the manifest file is configured to request and receive the segments to one or more streaming server,
Further, the at least a portion of the store has been quality metric, there have is that the service quality information is determined based on at least a portion of the store and quality metrics and / or one or more configuration parameters are supplied Ru is configured, and the content processing device,
A monitoring system configured to collect quality metrics associated with one or more paths between the client and the one or more streaming servers and store the quality metrics in a quality database in the network When,
Configure your Keru buffer to the content processing device, and / or, the contact Keru segment request function to the content processing device, based at least in part on the quality metric, wherein the service quality information, and / or the configuration parameter And a configuration module in the content processing device . A configuration module for use in a content processing device, wherein the configuration module is, in the content processing device is configured to permit low latency streaming to client HTTP-based adaptive streaming, the client Configured to request and receive segments from one or more streaming servers in the server system based on the manifest file, the configuration module further comprising:
Configure the buffer in the content processing device, and / or the segment request function in the content processing device, based on the quality metric, at least in part based on the determined quality of service information store has been quality metric Configure based on and / or based on one or more configuration parameters;
The quality metric is associated with one or more streaming paths between the client and one or more streaming servers in the server system;
The quality metrics are collected by a monitoring system in the network and stored in a quality database in the network;
A configuration module that is configured as follows. The configuration module of claim 11.
It said buffer, based on one or more configuration parameters, before the play-out of the data is started in the buffer, is configured to determine the size of the buffer, and / or the segment request function, based on one or more configuration parameters, the selected from the segment identified in the manifest file, configured such that the segment request function to determine a first segment for requesting the streaming server, configuration module. A monitoring system for monitoring a quality metric associated with a streaming path in a network between an HTTP-based adaptive streaming client and a streaming server in a content processing device, comprising:
Means for collecting device metrics associated with the content processing device and network metrics associated with at least a portion of the network from one or more monitoring agents;
Means for determining a quality metric associated with a streaming path based on the device and the network metric;
Based in part on the at no less of the quality metrics, a means for determining the one or more configuration parameters for the configuration module in the content processing device, the one or more configuration parameters, play of data in the buffer is selected from the segment identified in at least one buffer size parameters, and / or the manifest file to determine the size of the buffer before out is started, the segment request function the streaming - at least one segment request parameters for determining the first segment to request the server, and means,
Including monitoring system. A data structure is a manifest file for use by the client in the content processing device, the client, based on the manifest file, is configured to request a segment to at least one server receives The data structure enables low latency streaming to the client, the data structure including one or more segment identifiers and segment playout information, the data structure further comprising a quality metric, and And / or including quality of service information including one or more QoS levels associated with a streaming path between the client and a streaming server, wherein the quality metric and / or the quality of service information is associated with the client. Was configured modules make it possible to determine one or more configuration parameters for a segment request function in buffer contact and / or the content processing device, and / or the data structure, further, the client and streaming includes one or more configuration parameters associated with the streaming path between the server, the one or more configuration parameters, in the content processing device, configuration module associated with the client, buffer, and A data structure that makes it possible to configure a segment request function. When executed in the memory of the computer, including the configuration software code portions to perform the steps of how to any one of claims 1 to 9, the computer program.

Images