CA2314744A1 - A system and method for enhanced streaming media viewing - Google Patents

Abstract

A system for providing an enhanced quality of service in a streaming multimedia system wherein streaming media is delivered from a media server to a requesting client media player. The streaming media is modified before transmission by the media server and includes a plurality of markers located at predetermined positions indicating new content insertion points. The media player receives the modified data stream monitors the bandwidth or data rate of the connection and uses the markers to inset locally stored content in the stream to provide playback thereof while buffering the received media stream. The media player resumes playback of the streaming media when the buffer reaches a predetermined level or the data rate of the connection increases.

Description

A SYSTEM AND METHOD FOR ENHANCED STREAMING MEDIA VIEWING
The present invention relates to networked multimedia systems, and more particularly to a system and method for improving the perceived quality of multimedia information delivered to one or more clients over a limited bandwidth network.
BACKGROUND OF THE I1~TVENTION
The explosive growth of the :Internet has resulted in a growing need to provide multimedia 1o material such as video and audio to users. Interactive television, movies on demand and other multimedia technologies are 'becoming more popular. This popularity coupled with an ever-increasing demand for a high quality of service (QoS) has placed significant demands on bandwidth and hardware requirements of the Internet.
15 The Internet, like other networks, is based on an IP protocol. Packets of data are routed around the network between a sender and recipient based on the address of the intended recipient contained in the packet. Thus, a specific connection between the sender and recipient is not required. Generally, packets may be sent via different routes in an effort to increase the likelihood of delivery, which means that data packets do not inherently have a guaranteed arnval 2o time. In fact, packets corresponding to a signal message may even be received out of order.
This fact significantly affects how certain multimedia data are delivered.
Since multimedia data requires real time delivery, a particular media stream needs to be delivered in the proper time sequence, to allow the user to play back the audio/video selection as real motion or real audio.
25 In order to support the delivery of realtime content over the Internet, a realtime protocol (RTP) was introduced to replace the conventional transmission control protocol (TCP). The RTP
provides a framework that real time applications can use directly for data transport. Currently, the RTP standard supports a particulaa- type of message, namely one for carrying the media content data for streaming data. A separate protocol known as the realtime control protocol 30 (RTCP) is used with the RTP to pass control message for session management, rate adaptation and the like. Along with the realtime aspect of multimedia, as described above, multimedia data typically require an enormous number of bytes to represent the information contained therein.

Consequently, compression l:echniques are utilized to reduce the amount of data transmitted.
Generally, compression requires an encoder to compress source data and a decoder to decompress the compressed data.
The quality of multimedia tr<~nsmitted between a server and a client computer over a network depends largely on the bandwidth of the network. Bandwidth refers to the maximum number of bits that a network can transmit from one node coupled to the network to another node coupled to the network. If insufficient bandwidth is available, the number of bits transmitted is reduced and consequently, the image or sound of the multimedia is distorted or not fully represented. In 1o many computer networks, the primary element which limits the bandwidth is the network interface device which interf~.ces the server and client computers to the network. In the Internet realm, such interface devices include modems and Ethernet controllers.
A solution around this problem is the introduction and adoption of broadband delivery systems 15 such as asymmetric digital subscriber lines (ADSLs), cable modems, and ATMs, to name only a few. These broadband delivery systems are now capable of offering the consumer a relatively inexpensive access to the Intc;rnet. While these broadband delivery systems have made the reception of the streaming media more satisfying to the user, there is still a limitation in the ultimate content that can be provided. Currently, the streaming media content is still limited to 2o the reception of simulcasts of radio broadcasts over the Internet, video clips, and very low quality or low definition television.
Even though improved protocols, hardware, compression and decompression techniques and increased network bandwidth. are being introduced, the number of users requiring or requesting 25 such streaming media is increasing proportionally. Thus, users are still experiencing fluctuations in bandwidth in the network causing choking and picture freezing, making streaming media still annoying to the end user.
Real motion, high resolution video should be optimally displayed at 24 to 30 frames per second.
30 An advantage of the present invention. is to achieve such a perceived quality of service. The present invention attempts to replicate a user's familiarity with the format of current television broadcast programs to create the impression of smooth real time real motion video in a multicasting or multistreaming environment. More specifically, television viewers are accustomed to television programs which typically in five to ten minute segments followed by advertising or commercials which run for two to three minutes. These commercials are inserted in the broadcast video by local distributors. These commercials are generally known in advance of the broadcast.
Accordingly, there is a need for a system and method that is capable of allowing streaming media such as television and movies t:o be transmitted over the Internet while providing the user 1o with a perceived transmission improvement over current transmissions for realtime television, audio transmission or the likE;.
SUMMARY OF THE INVENTION
I5 According to an embodiment of the present invention there is provided a method for transmitting streamed digital data from a server, the server being configured for coupling to a client computer via a computer network. The: method comprises the following steps. A modified data stream is provided to the media player, the modified data stream including a plurality of markers located at predetermined positions for indicating new content insertion points. The modified data stream is 2o transmitted to a requesting client which monitors the data rate of the received media. The client inserts locally stored content at one or more marker positions while buffering the received data upon the data rate being below a predetermined level. Playback of the streaming data is resumed upon the data rate returning to the predetermined level.
25 Preferably, the data stream incorporates supplementary information to permit data be to obtained or selected items contained in the video stream.
BRIEF DESCRIPTION OF THE DRAWINGS
An embodiment of the invention will now be described by way of example only with reference 3o to the following drawings in which:
Figure 1 is a schematic block diagram of a multimedia delivery system;

Figure 2 is a schematic block diagram of a media player according to an embodiment of the present invention;
Figure 3 is a flow chart depicting a method for multimedia display according to the present invention;
Figure 4(a) and 4(b) are schematic diagrams of a data buffer in a first state;
and Figure 5(a) and 5(b) are schematic diagrams of a data buffer in a second state.
Figure 6 is a representation of a data stream received by the player of figure 2.
DETAILED DESCRIPTION OF TH:E PREFERRED EMBODIMENT
In the following description, like numerals refer to like structures in the diagrams. Referring to figure 1, a streaming media system is shown generally by numeral 100. The system 100 includes .a streaming media server 112 for serving media files 114 over a network 116 to a requesting client 117. The client 117 is typically a personal computer, hand-held computer or similar device, which includes a memory, a processor and software for providing control instructions to the media server and for displaying the; received media information on a display device 119.
'Typically, this client software is termed a media player and executes within a web browser such as Windows Internet ExplorerTM or Netscape NavigatorTM. Alternately, the media player may be 2o executable without the need for a web browser, such as RealAudioTM. The client computer also includes a storage device 118, such as a hard disk or RAM memory or similar.
On the server side 112, there is included a specialized piece of software known as the media server which may perform one or more functions of compressing the media data, multicasting and multistreaming.
Most commercially available media servers stream the media information using one or more protocols over the Internet standard TC,'P protocol. For example, a media server by VivoTM
offers an HTTP based streaming that enables almost any web site to offer streaming video clips, other vendors like VDONet, offer a UI)P (user datagram protocol) based server.
In general however, both the media server and the client player are matched to communicate via a mutually recognized protocol. The latter media player is downloaded by the user when attempting to 3o access the particular media server 112. Content for the media server 112 may be derived from prestored video, television, audio or similar. Alternatively, the content may be provided by a live feed 120 such as a radio broadcast or television transmission.
Refernng now to figure 2, a schematic diagram of a media player according to an embodiment of the present invention is shov~m generally by numeral 200. The player includes an executive 202 for controlling and conducting operation of the media player and a user input interface 204 for receiving user commands; a :module for determining the incoming data rate 206;
a cache 208 for buffering an incoming data stream under control of the executive 202, a storage for prestored information 210, and a display module 219 for transfernng the received media stream to the 1o display 119 as determined by a scheduling algorithm.
The executive 202 is also connected to a replay cache 220 aperating on a FIFO
principle that holds played media for a predetermined time. The operation of the system may be explained with reference to figure 3. Initially, television commercials or other predetermined information is 15 transferred to the client computer for storage in the prestored information memory 210 at one or more of a plurality of predetermined time periods. These commercials will be used in lieu of streaming on-demand commercials from a server, a.nd will be inserted automatically by the media player 200 when triggered by a, marker as discussed with reference to figure 4.
2o Commercial clips can be downloaded using several methods. The clips can be downloaded when the user switches the computers on. They can also be scheduled to download on a different connection to the streaming video connection in which case priority should be given to the video streaming. Commercials can also be downloaded during the normal commercial breaks that are provided by the broadcasters. While local prerecorded commercials are playing, additional 25 downloaded new commercial clips could be downloaded through a dedicated URL.
Furthermore, new commercials can be transmitted to a user's computer during off peak hours, while the computer is still connected to the Internet.
The media to be delivered from the media server is modified by the addition of markers to the 3o streaming content. Markers rnay be inserted in the data stream broadcast by a broadcasting station indicating the positions in the ;>treaming media wherein the prestored information of commercials 210 may be inserted. These markers may be of various forms. For example, a first type of marker termed "a broadcast marker" may instruct the media player as to the positions for inserting commercials or a second type of marker termed a "supplementary marker" may be inserted in the video stream to indicate positions in the stream where the streaming may be interrupted without affecting the continuity of content as perceived by the user. The insertion of the markers maybe done maamally on prerecorded material or by using a rule-based image analysis algorithm. The rule-based image analysis algorithm may be used to determine critical points, such as a scene change in a video stream or a pause in an audio stream for insertion of the markers.
An additional marker termed "a smart marker" 300 is also embedded within the media stream.
This marker 300 as shown in figure 6, has several fields, namely ID field 302 identifying its name and nature, 302, its pendency or lifetime, 304, a size field 306 indicating the length of data and its properties, a list field 308 indicating a list of properties and a property field 310 for each property containing an ID sub-field 312 and a known length data field 314.
In a typical application, the smart marker 300 is associated with information pertaining to an object to be displayed in the media stream. In such an application, the marker 300 is identified by its ID field 302 as separate from the media stream and the next field indicates the number of 2o active fields in the marker. T:he property field 310 has a number of sub-fields each containing data to indicate operational fianctions. In one example, the sub-fields indicate screen coordinates of an area containing a selected object (310a); an indicator to present and object when the cursor is aligned with the image (310b); a link sub-field indicating the URL address from which additional information can be: obtained (310c); and a launch sub-field indicating an application that can be launched as data regarding that information (31 Ud).
The indicator sub-field 310b may include data to define a change in shape of the cursor when it coincides with the selected object; a text dialogue box providing a message associated with the selected object; an audio message sent in the data field of the sub-field or a text that scrolls on the screen when the obj ect is selected.

Referring back to figure 3, at start up, the media player checks if the media cache buffer 208 as shown in figure 2 has sufficient content for display. If not, the prestored sequence of commercials is sent to the display from the storage 210. The media player continues to check the buffer 208. Upon the buffer being at a sufficient level, the media content is played: In the case of video, the video images are displayed or in the case of audio, the audio is played. As the data stream is played, it is stored in replay cache 220 so as to be available for replay as desired. The replay cache 220 will continue to retain data until full and thereafter delete the oldest data to accept new data stream. The: media player 200 continues to stream data into the buffer 208 while playing. As described earlier, the streaming data rate will depend on the bandwidth of the network. If the average bandwidth, as determined by the media player, is lower than the bandwidth for real time continuous playback then the buffered data is used for playback.
Consequently, the buffered data begins to decrease. When the buffer size reaches a critical point where choking is predicted, the playback is faded out and local commercials are shown from the storage 210, while the buffer 208 to rebuild itself to a sufficient capacity.
In order to avoid interruptions in video sequences which might be annoying to the user, the scheduling of the local content done at the appropriate marker position.
2o If the buffer 208 is depleted below the critical level, the scheduling algorithm is initiated.
Refernng to figure 4 (a) a schematic diagram of the buffer 208 is shown wherein the buffer contains a "broadcast marker" 402. The marker 402 notifies the user that a commercial is to be embedded. The buffer is allowed to deplete beyond the critical level 404, and the local commercial is switched in at the broadcast marker 402.
Referring to figure 4(b), the buffer 208 does not contain a "broadcast marker"
but does contain "supplementary markers", which is provided by a distributor server? The local commercial will be shown at the supplementaay marker that is the last one in the buffer.
3o The algorithm for displaying local content is a dynamic algorithm. For example, if there are two supplementary markers in the: buffer at detection time, and while the buffer depletes, a third marker arnves, the scheduling algorithm will wait for this latest arnval before playing the local information thereby allowing the maximum playback time without interruption.
During local information display, the buffer 208 fills up to sustainable level at the end of the current local information playback at which point normal video streaming resumes.
If, due to excess choking, a large number of commercials were played, the scheduling algorithm adjusts the local content playback where possible and resumes normal media streaming as soon as possible, perhaps even skipping scheduled local information playback. A
total number of to commercial broadcasts are stored in an accumulator. The accumulator is increased for excessive numbers of commercials. Re;fernng to figure 5(a), there is shown a buffer having a broadcast marker 402, which is encountered with a relatively full buffer. In this particular circumstance, other broadcast markers are present in the buffer and the commercial quota has been exceeded.
Therefore, the local content playback is skipped until the next broadcast marker. Referring to 15 figure S(b), there are no other broadcast markers 402, but there are supplementary markers and the commercial quota has been exceeded. The local content playback showing is delayed as long as there are markers in the buffer.
During the periods illustrated in figures S(a) and 5(b), the streaming media server transmits 2o markers while the buffered stream is being played. If the markers can be skipped as previously described, the accumulator is decreased appropriately. When the accumulator is empty, normal local content synchronization can start again at the next available marker.
The size of the buffer 208 is calculated with respect to the statistical measurement of average 25 bandwidth on the channel, and the critical level is determined as a percentage of a full buffer.
In addition to the above scheduling scheme, the case of sustained low bandwidth is addressed. If the average bandwidth drops below the required streaming needs for an extended period, and the local commercial scheduling has been exhausted, an auxiliary connection with the server may be 3o established to instruct the server to decrease the frame rate of the transmission. In that case the server will drop, for example, to 25 or 24 frames per second, or even to 15 frames per second as required. When bandwidth quality resumes to an acceptable level, the server will increase again the frame rate to a higher level, such as 30 frames per second. The synchronization between the client and the server is controlled so that the content is played back.
As the media stream is played, the markers 300 are identified and stripped from the media stream. The markers 300 cooperate with the executor 202 to condition the display in accordance with the contents of the sub-:fields.
For example, the marker 300 may be associated with a product displayed in the video and upon 1o moving the cursor over the object area of the screen is highlighted. The input device 204 controlled by the viewer is activated and the response initiated. Where the sub-field 310c contains a URL, the executor launches the browser and connects to the designated site. The contents on the site are then displayed to the user in a number of possible ways.
15 Where a movie is being viewed, activation of the marker 300 may pause the movie, allowing buffer to fill. Once the action. is complete, i.e. the site ahs been viewed, the connection is closed and the movie can resume playing.
Alternatively, the site contents may be displayed on another window with the main display to 2o avoid interruption of the movie. The jJRL associated with the smart marker 300 may provide for live video or tele-conferencing, thereby giving the user direct access to specific information.
The selected object will remain active for the duration determined by the pendency filed, 304, and thereafter will become dormant unless activated by a subsequent marker.
A plurality of objects may be activated at the same time, either by overlap of the pendency of successive markers or by the properties of a pair of objects being included in each marker. In this case, selection of an object m.ay provide a drop down menu giving a choice of objects to be selected with the properties of each retrieved from the sub-fields and executed by the executor 202.

Where the object selected moves with the display, it is possible to track such movement by successive markers 300 containing new position information periodically in the media stream;
for example, every 30 framers or so.
The provision of the marker 300 is particularly beneficial within the context of streamed media as synchronization of the activation of the object with the relevant position of the video is assured. Similarly, synchronization during replay is maintained and if the server providing the streamed media is switched due to transmission problems, the synchronization is retained.
1o Another feature for improving noticeable quality is having the media player notify a muter which checks for the existence of other connections or routing having a higher bandwidth. When this is detected, the client connection can seamlessly switch to the better connection (similar to telephone channel switching over satellite for overseas routing) without loss of frame and while maintaining frame synchronization.
As described earlier, streaming media servers use a client/server arrangement to allow insertion of streaming media advertisements (commercials). The client end begins a new stream for each inserted commercial in association with a downloaded script. In an alternate embodiment, the streaming media server uses a server push model to insert streaming media advertisements 2o directly into a live broadcast media stream or pre-recorded stream. The server inserts the commercial at a pre-determined time and location in the streaming media, and does not require any feedback or communication from the client.
In a further application of an embodiment of this invention, an on demand marketing and on demand advertising system may be implemented. In this instance, the client computer receives instructions from a user or viewer via a keyboard, mouse or any other suitable input means. The instructions received by the client is forwarded to the server for requesting certain types of advertising content. The content may be specified by the viewer or user in order to receive specifically targeted marketing inforniation during a streaming media session.
More specifically, 3o during a streaming media session, the client may during the playback of local content such as a commercial, interact with the: client computer to connect to an advertiser streaming media site.
to These sites may offer such things as motor vehicles and may also include a live video interactive session. In this instance, the client is provided with a web camera or similar for communication directly with a live person. Thus, the client may include a teleconferencing camera with an audio input to enable a user view to talk live to a customer service representative of the advertiser or the like. During this interaction, the player may continue to stream video into the buffer for resuming playback once the user has terminated the interactive session. The provision of markers 300 facilitates such interaction and permits the marketing to be targeted by particular products.
to A further embodiment of the invention provides for a plurality of channels which are available to the client. Each channel having a unique set of commercials. The server inserts markers corresponding to an appropriate set of commercials or locally stored media in the appropriate channel indicated provision of the allowable content. These in markers may be read by the client, inform the client as to whether the appropriate group of commercials is to be played or 15 not. Generally, the group of commercials may be a predetermined sequence of advertising clips which are thematically related.
The inserted commercial can reside on the same server as the pre-recorded clip or broadcast, or reside on a remote commercial server, and be automatically inserted at the media server end.
In yet another embodiment, a client pull streaming media model allows for communication between the streaming media server and client for determining when commercials are to be inserted. The server generates a script that is read by the client. The client then makes a request for content based on this script. The script can also be modified by the client based on a set of rules or information that it ha.s gathered from the end user.
Once the client has made the request i:or insertion of content, the server responds by modifying it's current stream of information, or launches another stream for the commercial. After the commercial has completed playing, the client issues a request to resume the original content stream.

In yet an alternate embodimf;nt, a targeted streaming media ad insertion model allows the streaming media server to gather user information from client applications installed on a user's computer.
The server collects information from the client application based on the following criteria: The user's responses to forms and survey:.; the user's viewing habits and timelines; the length of viewing; the number of commercials served; the type of content viewed; the user's "hot click"
actions; the user clicking on a commercial to launch a separate HTML/DHTML/XML/JAVA
window; purchases based on commercial content; the user's geographic location;
and any other to information gathered as statistics or usage logs The server then sorts and analyzes the information into a user profile, and allows for narrow target marketing. All of the data collection for this model is based upon a permissions system that requires user permission to use and analyze collected data.
Further, a streaming media portal site is provided as a human indexed directory of video and audio on the Internet. The portal is being provided as a preferred location on the Internet for showcasing and indexing rich media. Users are able to search for video and audio clips that are hosted on other companies' servers, and find scheduled live events from around the world.
The use of human indexing allows for a better quality Internet experience versus sites such as YahooTM, InfoseekTM, and AltavistaT'~~ that feature computer indexed content.
Video and audio are best experienced and described by humans.
The streaming media portal site is positioned extremely well. Current sites indexing video and audio are fragmented and developed by companies with an interest in featuring their own media formats. By offering all video formats, the streaming media portal builds customer loyalty, and can become the one stop for all media on the Internet.
3o The site will be expanded with weekly programming guides delivered directly to users, chatrooms, sponsored live events, and other community building features.

Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art.

Claims

THE EMBODIMENTS OF THE INVENTION IN WHICH AN EXCLUSIVE PROPERTY OR
PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS:

1. A method for transmitting streamed digital data from a server, said server being configured for coupling to a client computer via a computer network, the method comprising the steps of:
(a) providing a modified data stream to said media player, said modified data stream including a plurality of markers located at predetermined positions for indicating new content insertion points;
(b) transmitting said modified data stream to a requesting client;
(c) monitoring the data rate of the received media at the client;
(d) the client inserting locally stored content at one or more marker positions while buffering the received data upon said data rate being below a predetermined level;
and (e) resuming playback of the streaming data upon said data rate returning to said predetermined level.