KR20150055031A - Playback synchronization - Google Patents

Abstract

Various exemplary embodiments relate to a method and media device for synchronizing media playback between a receiving media device and a transmitting media device, comprising: receiving, at a receiving media device, a plurality of messages from a transmitting media device, The plurality of messages including a plurality of transmitter timestamps; Generating a plurality of clock offset values based on a plurality of transmitter time stamps and a clock of the receiving media device; Identifying a minimum clock offset value from among the plurality of clock offset values; Finding a first presentation time associated with the first media data and the first media data for playback; And rendering the first media data at a first time coinciding with the first presentation time based on the minimum clock offset.

Description

Playback Synchronization {PLAYBACK SYNCHRONIZATION}

Cross reference of related application

The present application is related to U.S. Provisional Patent Application No. 61 / 701,326, entitled " Playback Synchronization "filed on September 14, 2012, the entire disclosure of which is incorporated herein by reference. Incorporated herein by reference in its entirety for all purposes as if fully set forth).

This application claims the benefit of US Provisional Application No. 61 / 405,835, filed October 22, 2010, entitled " Media Distribution Architecture ", Lau et al. U.S. Application No. 13 / 278,799, entitled " Media Distribution Architecture "filed on October 21, entitled " Media Distribution Architecture ", the entire disclosures of which are hereby incorporated by reference as if fully set forth herein Incorporated herein by reference in its entirety for all purposes).

The various illustrative embodiments disclosed herein generally relate to media streaming and networked media playback.

As electronic devices such as smart phones and tablets become more popular, people are using these devices more often to play content such as music and video. Often, these media sources may not render the media sufficiently to satisfy the user. For example, the display may be too small, or the speaker volume may not have sufficient quality or volume. Moreover, the output from a media source may not be easily or comfortably enjoyed by a large number of people. In addition, if the user is not carrying a media source, he or she can not enjoy the media at various locations throughout his / her home.

A brief summary of the various exemplary embodiments is provided below. Some simplifications and omissions for the purpose of highlighting and introducing some aspects of various exemplary embodiments, rather than limiting the scope of the invention, may be made in the following description of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of preferred exemplary embodiments will be found in the following sections, which are suitable for enabling a person skilled in the art to make and use the inventive concepts.

The various embodiments described herein relate to a method of synchronizing media playback between a sending media device and a receiving media device, the method comprising receiving, at the receiving media device, a plurality of messages from the receiving media device The plurality of messages including a plurality of sender timestamps; Generating a plurality of clock offset values based on a plurality of transmitter time stamps and a clock of the receiving media device; Identifying a minimum clock offset value from among the plurality of clock offset values; Finding a first presentation time associated with the first media data and the first media data for playback; And rendering the first media data at a first time coinciding with the first presentation time based on the minimum clock offset.

The various embodiments described herein relate to a receiving-side media device for synchronizing media playback with a transmitting-side media device, the receiving-side media device comprising: a memory configured to store media data for playback; A network interface configured to communicate with a transmitting-side media device; And receiving, via the network interface, a plurality of messages from the transmitting media device, the plurality of messages including a plurality of transmitter timestamps; Generate a plurality of clock offset values based on a plurality of transmitter time stamps and a clock of the receiving media device; Identify a minimum clock offset value from among the plurality of clock offset values; Finds a first presentation time associated with the first media data and the first media data for playback; And to render the first media data at a first time that coincides with the first presentation time based on the minimum clock offset value.

The various embodiments described herein relate to non-transitory machine-readable storage media encoded with instructions for a receiving-side media device to execute to synchronize media playback between a transmitting-side media device and a receiving-side media device , The medium comprising, at a receiving-side media device, instructions for receiving a plurality of messages from a transmitting-side media device, the plurality of messages including a plurality of transmitter timestamps; Instructions for generating a plurality of clock offset values based on a plurality of transmitter time stamps and a clock of a receiving media device; Instructions for identifying a minimum clock offset value from among a plurality of clock offset values; Instructions for finding a first presentation time associated with first media data and first media data for playback; And rendering the first media data at a first time that coincides with the first presentation time based on the minimum clock offset value.

Various embodiments further include obtaining a plurality of timestamps from a round-trip transit between the receiving and transmitting media devices; Setting a lower bound offset value based on a plurality of time stamps; And determining that the minimum clock offset value represents a better estimate of the true clock offset between the transmitting device clock and the receiving device clock than the lower limit offset value after identifying the minimum clock offset value And rendering the first media data at a first time that coincides with the first presentation time based on the minimum clock offset is performed when the minimum clock offset value is greater than or equal to the lower limit offset value and the actual clock between the transmitting device clock and the receiving device clock Lt; RTI ID = 0.0 > a < / RTI > better estimate of the offset.

Wherein identifying a minimum clock offset value comprises generating a first clock offset value of a plurality of clock offset values; Setting a minimum clock offset value equal to a first offset clock value; Generating a second clock offset value of the plurality of clock offset values after setting the minimum clock offset value equal to the first offset clock value; Determining that the second clock offset value is less than the minimum clock offset value; And setting the minimum clock offset value equal to the second offset clock value based on determining that the second clock offset value is less than the minimum clock offset value.

Various embodiments further include modifying the value of the clock by subtracting the minimum offset value from the value of the clock of the receiving-side media device, wherein a first time corresponding to the first presentation time, based on the minimum clock offset value, Includes the value of the clock coinciding with the first presentation time.

Various embodiments additionally include, in the receiving media device, receiving a message from the transmitting media device, the message including second media data, a second presentation time, and a transmitter timestamp; Generating a clock offset value based on a transmitter timestamp; Determining that the clock offset value is a more accurate representation of the minimum clock offset value of the actual offset between the clock of the receiving-side media device and the clock of the transmitting-side media device; Adjusting a minimum clock offset value based on the clock offset value; Determining a third presentation time associated with the third media data and the third media data for playback; And rendering the third media data at a second time coinciding with the third presentation time based on the minimum clock offset value after adjusting the minimum clock offset value.

Determining that the clock offset value is a more accurate representation of the minimum clock offset value of the actual offset between the clock of the receiving media device and the clock of the transmitting media device includes determining that the clock offset value is less than zero And adjusting the minimum clock offset value based on the clock offset value includes modifying the value of the clock by subtracting the offset value from the value of the clock of the receiving media device.

Various embodiments further include converting at least one transmitter time stamp of the plurality of time stamps from a time domain of the transmitting media device to a time domain of the receiving media device, wherein the plurality of transmitter timestamps and the receiving Generating a plurality of clock offset values based on a clock of the side media device includes converting at least one transmitter timestamp of the plurality of timestamps from a time domain of the transmitting media device to a time domain of the receiving media device And generating at least one clock offset value based on one transmitter time stamp.

The method comprising: generating a first measure of intervals at which a first plurality of messages arrive, the plurality of messages comprising a first plurality of messages and a second plurality of messages; Determining that a first measure of intervals at which a first plurality of messages arrive indicates that the network is unstable; Instructing the sending media device to send additional messages; Generating a second measure of intervals at which a second plurality of messages arrive after instructing the sending media device to send additional messages; Determining that a second measure of the intervals at which a second plurality of messages arrive indicates that the network is stable, wherein identifying a minimum clock offset value from among the plurality of clock offset values is associated with a second plurality of messages Various embodiments are described that include utilizing at least one clock offset value of a plurality of clock offset values.

Wherein the plurality of messages comprises a first plurality of messages and a second plurality of messages, the method further comprising the steps of the transmitting media device transmitting a first plurality of messages, the transmitting media device transmitting a first plurality of messages Determining that the first network performance metric indicates that the network is unstable; determining if the first network performance metric indicates that the network is unstable; Generating a second network performance metric associated with a transmission of a second plurality of messages, and wherein the second network performance metric indicates that the network is stable Various embodiments are further described that include the step of determining All.

BRIEF DESCRIPTION OF THE DRAWINGS For a better understanding of the various exemplary embodiments, reference is made to the accompanying drawings.
1 illustrates an exemplary environment for media playback;
2 illustrates an exemplary method of forming and operating a virtual media network;
3 illustrates an exemplary virtual media network;
4 shows an exemplary component diagram of a media source;
5 shows an exemplary component diagram of a media node;
6 illustrates one exemplary hardware diagram of a media device;
7 illustrates an exemplary method for broadcasting a media signal;
8 illustrates an exemplary method by which a transmitting-side media device synchronizes playback with a receiving-side media device;
9 illustrates an exemplary method by which a receiving-side media device synchronizes playback with a transmitting-side media device;
10 illustrates an exemplary method as long as the receiving media device achieves better playback synchronization during media streaming;
11 illustrates an exemplary method for determining a lower bound offset;

The description and drawings presented herein illustrate various principles. It will be appreciated by those of ordinary skill in the art that, although not explicitly described or shown herein, it is possible to implement various principles that are within the scope of the present disclosure. As used herein, the term "or ", as used herein, unless otherwise indicated (e.g.," or otherwise "or" or alternatively "), non-exclusive, or -exclusive or) (i.e., and / or). In addition, the various embodiments described herein are not necessarily mutually exclusive, and may be combined to produce additional embodiments that include the principles described herein.

The various embodiments described herein utilize an architecture for distributing media content. Wired and wireless media transmission techniques may be provided that enable simultaneous transmission of media to multiple zones while maintaining accurate timing synchronization between various media devices. The user may have a network of speakers, displays, or other rendering devices, and may independently select which rendering devices are actively outputting the media. These rendering devices, along with other devices to be described herein, may belong to a virtual media network.

The media rendered by the virtual media network may originate from a media source. The media source may be a cell phone, tablet, stereo, set-top box, PC, or other device. The method of transmitting media in a virtual media network may be wired, such as via an auxiliary cable, or wireless, such as in Bluetooth or WiFi. Speakers and other rendering devices themselves can be controlled in a self-forming network. Media can be injected into the network by the media source and the endpoint network itself can control audio / video distribution, timing and rendering. In some embodiments, audio injected into the network is an audio portion of an audio-video signal. The video signal may be played on a media source (e.g., a tablet computer). The audio signal may remain synchronized with the video signal.

In various embodiments, the user may select any media application that will serve as the source of the media. For example, a user may select an MP3 application, an Internet radio application, and the like. The user can then select an output device, such as a speaker in its living room, to cause the media to be transmitted to the selected output device. Audio may be transmitted by the operating system to the selected output device. The user may call the second application for controlling the volume of the speakers as well as adding other speakers to the virtual media network, In some embodiments, the second application never modifies the media. Devices in the network can handle audio / video distribution, timing, and rendering. Therefore, the media source may not be burdened with such processing. Moreover, such a configuration would allow the user to select any media application of his or her choice as the source of the media without having to modify the media application.

In various embodiments, the media distributed over the virtual media network may remain synchronized. To achieve this playback synchronization, various media devices transmitting media data may include timestamps associated with frames of media data to indicate when the associated media should be rendered. To enable such a mechanism, the media devices may have methods for considering differences between the internal clocks of the media devices. For example, the two media devices may start to operate at various clock values, or the clock values may drift during operation due to the clocks operating at slightly different speeds.

Although some methods of networked clock synchronization have been developed in the past, these methods may be reliable only in wired networks where the network delay is relatively constant and therefore easily considered as part of the rendering process. On the other hand, in wireless networks such as WiFi or Bluetooth, the network delay can vary considerably over short periods, thereby complicating the process of estimating when a synchronization packet is transmitted. The various methods described herein implement a clock synchronization process that reduces or eliminates the effect of this variable delay on the process of networked clock synchronization. For example, by generating a number of potential clock offset values over a period of time, the receiving device can select a minimum offset value from among the groups and thereby use at least the offset value affected by the variable component of the network delay. Various additional features for improving networked clock synchronization will be described in greater detail below.

Throughout this description, the following definitions will be used:

Broadcaster - Any device capable of transmitting media streams formatted for virtual media networks, or a broadcasting mechanism within such devices.

Renderer - Any device capable of rendering a media stream that is formatted for a virtual media network, or a rendering mechanism within such a device.

Media Node - Any device that contains a renderer or broadcaster. The media nodes of some embodiments may be in charge of maintaining the state of the network, including network time synchronization and media routing information.

Media source - Any device that transmits source media to a sink. For example, a cellular phone, a smartphone, a tablet, a set top box, a television, a DVD / Blu-ray / other media player, a stereo system, a video game console, a laptop, a desktop PC, Devices, and the like.

Sync - Any device that receives media originating from a source or mechanism in a device for receiving media signals.

Gateway Capable Media Node - Any device that has a sink and broadcaster. The gateway receives the media through the sink and rebroadcasts it into the renderers into the virtual media network.

Virtual Media Network - A group of one or more nodes having at least one gateway. The virtual media network can be set by the user and render media signals that are synchronized between rendering devices in the network. It should be noted that, in some embodiments, only one media node acts as an active gateway of the virtual media network.

Media device - any device that operates in conjunction with a virtual media network, such as, for example, a media node or a media source.

Various aspects of various exemplary embodiments are now described with reference to the drawings in which like reference numerals refer to like elements or steps.

Figure 1 illustrates one exemplary environment 100 for media playback. In this example, there are a total of five network media nodes 104a, 104b, 106a-106c; The various exemplary embodiments may include fewer or greater numbers of media nodes (not shown). The exemplary embodiment 100 is shown as consisting of two virtual media networks. As shown, media source 102a serves as a source of media signals for one virtual media network, while media source 102b serves as a media source for other virtual media networks, although other configurations are possible . The media signal may be audio or video. In various embodiments, the media signal is an audio portion of an audio-video signal. The video signal may be played on the media sources 102a, 102b. It should be noted that, in the exemplary embodiment, when the various signals are rendered by different devices, the audio signal remains synchronized with the video signal. It should also be noted that the video signal may be sent to one of the devices in the virtual media network or to some device other than the media source node 102a, 102b. In various embodiments, each virtual media network has one gateway device, while in other embodiments, the virtual media network uses multiple gateway devices. As discussed above, the gateway device has a sink and a broadcaster that receive media signals. The gateway device may or may not have a renderer that renders audio and / or video. In the illustrated example, the device in the living room acts as a gateway to the first virtual media network, but a different device with a broadcaster can act as a gateway.

In some embodiments, the system enables simultaneous transmission of media to multiple regions while maintaining accurate timing synchronization. As an example, the user can configure the network of speakers and independently select which speakers are actively playing and synchronizing their playback. The method of transmitting media to the network may be wired, such as via an auxiliary cable, or wireless, such as in a Bluetooth, WiFi or other network communication protocol. As an example, the living room gateway 104a may provide an auxiliary output line (not shown) that provides the media signal to the speakers 110 connected thereto by one of the stereo receiver 108 and its auxiliary lines line. On the other hand, the living room gateway 104a may provide media signals to the office renderer 106a and the kitchen renderer 106b via wireless transmission. In addition, the living room gateway 104a may or may not have its own renderer. In some embodiments, various media nodes belonging to the network receive and render different channels of the media stream. For example, a media source may render a video signal, a first renderer may render a left speaker channel of a stereo mix audio signal, and a second renderer may render a stereo mix audio signal Channel, and the gateway can render both channels of the video signal and the stereo mix audio signal. Various different channel schemes and distributions of these channels between media devices will be apparent.

In some embodiments, the media nodes 104a, 104b, 106a through 106c themselves are controlled in a self-organizing network. Media nodes 104a, 104b, 106a through 106c themselves may control audio / video distribution, timing, and rendering. Thus, much of the processing load can be removed from the media source 102. The example of FIG. 1 relates to a home environment, but the embodiments are not limited thereto. Virtual media networks can be installed in any environment.

FIG. 2 illustrates an exemplary method 200 for forming and operating a virtual media network. In step 202, the media devices discover each other and exchange device status information. Step 202 may occur, for example, when the media nodes 104 and 106 are powered on. Because the media nodes 104 and 106 may be powered on at different times, this step may continue, repeat, or otherwise proceed. In some embodiments, the media nodes 104, 106 may be configured to allow the media nodes 104, 106 to have the ability to act as the presence of each other and, for example, as a source, sink, broadcaster, Quot; self-discovery " protocol (see " self-discovery " The exchanged device status information may also include information such as, for example, whether the device is currently active in the virtual media network, the identity of the virtual media network, whether the device is currently acting as a gateway, .

At step 204, the media source 102 is paired with the gateway media node 104. The user can select one media node 104 to be served as a gateway, or the gateway can be automatically determined without user intervention. For example, a user of the smartphone 102a may select the living room media node 104a as the primary listening device, which results in a gateway. In some embodiments, the gateway media node 104 is selected as the currently active output device for the media source node 102 based on its state. In some embodiments, the gateway media node 104 serves as a gateway and acts as an active output device for the media source node 102, thereby rendering at least some of the channels of media data. In some embodiments, the gateway media node 104 reports device or status information to the media source 102.

At step 206, a virtual media network is formed. Step 206 may be performed in response to the user selecting media nodes 104 and 106. For example, a user may access a software program on a media source 102 that allows a user to select media nodes 104, 106. It should be noted that if the media nodes 104 and 106 are already part of a different virtual media network then this media node 104 and 106 may be marked as not available through the media source 102. [ Additionally or alternatively, the user may request that the media node 104 (106) in use be freed for inclusion in the current virtual media network. In various embodiments, step 206 directs the gateway media node 104 to forward the media signal to the other media nodes 104, 106 in the virtual media network.

At step 208, media may be transmitted from the media source 102 to the gateway media node 104. [ This step 208 may be initiated in response to the user selecting the media to be provided on the output device associated with the media source. For example, a user may be running any application that plays media on the smartphone 102a. The user can then select the gateway media node 104 as an output device and the media can be transmitted to the gateway media node 104. [ It should be noted that this media transfer may occur at the operating system (OS) level. This transmission implies that any media application may be selected by the user as a media source for the virtual media network.

At step 210, the gateway media node 104 may broadcast the media signal to the other media nodes 104 in the virtual media network. For example, the living room gateway 104a may broadcast the media signals he received from the smartphone 102a to the office renderer 106a, the kitchen renderer 106b, and the stereo receiver 108. In some embodiments, each media node 104, 106 plays media at its own user-controllable level (e.g., volume). Thus, certain commands may be transmitted from the media source 102 to the gateway media node 104. However, the gateway can perform much of the processing. Therefore, the media source 102 does not become stuck with a heavy processing load.

FIG. 3 illustrates an exemplary virtual media network 300. FIG. As shown, since it has the sinks 322 and 332 for receiving the media signal and the broadcasters 324 and 334 for providing the media signal to the other media nodes 320, 330 and 340, There are two media nodes 320, 330 that can be used. For purposes of illustration, there is an access point 350 that is separate from the media nodes 320, 330, and 340. It should be noted that one of the media nodes 320, 330, 340 may function as an access point.

Some of the media nodes 320, 330 include broadcasters 324, 334. These nodes may be referred to herein as broadcasting nodes. Broadcasters 324 and 334 may be implemented by any combination of hardware or software. In various embodiments, broadcasters 324 and 334 transmit media in an airtime broadcast format that is understood by other media nodes 320, 330, and 340. It is noted that this format may be different from the format used to transmit the media 360 from the media source 310. [ Broadcasters 324 and 334 and renderers 326 and 336 may coexist in the same media node 320 and 330 so that local playback can be synchronized with playback on remote renderers. Source injection can be done through a source-sink link. Unlike source-to-sink transmission, airtime broadcast can be used for point-to-multipoint media transmission with synchronous playback.

As we have seen, the gateway supported media nodes 320 and 330 have sinks 322 and 332 and broadcasters 324 and 336, respectively. In some embodiments, gateways 320 and 330 receive media from media source 310 and rebroadcast media in a format compatible with other media nodes 320, 330, 340 in the virtual media network . The gateways 320 and 330 also include the renderers 326 and 336. In various embodiments, the gateway media node 320, 330 is considered an endpoint.

A number of gateway supported media nodes 320, 330 may reside on the network. In some embodiments, the gateway media nodes 320,330 use an election method to determine the best gateway for the media source 310 to use. For example, if only one media node 320, 330 with a renderer 326, 336 is active for the media source 310, then the render node may also be the best gateway, Reduce network bandwidth. On the other hand, if multiple renderers are active for media source 310, then the best gateway may be the strongest or best network connection. An election scheme may be made to identify the best candidates and stream handoffs to different gateways 320 and 330 may occur if necessary and in this case the original gateways 320 and 330 may be a source 310). This may occur during stream construction or mid-stream. If the active gateway is disabled, the network may self-heal and may elect a new gateway to reset airtime broadcast streams.

Some of the media nodes 320, 330, 340 include a renderer 326, 336, 346. These media nodes 320,330, 340 may be referred to herein as rendering nodes. The renderers 326, 336, 346 may be implemented by any combination of hardware or software. When using the example of audio for the media signal, the renderers 326, 336, 346 may be configured to provide a media stream through analog or digital outputs to an amplifier / speaker device, Decoded and played back. In the case of video, the renderers 326, 336, 346 may decode and play the media stream through an internal power display or through analog or digital outputs to a device that has or drives another display or display. In various embodiments, the media nodes 320, 330, 340 with the renderers 326, 336, 346 support the creation, maintenance and distribution of virtual wall clocks. The renderers 326, 336, 346 may use the wall clock to accurately render the stream at the time stamp specified in the broadcast time stream format.

In the example of FIG. 3, there is a connection between the media source 310 and the sink 322 in the gateway media node 320. The media 360 is played by the renderer 326 in the gateway media node 320. To establish the connection, the user may have selected the gateway media node 320 as an output device for the media source 310. [ For example, the media source 310 may be a cellular phone that allows the user to select which speaker to send the audio to. Any audio being played back by the cellular phone may be transmitted to the selected speaker. As such, audio may be routed to the gateway media node 320, regardless of which application is providing audio (e.g., Internet radio, MP3, etc.). It should be noted that no changes need to be made to the application providing the audio to do this. The connection between the media source 310 and the gateway media node 320 may be wireless or wired. In various embodiments, this is a wireless Bluetooth connection. However, wireless protocols other than Bluetooth may be used.

The broadcaster 324 in the media node 320 may receive the media 360 from the renderer 336 in the media node 330. In addition to the connection between the media source 310 and the sink 322 in the gateway media node 320, And to the renderer 346 in the media node 340. In this example, the access point 350 acts as an intermediary. However, the access point 350 may not be necessary. In various embodiments, the media node 320 serves as an access point. Connections from media source 310 to media node 330 and media node 340 may have been established in a manner similar to the connection between media source 310 and media node 320. [ The user may also have configured the media nodes 330, 340 as part of the virtual media network 300. Media source 310 may have a software application that allows a user to select which media nodes 320, 330, 340 to add to the virtual network. The application may send commands to the media node 320 instructing the media node 320 to forward the media signal to other media nodes 330,340 that are active portions of the virtual media network. The media node 320 may process details such as reformatting the media signal, routing, synchronizing playback between the media nodes, and so on. Thus, the media source 310 is not burdened with heavy processing.

It will be appreciated that the virtual media network 300 is only one possible configuration of one possible device set. The various alternative media networks 300 may include fewer or additional devices and may distribute the media in a different manner. For example, media source 310 may send media directly to an access point, media node 330 may act as a gateway instead of media node 320, and media node 340 may participate in a virtual network I can not. Various alternative arrangements will be apparent.

As discussed above, media sources 310 may inject media 360 into virtual media network 300. Examples include PCs or smartphones. Available media injection methods may include cables that support analog or digital transmission, Bluetooth, and WiFi. In some embodiments, the media source 310 is a broadcaster that transmits media data in a format compatible with the virtual media network. In other embodiments, the technical limitations limit the ability of the media source 310 to broadcast. For example, the security model of many phones may prevent audio drivers from being modified by third parties. In addition, the media source device 310 itself may not have available processing or network bandwidth. In addition, in some embodiments, the QoS level for the initial link of the media source utilizes a higher QoS than other endpoints, so that at least one endpoint can render with the highest possible fidelity.

It is noted that many formats and connections may be used for transmission from the media source 310 to the sink 322. The media source 310 may transmit to the sink 322 via a wired, BT A2DP, or certain protocol over WiFi, as some non-limiting examples. The WiFi protocol may be designed to provide a tradeoff between quality and latency or to ensure accuracy. For example, the protocol may detect errors and request retransmission of data. Often this may not be the purpose of the broadcast, but it is important that the media arrive securely before broadcasting. The embodiments disclosed herein maintain compatibility with existing devices.

In various embodiments, the network is based on a standard WiFi infrastructure. Each media node may be connected to an access point 350 through which it obtains an IP address through DHCP. Some nodes 310, 320, and 330 may not have a UI (display, keyboard input, etc.) that allows input of a wireless access key. In these cases, WPS-PBC may be used to achieve the connection. Other methods may include an ad hoc mode in which a user connects to an endpoint directly from a GUI enabled device and enters network parameters via a web page serviced by the node, or an application page communicating directly with the node. Another way is for an application running on the phone or other device to communicate with the media node via Bluetooth. The application can ask the user which access point to connect to and the corresponding network access code. In some embodiments, the media node 320, 330, 340 is given a name by the user during this setup step.

In the absence of an infrastructure such as access point 350, the node may be a virtual access point. Other nodes may be connected to discover the access point 350 and form a private network. WPS-PBC and ad hoc methods can be used to achieve secure connections.

4 depicts an exemplary component diagram of a media source 400. As shown in FIG. The media source 400 may correspond to either the media sources 102a and 102b of the exemplary environment 100 or the media sources 310 of the exemplary virtual media network 300. [ The media source 400 may include a network interface 410. In various embodiments, network interface 410 includes a number of separate interfaces. For example, the network interface 410 may include a WiFi compatible interface and a Bluetooth compatible interface. Additionally or alternatively, the network interface 410 may include interfaces compatible with any other protocols. In this example, a media signal (e.g., an audio stream or a video stream) may be transmitted using the Bluetooth compatible interface of the network interface 410. The WiFi compatible interface of the network interface 410 may be used to transmit commands for controlling the virtual media network.

The user may access the virtual network media application 420 to control the virtual media network. As one example, the virtual network media application 420 may provide a user interface that allows a user to select media nodes 104, 106 and control their volume, playback, and the like. In some embodiments, there are a master volume for the network and individual volumes for each media node 104,106.

The media source application 430 may be any application capable of playing audio on the media source 400. For example, the media source application may be an MP3 player, Internet audio, web browser, and the like. In various embodiments, the media will be played on the output device no matter which output device is selected by the user. This output device selection may be under the control of the operating system (OS) 440. For example, the OS 440 may provide a pop-up window that allows the user to select an output device. One or more of the media nodes 104, 106 may appear as selectable ones. By selecting one of the media nodes 104 and 106, a media signal associated with the audio application can be transmitted from the media source 400 via the network interface 410 to the selected media node 104, 106. In some embodiments, the media library 450 is used to decode the media. The media library sends the decoded media to the network media driver 445 and the network media driver 445 sends the media signal to the selected output device. If the media nodes 104, 106 are selected as output devices, the media signals are transmitted via the network interface 450. In some embodiments, the network media driver 445 is a Bluetooth driver. However, the network media driver 445 may be compatible with any protocol.

Note that, in the above embodiment, the virtual media application 420 never touches the media signal. This has the advantage that any media source 430 can be used when transmitting the media signal to the media node 104, 106, by selecting the appropriate output device for the media source 400. As such, some embodiments of the virtual network media application 420 are compatible with any media source application 430. Moreover, no changes to the media source application 430 may be required.

As discussed above, some embodiments of the gateway media node 104 may perform reformatting and processing of media signals such that the media signals are compatible with the virtual media network. As such, the gateway media node 104 relieves much of the processing from the media source 102.

It will be appreciated that the media source 400 is an illustration of one example and that many modifications may be made to the media source 400 while still implementing the methods and techniques described herein. For example, in some embodiments, the network media driver 445 includes a virtual network media driver, and the virtual network media application 420 may not be present. In this embodiment, a user may install a virtual media network driver 445 to help send media signals to the media nodes 104, 106. When a user wishes to transmit a media signal to media nodes 104 and 106, the user simply selects the media node from the interface provided by OS 440. [ This selects the virtual network media driver 445. For example, a media signal may be provided from the media library 450 to the virtual network media driver 445. As in the previous example, the media source application 430 may be any application used to play media.

The virtual network media application 420 may be similar to that previously described. For example, the virtual network media application 420 may provide an interface for a user to select the media nodes 104, 106 to add to the virtual network and to control the network. In some embodiments, the virtual network media application 420 is optional, because its functionality may be included in the virtual network media driver 445. [

In addition, a command channel may be used to transmit commands, and a data channel may be used to transmit media signals via network interface 410 using the same protocol. For example, both commands and data may be transmitted in accordance with the WiFi protocol or the Bluetooth protocol. As previously noted, commands and data may alternatively be transmitted in accordance with different protocols.

It is noted that media signals from any media source application 430 may be transmitted to media nodes 104 and 106 by including driver 445 in OS 440. [ The user has only to select one of the media nodes 104, 106. In response, a virtual network media driver 445 is used. Thus, a virtual media network may be used in any media source application 430 running on the media source 400.

In some embodiments, a media source application 430 is embedded in the virtual network media application 420. In some of these embodiments, any media played by the media source application 430 is transmitted to the media nodes 104, 106.

In various embodiments, the media is simultaneously rendered by the media source 400 and the media nodes 104, 106 belonging to the virtual media network. For example, media source 400 may render a video channel of media, while media nodes 104 and 106 may render audio channels of media. In some of these embodiments, the various media channels remain synchronized. For example, the media source 400 may transmit timestamps with the media data to indicate when the media data should be rendered by the media nodes 104, 106. Similarly, the gateway 104 may also include timestamps associated with the media data when forwarding the media data to other media nodes 104, 106. To enable this exchange of timestamps, the various media devices 104,106, 400 may have a common reference clock, such as a virtual wall clock, or a method of transforming timestamps between various time regions, Will be described in more detail below.

FIG. 5 illustrates an exemplary component diagram of a media node 500. FIG. The media node 500 may correspond to one or more of the media nodes 104, 106 of the exemplary environment 100. The media node 500 may have a network interface 510. The network interface 510 may enable communication according to one or more wireless or wired protocols. In various embodiments, one or more antennas are coupled to the network interface 510. In some embodiments, the network interface 510 is WiFi compatible and Bluetooth compatible. Additionally or alternatively, the network interface 510 may be compatible with any other protocol. In some embodiments, the network interface 510 includes one or more wired network interfaces.

The renderer 520 may be in charge of processing the media signals for presentation via the speakers 530, the display 540, or other output device (not shown). It will be appreciated that various alternative media nodes may not include speakers 530 or display 540, depending on the type of media the media node is designed to render. In addition, the media node 500 may not include the renderer 520, in which case the media node 500 is designed to function only as a gateway or other broadcaster, not as a rendering device. The rendering module may receive the media signal from the network interface 510.

The broadcaster 550 will be able to forward the media signal to the other suitable media nodes 104,106 via the network interface 510. [ An auxiliary output 560 may be used to provide the media signal to a device such as a home stereo system. In some embodiments, the broadcaster 550 processes forwarding the media signals to the auxiliary output 560. [ In various embodiments, the media node 500 does not include an auxiliary output 560. In addition, the media node 500 may not include a broadcaster 550, in which case the media node 500 is designed to function only as a rendering device, not as a gateway or other broadcasting device.

Command module 570 may be able to process commands that control the media signal. These commands may include volume, play, pause, and so on. The synchronization module 580 may be responsible for accurate synchronization of media signals during playback on various media nodes in the network. As will be described in greater detail below, the synchronization module 580 may send or receive beacon messages for use to establish initial clock synchronization. In addition, after the media stream is started, the synchronization module 580 may insert or extract timestamps in the media packets for use in improving or correcting clock synchronization during media playback.

The media nodes 104 and 106 may be controlled through various mechanisms. The controllers may include a SmartPhone App, a Tablet App, a UI on a TV or set-top box, buttons with or without a display on the node, or a PC app (PC app). In some embodiments, the devices may control whether the renderer 520 renders a particular stream or its particular channels, the volume output of the renderer 520, and the master volume.

In some embodiments, the media node 500 supports a command protocol. The command protocol includes a method for turning on / off audio playback, a method for integrating audio playback into synchronized zones, a method for transmitting controls such as playback, fast forward, rewind and seek, a method for transmitting metadata to nodes, A method of notifying devices that have joined the network, a method of updating status when devices leave the network, a method of controlling via remote user interfaces, and other messages and methods for maintaining a broadcast time network Lt; / RTI >

It should be noted that the elements of the media node 500 may be implemented in software, hardware, or a combination of software and hardware. The media node 500 may have one or more processors, and a computer-readable storage medium having instructions that, when executed on one or more processors, implement the functionality of the various elements of the media node 500.

6 depicts an exemplary hardware diagram of media device 600. Media device 600 includes a plurality of media devices. Exemplary media device 600 may correspond to any of media devices 102, 104, 106, media source 400, or media node 500 in exemplary environment 100. As shown, the hardware device 600 includes a processor 610, a memory 620, a user interface 630, a network interface 640, and a storage 650 interconnected via one or more system buses 660. [ . ≪ / RTI > It will be appreciated that FIG. 6, at some points, constitutes an abstraction, and that the actual configuration of the components of the media device 600 may be more complex than illustrated. For example, the processor 610 and the memory 620 may be connected via a local microprocessor bus and the user interface 630, the network interface 640, and the storage 650 may be connected via one or more input / have.

The processor 610 may be any hardware device capable of executing instructions stored in the memory 620 or the storage 650. Accordingly, the processor may include a microprocessor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other similar devices.

Memory 620 may include various memories, such as, for example, L1, L2 or L3 caches or system memory. Accordingly, memory 620 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.

The user interface 630 may include one or more devices that enable communication with a user or rendering media for a user. For example, the user interface 630 may include a display, speakers, printer, auxiliary output, mouse, keyboard, alphanumeric keypad, trackball, stylus, or buttons.

Network interface 640 may include one or more devices that enable communication with other hardware devices. For example, network interface 640 may include one or more network interface cards (NICs) configured to communicate in accordance with Ethernet protocols, TCP / IP protocols, WiFi protocols, or Bluetooth protocols. Various alternative or additional hardware or configurations for network interface 640 will be apparent.

Storage 650 includes one or more machine-readable storage media, such as read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, can do. The repository may also include a portable non-volatile storage medium, such as a floppy disk, for inputting and outputting data and code to and from the media device 600. In various embodiments, the store 650 stores instructions for the processor 610 to execute or data that the processor 510 can process. For example, the repository may include an operating system 670 for coordinating the basic functions of the media device 600, and other media devices for transferring sufficient information to synchronize playback or for synchronizing playback And may store synchronization instructions 672 for processing such information being transmitted.

Depending on the role or capability of the media device 600, the repository 650 may store various additional instructions. For example, if the media device 600 can function as a media source, the storage 650 may include media source application commands 674 for acquiring or reproducing media on the device 600, Media library commands 676, or virtual network media application commands 678 for enabling a user to transmit commands to the virtual media network. Based on the above description, such as the description of exemplary media source 400, various other functions for these commands 674, 676, 678 will be apparent. When the media device 600 is additionally or alternatively capable of functioning as a media node, the storage may include command module instructions 680 for processing commands issued by a media source or other controller, Broadcaster commands 682 for forwarding to other media nodes, and renderer commands 684 for rendering media in sync with other devices. Based on the above description, such as the description of exemplary media node 500, various other functions for these commands 680, 682, 684 will be apparent.

The components contained in the media device 600 are those typically found in computer systems suitable for use in the methods and systems described herein and are intended to represent a broad category of such computer components . As such, the media device 600 may be a cellular phone, a smartphone, a PDA, a tablet computer, a personal computer, a mobile computing device, a workstation, a server, a microcomputer, a mainframe computer, or any other computing device. The computer may also include different bus configurations, networked platforms, multi-processor platforms, and the like. Various operating systems 660, including Unix, Linux, Windows, Macintosh OS, Palm OS, Android OS, iOS, and other suitable operating systems ) Can be used.

Figure 7 illustrates an exemplary method 700 for broadcasting a media signal. Method 700 may correspond to step 210 of method 200. In step 710, the gateway media node 104 and other media nodes 102, 104, 106 may set timing parameters. In some embodiments, the media nodes 104, 106 are synchronized to a virtual wall clock. A virtual wall clock can be used by the broadcaster to timestamp the media stream with the intended rendering time. A virtual wall clock can be used by the renderers to accurately render media samples at a given time. The virtual wall clock can help ensure that the media nodes 104, 106 have a common understanding of the rendering time. In some embodiments, each rendering device renders samples at a time specified in the media stream. Other information for rendering the stream, including sampling frequency, word size, number of channels, encoding format, etc., may also be included in the stream format.

In step 710, a virtual wall clock or some other common timing reference may be set. For example, the gateway media node 104 may initiate a "flood" of beacon messages that include timestamps indicating when each beacon message was transmitted. The receiving media node 104, 106 may then calculate the offset value by determining the difference between the time stamp of the transmitter and the time each respective beacon message is transmitted. This calculated offset can reflect the sum of three independent values: the actual offset between the clocks of the transmitting and receiving devices; Time such as network propagation time, the time between the insertion of the time stamp by the transmitting device and the actual transmission of the beacon message, and the time between the receiving device determining the receiving time of the beacon message and the receiving time A fixed delay associated with < RTI ID = 0.0 > And a variable delay associated with network variations and common to various wireless network connections. Because the fixed delay is substantially constant, the receiving device knows that the smallest calculated offset is the offset, including the smallest variable network delay, among the calculated offsets, and thus the "actual clock offset + fixed network delay" It can be assured that it is the closest estimate available. The receiving device may then adjust its own clock based on the minimum offset or may sustain the offset for use in comparing subsequent transmitting device timestamps with the local clock. Fixed network delays can be considered during the rendering process to ensure proper synchronization. Similar methods may be used to establish synchronization between the media source 102 and the media gateway 104.

At step 720, the gateway media node 104 receives a media signal from the media source 102. [ At step 730, the gateway media node 104 decodes the media. The gateway may de-multiplex the media signal prior to decoding.

At step 740, the gateway media node 104 re-encodes the media to broadcast to the other media nodes 104,106. It should be noted that the gateway may use an encoding different from the media source 102 in which it is used. For example, the media signal may have been encoded in media source 102 in a format compatible with Bluetooth. The media signal can be re-encoded in a format compatible with WiFi.

At step 750, the gateway media node 104 encapsulates the media signal. In various embodiments, the gateway media node 104 compresses the media signal. As an example of compressing audio media signals, in high quality networks, light lossless compression techniques such as Free Audio Lossless Codec (FLAC) may be used to reduce the bandwidth in half with minimal processing overhead. In low quality networks, higher compression standards such as OGG or AAC (Advance Audio Coding) may be used to minimize network bandwidth in exchange for sound quality and processing overhead. In addition to the compression algorithm itself, the signal may be resampled at a lower sampling rate, down-mixed into a mono stream, or down-sampled at a lower sample resolution. have. Encoding or transcoding media streams in compressed form can improve airtime reliability by using less network bandwidth in exchange for processing overhead. The supported codecs may include lossless and lossy compression techniques of various bit rates, sampling frequencies, channels, and sample sizes.

In some embodiments, all of the media nodes 104, 106 are aware of the supported encoding formats. In some embodiments, all broadcasters may encode in supported formats. In some embodiments, all renderers may decode the supported formats. The encoding format used for each stream is determined by feedback from network quality, available processing resources, the number of rendering zones supported, the number of active streams supported, and the maximum allowed delay time, (104, 106).

At optional step 760, redundant packets are added. If the media signal is compressed, additional packets may be added. In some embodiments, a group of packets is interleaved with a group of redundant packets. For example, in the case of a 2: 1 compression ratio, a two second source media signal can be compressed in one second. As an example, one second of data packets may be interleaved with one second of redundant packets. The number of packets in the group may be one or more.

In some embodiments, broadcasting has two options. In Option A, as illustrated, at step 770, the gateway media node 104 may broadcast the media signal to the other media nodes 104. In Option B (not shown), the gateway media node 104 may send a media signal to the wireless access point. The wireless access point may broadcast the media signal to other media nodes.

Broadcast media can be the largest consumer of network bandwidth. Typical uncompressed audio streams may exceed 1.5 mbps. The transmission may consume 1.5 mbps per stream upstream to access point 310 and an additional 1.5 mbps per stream downstream to renderer 306, a total of 3 mbps. For point-to-point simulcasting, the typical bandwidth may be "3 mbps x number of streams < RTI ID = 0.0 > This is likely to saturate the network.

Various embodiments support multiple transport protocols. In some embodiments, UDP over IP is used. Note that, in some embodiments, the receiving media node does not need to acknowledge receipt of packets. For example, UDP over IP may not require reception of packets. In some embodiments, the receiving media node may request the gateway to retransmit data packets not received. Note that this can happen in one embodiment using UDP over IP. As noted above, in some embodiments, redundant data packets are transmitted.

Network statistics may be maintained by the media nodes 104, 106. In various embodiments, the elected broadcaster or gateway is in charge of determining the best transmission methods to balance service quality, latency, processor utilization, and network utilization. For example, a guaranteed transmission protocol may be used if the network has good quality, due to the high available bandwidth and strong connections to the individual nodes 104,106. If the network is saturated or poor quality, a multicasting technique may be desirable. Additional methods can help to reduce bandwidth and detect, correct, or conceal transmission errors. Generally, multicasting, simulcasting, and point-to-point protocols are supported in the most appropriate protocol determined at the time of stream configuration, and network quality, available processing power, and number of streams are contributing factors in the decision process.

The media clock can be recovered via the media stream with reference to the wall clock and synchronized to media frames or sample groups. The media clock can lead to the formation of hardware frame clocks, word clocks, and bit clocks. Synchronization through the media stream can ensure that accurate clocks are generated at media nodes 104 and 106 from a logical point of view. Some variations in hardware, such as crystal, can cause clock drift and other variations in clock timing. The constant measurement and comparison of media clocks and wall clocks allows the system to detect drift. In some embodiments, the software-only media clock recovery mechanism involves adding or removing media samples to media render buffers to resynchronize media clocks across the devices. In some embodiments, rendering buffer manipulation is done in a manner that does not result in obvious clicking or skipping effects. A hardware mechanism using voltage controlled oscillators (VCXOs) can be controlled from the processor based on drift measurements and can push or pull hardware oscillators with more stringent synchronization.

As discussed above, the various systems described herein can synchronize media playback among multiple devices by setting a common timing reference. For example, media sources and media gateways can cooperate to set up these common timing criteria, or media gateways and media nodes can cooperate to set up this common timing criterion. With regard to setting the timing parameters, the method may be partitioned between two media devices (a transmitting-side media device and a receiving-side media device). In various embodiments, the common timing reference is an estimate on the receiving-side media device of the clock value at the transmitting-side media device.

8 illustrates an exemplary method 800 as far as the transmitting media device synchronizes playback with the receiving media device. The exemplary method 800 may be performed by any media device that functions as a transmitting-side media device, such as, for example, the media source 102 of the exemplary environment 100 or the media gateway 104. The method 800 may be performed as part of step 710 of the exemplary method 700 or at any time when synchronization of timing parameters between media devices is appropriate.

The method 800 begins at step 805 and includes steps that the sending device may determine that the sending device must begin to flood "beacon messages" for use in setting the timing parameters at the receiving device (810). For example, the transmitting device may determine that the receiving device is powered on, that the receiving device is attached to the virtual media network, or that the transmitting device should begin transmitting media to the receiving device. At step 815, the transmitting device generates a new beacon message. The beacon message may be any type of packet or other data message to be recognized by the receiving device. For example, a beacon message may be formed according to a proprietary protocol implemented by both the transmitting device and the receiving device. In various embodiments, such as embodiments in which the beacon message may pass through one or more intermediate devices, such as a router or switch on the way to the destination device, the sending device may send a flag indicating that the beacon message constitutes high- Configure a beacon message containing the other indication. The various mechanisms for prioritizing beacon messages within a network will vary based on the individual prioritization schemes used by various possible networking technologies. This prioritization of beacon messages can help to optimize routing times over mixed topology networks and thereby reduce variable network delay factors.

Then, at step 820, the transmitting device timestamps the beacon message with the time currently represented by the clock of the transmitting device. Such a time stamp may be referred to as a "transmitter time stamp ". Then, at step 825, the transmitting device sends a beacon message to one or more receiving devices. The elapsed time between steps 820 and 825 forms part of the fixed delay component of the clock offset values to be calculated by the receiving device. Accordingly, various implementations of the method 800 attempt to reduce or minimize the number of operations occurring between steps 820 and 825. [ In various embodiments, the transmitting device is responsible for setting timing parameters for a number of different media devices. For example, the media gateway 104 may send beacon messages to a number of other media nodes 104, 106. In some such embodiments, the sending device may send a beacon message, e. G., By individually addressing each copy of the beacon message to each media device, or by multicasting the message to a plurality of media devices, To a plurality of media devices. Alternatively or additionally, the sending device executes the method 800 multiple times to accommodate a number of different media devices.

At step 830, the sending device determines whether the sending device has completed flooding the beacon messages to the receiving device. For example, the sending device may continue to flood the beacon messages until a predetermined number of beacon messages have been transmitted. In various embodiments, the transmitting device is based on feedback from the receiving device as a further alternative, or in addition, to the conditions of step 830. [ For example, the receiving device may send a message indicating that sufficient synchronization has not been achieved despite the transmission of the message when a sufficient synchronization has been achieved or a predetermined number of beacon messages. As another method of determining if flooding should be interrupted, the sending device monitors network performance during the flooding period and continues flooding until the network performance meets a certain minimum allowed threshold. For example, in addition to beacon messages, the transmitting device may send roundtrip diagnostic messages to the receiving device. As another alternative, the receiving device may be further configured to send beacon messages back to the transmitting device for network diagnosis. Upon receiving the message again from the receiving device, the sending device generates one or more measures of the network performance metric. For example, the sending device may generate scales of network delay or jitter over the previous flooding window, and if the scales are lower than some minimum allowed network capability, a predetermined number of beacon messages are sent , Beacon messages can continue to flood. Based on the teachings herein, it will be clear that various combinations of these and other methods for determining the sufficiency of beacon message flooding may be utilized.

If the sending device determines in step 830 that beacon message flooding should continue, the method 800 loops back to step 815 to send additional beacon messages. In various embodiments, where the flooding window includes transmission of a predetermined number of beacon messages, the transmitting device determines that the previous window is insufficient based on network capabilities or other factors, and then proceeds to loop back to step 815 It is possible to reset the beacon message counter beforehand, thereby initiating transmission of another beacon message set in the new window. On the other hand, if the sending device determines in step 830 that the flooding should be terminated, the method 800 terminates in step 835. After that, the transmitting-side device starts to transmit the media to the newly synchronized receiving-side device.

It should be noted that, in various embodiments, in embodiments in which the transmitting device can not receive any return messages from the receiving device based on beacon messages, or the receiving device receives return messages, The device does not use any return messages to set the timing parameters. In some embodiments, return messages are used only to determine sufficiency of the flooding period. Hence, and unlike other clock synchronization methods, the methods described herein can be said to be a "one-way" synchronization method in which the majority of the synchronization calculations are performed by the receiving device rather than the transmitting device.

Figure 9 illustrates an exemplary method 900 as long as the receiving media device synchronizes playback with the receiving media device. The exemplary method 900 may be performed by any media device that functions as a receiving media device, such as, for example, the media gateway 104 of the exemplary environment 100 or other media nodes 104 and 106 . The method 900 may be performed as part of step 710 of the exemplary method 700 or at any time when synchronization of timing parameters between media devices is appropriate.

The method 900 begins at step 905 and sets the minimum offset variable "MinO" for use by the receiving device to maintain a current running minimum offset value when new messages are received or processed The flow advances to step 910 to initialize. Then, at step 915, the receiving device receives a beacon message from the transmitting device. Then, at step 920, the receiving device generates a timestamp based on the time currently represented by the clock of the receiving device. This timestamp can be called "receiver timestamp" " R (x) ". The elapsed time between steps 915 and 920 forms part of the fixed delay component of the clock offset values to be calculated by the receiving device. Accordingly, various implementations of the method 900 attempt to reduce or minimize the number of operations that take place between steps 920 and 925.

At step 925, the receiving device extracts the transmitter timestamp "S (x)" from the beacon message. As discussed above, the transmitter timestamp is inserted into the beacon message by the transmitter device just prior to transmission, as in step 820 of the exemplary method 800. At step 930, the receiving device determines if the sending device is a media source of the virtual media network. For example, when the receiving-side device is operating as a gateway to the virtual media network, the receiving-side device determines that the transmitting-side device is the media source. In such a case, the method 900 proceeds to step 935. The receiving device then converts the transmitter timestamp from the time domain of the transmitting device to the time domain of the virtual media network. This conversion may involve adding or subtracting previously negotiated offsets between the two devices. Such negotiation and conversion between time zones may be performed according to any method known to those of ordinary skill in the art. In some alternative embodiments, the source device and the media nodes maintain clocks in the same time domain. In some such embodiments, steps 930 and 935 do not exist.

After converting the transmitter timestamp into a virtual media network area in step 935 or after determining in step 930 that the transmitter is not a media source, the method 900 determines whether the receiving device has, for example, two The method proceeds to step 940 of calculating an offset value based on a transmitter timestamp and a receiver timestamp, such as the difference between timestamps. This current offset value "CurO" is equivalent to "any delay experienced by the beacon message between the actual offset between the transmitter clock and the receiver clock plus the generation of the two time stamps S (x) and R (x). As we have seen, this delay includes two components. The first component of the delay may be a time delay, for example, a certain delay associated with circuits and data paths through which the messages travel, along with the time between when the OS generates a timestamp associated with transmitting / It is a fixed delay associated with the time spent passing through the hardware and software components of the network. This fixed delay may already be considered as part of the rendering process. The second component of the delay is a variable network delay associated with a time varying delay. For example, shared media networks such as WiFi may wait for the media to become clear before transmission, and thus introduce different delays at different times.

A better estimate of the actual clock offset is obtained from the least delayed message since the variable delay only introduces additional delay (without eliminating the delay). Accordingly, the method 900 searches for the minimum offset value obtained during flooding as the best available estimate of the actual offset. At step 945, the receiving device compares the current offset, CurO, with the previously found minimum offset, or, if the current iteration of the loop is the first iteration to the minimum offset value initialized at step 910, with the MinO. If CurO is smaller than MinO, it is found that CurO represents a closer estimate of the actual offset between the transmitter clock and the receiver clock. At step 950, the receiver device overwrites the value of MinO with the value of CurO.

At step 955, the receiver device determines whether the transmitter device has completed flooding the beacon messages. For example, when waiting for an additional beacon message, the receiver device may determine whether a timeout has occurred, or that the transmitter device has determined to start transmitting media messages, or that a predetermined number of beacon messages have been received , Or may determine that the sending device has sent a special message indicating the end of flooding. In various embodiments, the receiver device determines if flooding was sufficient to establish the desired accuracy of the offset. For example, the receiver device may track intervals over which beacon messages are received and determine whether the network is sufficiently stable to generate an offset value of the desired accuracy, based on a comparison of the measured intervals with a known time interval You can decide. If the network is not sufficiently stable, the receiving device sends a message to the transmitting device indicating that additional flooding should be performed. Various modifications will be apparent. Based on the teachings herein, it will be clear that various combinations of these and other methods for determining the sufficiency of beacon message flooding may be utilized.

If the receiving device determines that additional flooding is being performed or is to be performed, the method 900 loops back to step 915 from step 955 to process additional beacon messages. Otherwise, the method 900 proceeds to step 960 where the receiving device resets the local clock based on the determined minimum offset. For example, the receiving device may subtract MinO from the current clock value to set the local clock to a new value estimated to be closer to the actual clock value of the transmitting device. In some embodiments in which a fixed delay of the network is known or estimated, the receiving device subtracts MinO from the current clock value and attempts to subtract the actual clock offset value of the calculated offset value from the fixed delay value Lt; / RTI > In some embodiments, the receiving device does not change the local clock at all, but instead can maintain the minimum offset value MinO for use in comparing the local clock with the time stamps received from the transmitter device. For example, the receiving device may add MinO to the timestamp before any such comparison. Various alternative modifications will be apparent. The method 900 then ends at step 965.

In various alternative embodiments, the receiving device utilizes a previously set lower limit offset to help ensure that a tremendously large offset computed during the flooding period is not used to reset the clock. For example, if the flooding period is surrounded by a period of variable network delay, the calculated offset may be much larger than the actual value of the offset between the transmitter clock and the receiver clock. In some such embodiments, the receiver first compares the calculated minimum offset in steps 940-950 with a previously set lower offset, and determines whether the minimum offset is above the lower limit offset. In such a case, the receiver does not update the clock based on the minimum offset, but continues to use the previously set lower limit. Otherwise, the receiver updates the clock as described in detail in step 960 because the minimum offset value is less than the lower limit and is therefore a better estimate than the lower limit. An exemplary method for determining the lower limit will be described in more detail below with respect to FIG.

In various embodiments, the receiving device periodically performs the method 900 to reestablish synchronization. In some such embodiments, the receiving device may reset the clock to its original value, delete the stored offset value, or otherwise make any changes made based on previous executions of the method 900, roll back "to" restart " the task of determining the clock offset. By periodically resetting the clock offset, the receiving device may better consider the clock drift between the transmitting device clock and the receiving device clock.

Based on the teachings herein, while the method 900 is described as a real-time method for processing each beacon message as it is received, various alternative embodiments utilize methods for collectively processing beacon messages . For example, in some such embodiments, the receiving device receives a plurality of beacon messages, timestamps the messages upon receipt, and provides a minimum offset < RTI ID = 0.0 > And subsequently processes the received messages sequentially.

While the above approaches are attempting to generate the best estimate of the clock offset between the two devices, it is well known that network conditions can be temporarily improved after this initial flooding period and a better estimate can be obtained later will be. Accordingly, several methods can be used to attempt to better estimate clock offsets after initial timing parameter settings. These methods can also address the possibility of clock drift where the modification, temperature, or other differences in parameters can cause the transmitting device clock and the receiving device clock to operate at slightly different rates.

Figure 10 illustrates an exemplary method 1000 as long as the receiving media device achieves better playback synchronization during media streaming. The exemplary method 1000 may be performed by any media device that functions as a receiving media device, such as, for example, the media gateway 104 of the exemplary environment 100 or other media nodes 104, . Method 1000 may be performed as part of step 780 of exemplary method 700 or at any time when synchronization of timing parameters between media devices is appropriate.

The method 1000 begins at step 1005 and proceeds to step 1010, where the receiving device receives a media data packet from the transmitting device. Then, at step 1015, the receiving device generates a timestamp based on the time currently represented by the clock R (x) of the receiving device. At step 1020, the receiving device extracts the transmitter timestamp "S (x)" from the media data message. The transmitter timestamp may have been inserted into the media data message by the transmitter device immediately prior to transmission. In step 1025, the receiving device determines whether the sending device is a media source of the virtual media network. For example, if the receiving device is operating as a gateway to the virtual media network, the receiving device may determine that the transmitting device is a media source. In this case, the method 1000 proceeds to step 1030. [ The receiving device then converts the transmitter timestamp from the time domain of the transmitting device to the time domain of the virtual media network. This conversion may involve adding or subtracting previously negotiated offsets between the two devices. Such negotiation and conversion between time zones may be performed according to any methods known to those of ordinary skill in the art. In some alternative embodiments, the source device and the media nodes maintain clocks in the same time domain. In some such embodiments, steps 1020 and 1030 exist.

After converting the transmitter timestamp into a virtual media network area in step 1030 or after determining in step 1025 that the transmitter is not a media source, the method 1000 may be repeated until the receiving device receives, for example, The method proceeds to step 1035 of calculating an offset value based on a transmitter timestamp and a receiver timestamp, such as a difference between timestamps. If the transmitter timestamp has been transformed, the transformed timestamp is used to compute the offset. This offset value " O "is the difference between the generation of two timestamps S (x) and R (x), including both the actual offset between the transmitter clock and the receiver clock plus both fixed and variable delays. &Quot; any delay experienced ". At step 1040, the receiving device determines if the offset value represents a better estimate of the offset between clocks than previously used. For example, in various embodiments where the previously determined minimum offset is used to reset the clock of the receiving device, the receiving device determines if the current offset O is less than zero (0). A positive result for this comparison indicates that the previously used minimum offset may have included some variable network delay and that its subtraction from the local clock has "overshooted" the ideal set point, Is set to be slower than the clock of the transmitter. The current offset O may indicate this overshoot as being negative by including a smaller (or zero) variable delay than the previously used minimum. In this case, the current offset O will be determined to represent the new best estimate of the actual clock offset, and at step 1045, the local clock may be used to reset again, thereby at least partially correcting the previous overshoot . Various modifications to other embodiments will be apparent. For example, in embodiments in which the previously determined minimum offset is not used to modify the local clock and instead is retained for use in timestamp comparisons, step 1040 determines whether the current offset O is greater than the previous minimum offset < RTI ID = 0.0 > MinO, and in such a case, the receiving device sets MinO equal to O in step 1045. [ Various other modifications will be apparent.

In various alternative embodiments, the receiving device utilizes a previously set lower limit offset to help ensure that a tremendously large offset computed during the flooding period is not used to reset the clock. In some such embodiments, the receiver first compares the offset calculated in step 1035 with a previously set lower limit offset to determine if the offset represents a better estimate of the actual offset than the lower limit offset. In such a case, the receiver does not update the clock based on the minimum offset, but continues to use the previously set lower limit. Otherwise, the receiver updates the clock as described in detail in step 1045, since the offset value is a better estimate than the lower limit. An exemplary method for determining the lower limit will be described in more detail below with respect to FIG.

At step 1050, the receiving device continues to process the received media packet, for example, to render the media output at an appropriate time. For example, the receiving device may extract from the media data packet a serving time that is separate from the transmitter timestamp and the receiver timestamp. This presentation time represents the time that the media data contained in the message must be rendered. After extracting the presentation time, the receiving device renders the media data when it coincides with the presentation time. For example, the receiving device may buffer the media data for playback by a local playback device, or may forward the message to another media node for playback. It will be appreciated that the current time to "match" the serving time may include an equality between the current time and the serving timestamp, but may also include other forms of matching. For example, in various embodiments, the current time coincides when the "current time-sustained minimum offset value" matches the provided timestamp. In addition, or as a further alternative, a comparison of whether to match adds, subtracts, or otherwise considers the fixed delay value. Various other ways of determining the appropriate time for playback based on the local clock, the provisioned timestamp, and possibly other possible values will be apparent. In addition, it will be appreciated that the concept that the current time coincides with the provision time based on the minimum offset value includes comparisons using a local clock that has previously been modified by the minimum offset value, but does not explicitly consider the minimum offset value. Various embodiments perform this comparison just before the output so that the data is output at the appropriate time. Other embodiments use this comparison to insert media data into the playback buffer at a location where the media is likely to be played back at presentation time. Such insertion may involve insertion of "dummy" data prior to insertion of the media data to adjust the timing of playback. Various additional methods of controlling the playback timing of data in the buffer will be apparent.

FIG. 11 illustrates an exemplary method 1100 for determining a lower bound offset. As discussed above, various alternative embodiments additionally set a lower limit offset prior to analysis of beacon flooding and media packets to determine better clock offset estimates. The exemplary method 1100 may be performed by any media device that functions as a receiving media device, such as, for example, the media gateway 104 of the exemplary environment 100 or other media nodes 104 and 106 . The method 1100 may be performed as part of step 710 of the exemplary method 700 or at any time when synchronization of timing parameters between media devices is appropriate.

The method 1100 begins at step 1105 and the receiving device collects timestamps from the reciprocal movement between the receiving device and the transmitting device to calculate the lower bound by receiving the handshake message from the transmitting device (1110). In various embodiments, the handshake message is transmitted over a different channel than the channel through which beacon messages or media data packets are transmitted. For example, handshake messages may be transmitted over a Bluetooth channel, while beacon messages and media data packet messages may be transmitted over a WiFi channel. As part of the handshake protocol used, the transmitter includes a time stamp t1 according to the transmitter clock in the handshake message, which indicates the time at which the transmitter transmitted the handshake message. Then, at step 1115, near the reception of the handshake message at step 1110, the receiver records the reception time stamp t2 according to the receiver clock.

The receiver device then prepares, in step 1120, to transmit the handshake message back to the transmitter device by generating a time stamp t3 according to the receiver clock, which indicates the time at which the receiver retransmits the handshake message to the transmitter do. In some embodiments, the receiver may insert the time stamp t3 into the handshake message received from the transmitter or into the newly generated handshake message. Then, at step 1125, near the occurrence of the time stamp t4, the receiver sends a handshake message to the transmitter. Then, at step 1130, the receiver may receive the handshake message back from the transmitter. As part of the transmitter processing the handshake message, the handshake message now includes a time stamp t4 according to the transmitter clock, indicating the time at which the transmitter received the handshake message.

As will be described below, the four time stamps tl through t4 can be used to calculate the lower limit offset. However, in some embodiments, the receiver device first calculates the network transit time experienced by the handshake message to determine if the network delay during the handshake process is low enough to provide an accurate or otherwise acceptable lower bound The time stamps t1 to t4 are used. Accordingly, the receiver device calculates the transit time using the following equation at step 1135: Transit time = ((t2-t1) + (t4-t3)) / 2. The receiver then determines, for example, whether the calculated transit time is acceptable by determining that the transit time is below a predetermined threshold. If the calculated transit time is not acceptable, the receiver device instructs the transmitter to retry the handshake process at step 1145 and loops back to step 1110 to retry the process. On the other hand, if the network transit time is acceptable, the receiver computes the lower limit offset using the time stamps in step 1150, using the following equation: Lower limit offset = ((t2-t1) - (t4- t3)) / 2. The method then ends at step 1155.

In some embodiments, it will be apparent that the transmitter clock may be slower than the receiver clock so that the time stamps generated by the receiver device are smaller than the time stamps generated by the transmitter device. It will be appreciated that the lower clock offset equation may indicate the direction of clock adjustment based on the sign of the calculated value. In some embodiments, the receiver device may use absolute values when comparing various calculated offsets to determine which offset is a better estimate, so that only the magnitude, not the relative adjustment direction, is compared.

In various embodiments, the receiving device periodically performs the method 1100 to reestablish synchronization. In some such embodiments, the receiving device may reset the clock to its original value, delete the stored lower limit offset value, or otherwise make any changes made based on previous executions of the method 1100 to " Rollback "to" resume " the task of determining the clock offset. By periodically resetting the lower-limit offset, the receiving-side device may better consider the clock drift between the transmitting-side device clock and the receiving-side device clock.

Based on the above, various embodiments enable synchronization of media playback between media devices belonging to networks that exhibit variable delays. For example, by implementing a unidirectional synchronization method in which a receiving device identifies a minimum clock offset from a plurality of messages, the effects of variable delay on clock synchronization can be reduced. In addition, by continually searching for better synchronization parameters after the media stream is started, the receiving device can improve synchronization while taking into account clock drift. Based on the above, various additional advantages will be apparent.

It will be clear from the foregoing description that the various illustrative embodiments of the invention may be implemented in hardware. In addition, various illustrative embodiments may be implemented as instructions stored on a machine-readable storage medium, which may be read and executed by at least one processor to perform the operations described in detail herein. The machine-readable storage medium may include any mechanism for storing information in a machine-readable form, such as a personal or laptop computer, a server, or other computing device. As such, tangible and non-transitory machine-readable storage media can be read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices , And similar storage media. Moreover, as used herein, the term "processor" is intended to encompass a microprocessor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC) Lt; RTI ID = 0.0 > device. ≪ / RTI >

It will be appreciated by those of ordinary skill in the art that any of the block diagrams herein represent conceptual aspects of exemplary circuits that implement the principles of the present invention. Likewise, any flow charts, flowcharts, state transitions, pseudo code, etc., may be substantially represented on a machine-readable medium and may thus be accessed by a computer or processor (whether such computer or processor is explicitly shown or otherwise Whether or not the process is being executed).

While the various illustrative embodiments have been described in detail with particular reference to certain specific exemplary aspects thereof, it will be appreciated that the invention may be practiced in other ways, and its details may be modified in various obvious respects. Variations and modifications may be practiced within the spirit and scope of the invention, as will be immediately apparent to one of ordinary skill in the art. Accordingly, the disclosure, description, and figures are for illustrative purposes only, and are in no way limiting the invention, which is limited only by the claims.

102a, 102b: media source 104a: living room gateway
104b: bedroom gateway 106a: office renderer
106b: Kitchen Renderer 106c: Guest Renderer
108: stereo receiver 110: speakers
310: media source 320: media node
322: Sink 324: Broadcaster
326: Renderer 330: Media Node
332: Sink 334: Broadcaster
336: Renderer 340: Media Node
346: Renderer 350: Access Point
360: Media 410: Network Interface
420: Virtual network media application
430: media source application 440: operating system
445: Network Media Driver 450: Media Library
510: Network Interface 520: Renderer
530: Speakers 540: Display
550: Broadcaster 560: Auxiliary output
570: command module 580: synchronization module
610: Processor 620: Memory
630: User interface 640: Network interface
650: Storage 660: System bus
670: Operating System 672: Synchronization Instructions
674: Media source application 676: Media library commands
678: Virtual network media application commands
680: Command module instructions 682: Broadcaster instructions
684: Renderer commands

Claims (15)

A method of synchronizing media playback between a transmitting-side media device and a receiving-side media device,
Receiving, at the receiving-side media device, from the transmitting-side media device a plurality of messages, the plurality of messages including a plurality of transmitter timestamps;
Generating a plurality of clock offset values based on the plurality of transmitter time stamps and the clock of the receiving media device;
Identifying a minimum clock offset value from the plurality of clock offset values;
Determining a first presentation time associated with the first media data for playback and the first media data; And
Rendering the first media data at a first time coinciding with the first presentation time based on the minimum clock offset
Wherein the media playback is synchronized between the transmitting and receiving media devices. 2. The method of claim 1, wherein identifying a minimum clock offset value comprises:
Generating a first clock offset value of the plurality of clock offset values;
Setting the minimum clock offset value equal to the first offset clock value;
Generating a second clock offset value of the plurality of clock offset values after setting the minimum clock offset value to be equal to the first offset clock value;
Determining that the second clock offset value is less than the minimum clock offset value; And
Setting the minimum clock offset value equal to the second offset clock value based on a determination that the second clock offset value is less than the minimum clock offset value
Wherein the media playback is synchronized between the transmitting and receiving media devices. 2. The method of claim 1, further comprising: modifying the value of the clock by subtracting the minimum offset value from the value of the clock of the receiving media device,
And wherein the first time coinciding with the first presentation time based on the minimum clock offset value comprises a value of the clock coinciding with the first presentation time. How to synchronize media playback. 2. The method of claim 1, wherein the plurality of messages comprises a first plurality of messages and a second plurality of messages,
Generating a first measure of intervals at which the first plurality of messages arrive;
Determining that the first measure of intervals at which the first plurality of messages arrive indicates that the network is unstable;
Instructing the transmitting-side media device to transmit additional messages;
Generating a second measure of intervals at which the second plurality of messages arrive after instructing the transmitting media device to transmit additional messages;
Determining that the second measure of intervals at which the second plurality of messages arrive indicates that the network is stable
Further included,
Wherein identifying a minimum clock offset value among the plurality of clock offset values comprises using at least one clock offset value of the plurality of clock offset values associated with the second plurality of messages. A method for synchronizing media playback between a device and a receiving media device. 2. The method of claim 1, wherein the plurality of messages comprises a first plurality of messages and a second plurality of messages,
The sending media device sending the first plurality of messages;
The sending media device generating a first network performance metric associated with the transmission of the first plurality of messages;
Determining that the first network performance metric indicates that the network is unstable;
The transmitting media device transmitting the second plurality of messages based on the determination that the first network performance metric indicates that the network is unstable;
Generating a second network performance metric associated with the transmission of the second plurality of messages by the sending media device; And
Determining that the second network performance metric indicates that the network is stable
Wherein the media device is further configured to synchronize media playback between the transmitting and receiving media devices. A receiving-side media device for synchronizing media playback with a transmitting-side media device,
A memory configured to store media data for playback;
A network interface configured to communicate with the sending media device; And
Processor
Including,
The processor comprising:
Receive, via the network interface, a plurality of messages, the plurality of messages including a plurality of transmitter timestamps from the transmitting media device;
Generate a plurality of clock offset values based on the plurality of transmitter time stamps and the clock of the receiving media device;
Identify a minimum clock offset value from the plurality of clock offset values;
Find first media data for playback and a first presentation time associated with the first media data;
And render the first media data at a first time that coincides with the first presentation time based on the minimum clock offset value. 7. The method of claim 6, wherein, when identifying a minimum clock offset value,
Generate a first clock offset value of the plurality of clock offset values;
Setting the minimum clock offset value equal to the first offset clock value;
Generate a second clock offset value of the plurality of clock offset values after setting the minimum clock offset value to be equal to the first offset clock value;
Determine that the second clock offset value is less than the minimum clock offset value;
And to set the minimum clock offset value equal to the second offset clock value based on a determination that the second clock offset value is less than the minimum clock offset value. 7. The apparatus of claim 6,
And to modify the value of the clock by subtracting the minimum offset value from the value of the clock of the receiving media device,
Wherein the first time coinciding with the first presentation time based on the minimum clock offset value is a value of the clock coinciding with the first presentation time. 7. The apparatus of claim 6, wherein the plurality of messages comprises a first plurality of messages and a second plurality of messages,
Generate a first measure of intervals at which the first plurality of messages arrive;
Determine that the first measure of intervals at which the first plurality of messages arrive indicates that the network is unstable;
Direct the sending media device to send additional messages;
Generate a second measure of intervals at which the second plurality of messages arrive after instructing the transmitting media device to send additional messages;
Wherein the second measure of intervals at which the second plurality of messages arrive determines that the network is stable,
Wherein the processor is configured to use a clock offset value of at least one of the plurality of clock offset values associated with the second plurality of messages when identifying a minimum clock offset value from the plurality of clock offset values, Side media device. In the system,
The receiving media device of claim 6, wherein the plurality of messages comprises a first plurality of messages and a second plurality of messages; And
Side media device
Including,
Wherein the transmitting-
A transmitting-side media device network interface configured to communicate with the receiving-side media device; And
The sender media device processor
Including,
Wherein the transmitting-side media device processor comprises:
Transmitting the first plurality of messages,
Generate a first network performance metric associated with the transmission of the first plurality of messages,
The first network performance metric determining that the network is unstable,
Sending the second plurality of messages based on a determination that the first network performance metric indicates that the network is unstable,
Generating a second network performance metric associated with the transmission of the second plurality of messages,
And the second network performance metric is determined to indicate that the network is stable. A non-transitory machine-readable storage medium encoded with instructions for a receiving-side media device to execute to synchronize media playback between a transmitting-side media device and a receiving-side media device,
Instructions, at the receiving media device, for receiving a plurality of messages from the transmitting media device, the plurality of messages including a plurality of transmitter timestamps;
Instructions for generating a plurality of clock offset values based on the plurality of transmitter time stamps and the clock of the receiving media device;
Instructions for identifying a minimum clock offset value from the plurality of clock offset values;
Instructions for finding first media data for playback and a first presentation time associated with the first media data; And
And rendering the first media data at a first time that coincides with the first presentation time based on the minimum clock offset value. 12. The method of claim 11, wherein the instructions for identifying a minimum clock offset value comprise:
Instructions for generating a first clock offset value of the plurality of clock offset values;
Instructions for setting the minimum clock offset value equal to the first offset clock value;
Instructions for generating a second one of the plurality of clock offset values after setting the minimum clock offset value equal to the first offset clock value;
Instructions for determining that the second clock offset value is less than the minimum clock offset value; And
Instructions for setting the minimum clock offset value equal to the second offset clock value based on a determination that the second clock offset value is less than the minimum clock offset value. media. 12. The apparatus of claim 11, further comprising instructions for modifying a value of the clock by subtracting the minimum offset value from a value of a clock of the receiving media device,
Wherein the first time coinciding with the first presentation time based on the minimum clock offset value is a value of the clock coinciding with the first presentation time. 12. The method of claim 11, wherein the plurality of messages comprises a first plurality of messages and a second plurality of messages,
Instructions for generating a first measure of intervals at which the first plurality of messages arrive;
Instructions for determining that the first measure of intervals at which the first plurality of messages arrive indicates that the network is unstable;
Instructions for instructing the transmitting media device to send additional messages;
Instructions for generating a second measure of intervals at which the second plurality of messages arrive after instructing the transmitting media device to transmit additional messages; And
Instructions for determining that the second measure of intervals at which the second plurality of messages arrive indicates that the network is stable,
Wherein the instructions for identifying a minimum clock offset value among the plurality of clock offset values comprise instructions for utilizing a clock offset value of at least one of the plurality of clock offset values associated with the second plurality of messages , Non-transitory machine-readable storage medium. For a non-transitory machine-readable medium set,
12. The non-transitory machine-readable storage medium of claim 11, wherein the plurality of messages comprises a first plurality of messages and a second plurality of messages; And
Additional non-volatile machine readable storage medium encoded with instructions for executing the transmitting media device
Including,
The additional non-volatile machine-readable storage medium comprising:
Instructions for transmitting the first plurality of messages;
Instructions for generating a first network performance metric associated with the transmission of the first plurality of messages;
Instructions for determining that the first network performance metric indicates that the network is unstable;
Instructions for sending the second plurality of messages based on the determination that the first network performance metric indicates that the network is unstable;
Instructions for generating a second network performance metric associated with the transmission of the second plurality of messages; And
And the second network performance metric indicates instructions to determine that the network is stable
≪ / RTI >

Images