JP2015536064A - Playback synchronization - Google Patents

Abstract

Various exemplary embodiments include, at a receiving media device, receiving a plurality of messages including a plurality of transmitting time stamps from a transmitting media device; a plurality of transmitting time stamps and a receiving media device clock; Generating a plurality of clock offset values, identifying a minimum clock offset value from the plurality of clock offset values, first media data for reproduction, and associated with the first media data Receiving a first presentation time; and causing the first media data to be rendered at a first time that matches the first presentation time based on a minimum clock offset. Method and medium for synchronizing media playback between a media device and a sending media device On the A device. [Selection] Figure 1

Description

[Cross-reference with related applications]
This application claims the benefit of US Provisional Patent Application No. 61 / 701,326, filed September 14, 2012, entitled “Playback Synchronization” granted to Lee et al. The disclosure of this patent application is hereby incorporated by reference for all purposes as if fully set forth herein.

  This application claims the benefit of US Provisional Patent Application No. 61 / 405,835, filed Oct. 22, 2010, entitled “Media Distributed Architecture” granted to Lau et al., October 2011. Which is a continuation-in-part of U.S. Patent Application Publication No. 13 / 278,799 filed 21 days and entitled "Media Distributed Architecture" granted to Lau et al. Are hereby incorporated by reference for all purposes as if fully set forth herein.

  Various exemplary embodiments disclosed herein generally relate to media streaming and network media playback.

  As electronic devices such as smartphones and tablets have become popular, the frequency with which people play content such as music and video using such devices is increasing. Often, these media sources may not provide the media to the user's satisfaction. For example, the display may be too small, or the speaker volume may be insufficient in terms of quality or volume. Also, the output from the media source may not be easily or comfortably enjoyed by multiple people. Furthermore, the user cannot enjoy the media everywhere in the home unless he / she carries the media source.

  The following provides an overview of various exemplary embodiments. In the following summary, some simplifications and omissions may be made, which are intended to highlight and highlight some aspects of various exemplary embodiments. It is not intended to limit the scope of the invention. The following sections provide detailed descriptions of preferred exemplary embodiments suitable for enabling those of ordinary skill in the art to make and use the inventive concepts.

  The various embodiments described herein relate to a method of synchronizing media playback between a sending media device and a receiving media device, the method comprising: Receiving a plurality of messages including a plurality of sender timestamps, generating a plurality of clock offset values based on the plurality of sender timestamps and the receiving media device clock, and the plurality of clock offset values Identifying a minimum clock offset value; determining first media data for reproduction; and a first presentation time associated with the first media data; and a first presentation based on the minimum clock offset. The first media data is rendered at a first time that matches the time. And a step.

  Various embodiments described herein relate to a receiving media device for synchronizing media playback with a transmitting media device, the receiving media device configured to store media data for playback. And a network interface configured to communicate with the sending media device and a processor, the processor including a plurality of sender timestamps from the sending media device via the network interface. And generating a plurality of clock offset values based on the plurality of sender time stamps and the receiving media device clock, identifying a minimum clock offset value from the plurality of clock offset values, and 1 media data and associated with this first media data It defines a first presentation time, the first time that matches the first presentation time based on the minimum clock offset value, configured such that the first media data is to be rendered.

  Various embodiments described herein are non-transitory machines encoded with instructions for synchronizing media playback between a sending media device and a receiving media device, executed by a receiving media device. In connection with a readable storage medium, the medium includes, at a receiving media device, instructions for receiving a plurality of messages including a plurality of sender timestamps from the transmitting media device, a plurality of sender timestamps and a receiving media An instruction for generating a plurality of clock offset values based on a clock of the apparatus, an instruction for identifying a minimum clock offset value from the plurality of clock offset values, first media data for reproduction, and Based on a command for determining a first presentation time associated with the first media data and a minimum clock offset value. First time that matches the first presentation time Te to, and instructions for the first media data is to be rendered.

  Various embodiments include obtaining a plurality of time stamps from a round trip between a receiving media device and a sending media device, setting a lower limit offset value based on the plurality of time stamps, and a minimum clock. After identifying the offset value, further comprising determining that this minimum clock offset value represents a better estimate of the actual clock offset between the transmitting device clock and the receiving device clock than the lower limit offset value. Including causing the first media data to be rendered at a first time that coincides with the first presentation time based on the minimum clock offset, wherein the minimum clock offset value is the transmitter device clock and the receiver Better than the lower limit offset value of the actual clock offset between device clocks It is performed based on the determination that represents an estimated.

  Various embodiments are described herein, wherein identifying a minimum clock offset value includes generating a first clock offset value of the plurality of clock offset values, and determining a minimum clock offset value as a first. Setting a clock offset value equal to the first clock offset value, setting a minimum clock offset value equal to the first clock offset value, and generating a second clock offset value of the plurality of clock offset values; And determining that the second clock offset value is smaller than the minimum clock offset value, and determining that the second clock offset value is smaller than the minimum clock offset value. And setting the same.

  Various embodiments further include modifying the clock value by subtracting the minimum offset value from the clock value of the receiving media device to match the first presentation time based on the minimum clock offset value. The first time includes a clock value that matches the first presentation time.

  Various embodiments receive a message from a sending media device at a receiving media device that includes second media data, a second presentation time, and a sender type stamp, and a clock offset based on the sender timestamp. Generating a value; determining that this clock offset value is a more accurate representation of the minimum clock offset value of the actual offset between the receiving media device clock and the transmitting media device clock; Adjusting a minimum clock offset value based on the clock offset value; determining a third media data for reproduction; a third presentation time associated with the third media data; and a minimum clock offset. After adjusting the value, this minimum clock offset value The second time that matches the third presentation time based, further comprising a third media data is to be rendered.

  Various embodiments are described herein and the clock offset value is more accurate than the minimum clock offset value of the actual offset between the receiving media device clock and the transmitting media device clock. The determining step includes determining that the clock offset value is less than zero, and adjusting the minimum clock offset value based on the clock offset value is received by subtracting the offset value from the clock value. Modifying the clock value of the side media device.

  Various embodiments further comprise converting at least one sender time stamp of the plurality of type stamps from the time domain of the sending media device to the time domain of the receiving media device, wherein the plurality of sender times The step of generating a plurality of clock offset values based on the stamp and the clock of the receiving media device includes at least one sender time stamp of the plurality of time stamps from the time domain of the transmitting media device. Generating at least one clock offset value based on the at least one sender timestamp after conversion to the time domain of the device.

  Various embodiments are described, wherein the plurality of messages includes a first plurality of messages and a second plurality of messages, and the method includes a first measurement of an interval at which the first plurality of messages arrive. Generating, determining that the first measurement for the interval at which the first plurality of messages arrive indicates that the network is unstable, and transmitting an additional message to the transmitting media device Instructing the sending media device to send an additional message, generating a second measurement for the interval at which the second plurality of messages arrive, and Determining that the second measurement for the interval of arrival of the plurality of messages indicates that the network is stable. See, identifying a minimum clock offset value from a plurality of clock offset value includes a plurality of at least one step of using the clock offset value of the clock offset values associated with the second plurality of messages.

  Various embodiments are described, wherein the plurality of messages includes a first plurality of messages and a second plurality of messages, the method comprising: transmitting a first plurality of messages by a sending media device; Generating a first measurement value for network performance associated with transmission of the first plurality of messages by the side media device, and the first measurement value for network performance indicates that the network is unstable. Transmitting a second plurality of messages by the transmitting media device based on determining that the network performance is stable and determining that the first measurement of network performance indicates that the network is unstable. And a network associated with transmission of the second plurality of messages by the sending media device Further comprising generating a second measure for ability, a second measurement of the network performance, and the step of the network is determined to indicate that it is stable.

  For a better understanding of various exemplary embodiments, reference is made to the following accompanying drawings.

FIG. 3 illustrates an exemplary environment for media playback. FIG. 2 illustrates an exemplary method for forming and operating a virtual media network. FIG. 1 illustrates an example virtual media network. FIG. 4 is an exemplary component diagram of a media source. FIG. 3 is an exemplary component diagram of a media node. FIG. 3 is an exemplary hardware diagram of a media device. FIG. 3 illustrates an exemplary method for broadcasting a media signal. FIG. 6 illustrates an exemplary method for a sending media device to synchronize playback with a receiving media device. FIG. 6 illustrates an exemplary method for a receiving media device to synchronize playback with a sending media device. FIG. 6 illustrates an exemplary method for a receiving media device to obtain better playback synchronization during media streaming. FIG. 6 illustrates an exemplary method for determining a lower limit offset.

  In the description and drawings presented herein, various principles are illustrated. Those skilled in the art will appreciate that although not explicitly described or illustrated herein, various configurations within the scope of the present disclosure can be devised that embody these principles. Will. As used herein, the term “or” is non-exclusive unless otherwise indicated (eg, “or else” or “in the alternative”). or (ie, and / or). In addition, the various embodiments described herein are not necessarily mutually exclusive and can be combined to yield further embodiments that incorporate the principles described herein.

  Various embodiments described herein utilize an architecture for distributing media content. Wired or wireless media transport technology is provided to allow media to be transmitted simultaneously to multiple areas while maintaining accurate timing synchronization between various media devices. The user can have a network of speakers, displays, or other rendering devices, and can individually select which rendering device is actively outputting media. Such rendering devices, and other devices described herein, can belong to a virtual media network.

  Media rendered by a virtual media network can originate from a media source. The media source can be a mobile phone, tablet, stereo, set-top box, PC or other device. The media transmission method in the virtual media network may be wired via an auxiliary cable or the like, or may be wireless such as Bluetooth or WiFi. Speakers and other rendering devices themselves can be managed within a self-forming network. Media can be introduced into the network by media sources, and the endpoint network itself can control audio / video distribution, timing and rendering. In some embodiments, the voice introduced into the network is the audio portion of the audio video signal. The video signal can be played on a media source (such as a tablet computer). The audio signal can be kept synchronized with the video signal.

  In various embodiments, a user can select any media application to function as a media source. For example, the user can select an MP3 application, an Internet radio application, or the like. The user can then select an output device such as a speaker in the living room so that the media is transmitted to the selected output device. The operating system can send audio to the selected output device. The user can call the second application to add another speaker to the virtual media network and control the volume of the speaker. In some embodiments, this second application will never modify the media. Each device in the network can handle audio / video distribution, timing and rendering. Therefore, the media source does not have to carry such processing. Further, such a configuration allows the user to select any preferred media application as the media source without having to modify the media application.

  In various embodiments, media distributed over a virtual media network can be kept in sync. In order to achieve such playback synchronization, various media devices that transmit media data can include a time stamp associated with the frame of media data to indicate when the associated media should be rendered. To enable such a mechanism, each media device can have a method that takes into account the differences between the internal clocks of these media devices. For example, two media devices may start to operate at different clock values, or these clock values will gradually move away during operation due to the clocks operating at slightly different speeds. There is also.

  To date, several network clock synchronization methods have been developed, but such methods are only reliable on wired networks where the network delay is relatively constant and thus easily resolved as part of the rendering process. There is no sex. On the other hand, in a wireless network such as WiFi or Bluetooth, the process of estimating the time when a synchronization packet is transmitted may be complicated due to a large change in network delay in a short period. Various methods described herein implement a clock synchronization process that reduces or eliminates the effects of such variable delays on the network clock synchronization process. For example, by generating multiple potential clock offset values over a period of time, the receiving device selects the minimum offset value from this group and thereby the least affected by the network delay variable component Values can be used. Various additional features for improving network clock synchronization are described in further detail below.

The following definitions are used throughout this description.
Broadcaster: Any device capable of transmitting a media stream formatted for a virtual media network, or a broadcast mechanism within such a device.
Renderer: Any device capable of rendering a media stream formatted for a virtual media network, or a rendering mechanism within such a device.
Media node: Any device that includes a renderer or broadcaster. The media nodes of some embodiments may be responsible for maintaining network state, including network time synchronization and media routing information.
Media source: Any device that sends original data to a sink. For example, mobile phones, smartphones, tablets, set-top boxes, televisions, DVD / Blu-Ray / other media players, stereo systems, video game consoles, laptops, desktop PCs, servers, or media data can be transmitted Virtually any type of hardware can be included, such as any other device.
Sink: Any device that receives outgoing media from a source or a mechanism within a device that receives media signals.
Gateway-compatible media node: Any device that combines a sink and a broadcaster. The gateway can accept the media through the sink and can rebroadcast the media into the virtual media network towards the renderer.
Virtual media network: A group of one or more nodes having at least one gateway. A virtual media network can be established by a user and can render media signals synchronized between rendering devices in the network. Note that in some embodiments, only one media node functions as the active gateway of the virtual media network.
Media device: Any device that operates in conjunction with a virtual media network, such as a media node or a media source.

  DETAILED DESCRIPTION The broad aspects of various exemplary embodiments are now disclosed with reference to the figures, wherein like numerals indicate like components or steps.

  FIG. 1 illustrates an exemplary environment 100 for media playback. In this example, there are a total of five network media nodes 104a-b, 106a-c, and various exemplary embodiments may include fewer or additional media nodes (not shown). The exemplary environment 100 is shown as being configured in the form of two virtual media networks. As shown, media source 102a functions as a media signal source for one virtual media network and media source 102b functions as a media source for the other virtual media network, although other configurations are possible. The media signal can be audio or video. In various embodiments, the media signal is the audio portion of the audio video signal. The video signal can be played on the media sources 102a, b. Note that in the exemplary embodiment, the audio signal remains synchronized with the video signal when the various signals are rendered by different devices. The video signal can also be sent to one of the devices in the virtual media network or any device other than the media source nodes 102a, b. In various embodiments, each virtual media network includes one gateway device, and in other embodiments, the virtual media network utilizes multiple gateway devices. As described above, the gateway device has a sink for receiving a media signal and a broadcaster. The gateway device may or may not have a renderer for rendering audio and / or video. In the illustrated example, the device in the living room functions as the gateway of the first virtual media network, but a different device having a broadcaster can also function as the gateway.

  In some embodiments, the system allows simultaneous media transmission to multiple areas while maintaining accurate timing synchronization. As an example, a user can configure a network of speakers, individually select speakers that are actively playing, and synchronize their playback. The method of transmitting media into the network can be wired via an auxiliary cable or the like, wireless such as Bluetooth or WiFi, or another network communication protocol. As an example, the living room gateway 104a has an auxiliary outline for providing media signals to the stereo receiver 108, so that the speaker 110 is connected to the stereo receiver 108 by one of the auxiliary lines. . Meanwhile, the living room gateway 104a can also provide media signals to the office renderer 106a and the kitchen renderer 106b via wireless transmission. The living room gateway 104a may or may not have its own renderer. In some embodiments, various media nodes belonging to the network have different channels of media streams and render them. For example, a media source renders a video signal, a first renderer renders a stereo mix audio signal for the left speaker channel, a second renderer renders a stereo mix audio signal for the right speaker channel, and the gateway A video signal and a stereo mix audio signal of both channels can be rendered. Various other channel schemes and the distribution of such channels among media devices will also be apparent.

  In some embodiments, the media nodes 104a-b, 106a-c themselves are managed in a self-forming network. The media nodes 104a-b, 106a-c themselves can control audio / video distribution, timing and rendering. Accordingly, most of the processing load is removed from the media source 102. The example of FIG. 1 relates to the home environment, but the embodiment is not limited in this way. Virtual media networks can be deployed in any environment.

  FIG. 2 illustrates an exemplary method 200 for forming and operating a virtual media network. In step 202, media devices discover each other and exchange device status information. Step 202 can be performed, for example, when the media nodes 104 and 106 are powered on. Since the media nodes 104, 106 are powered on at different times, this step can be performed continuously, repeated, or otherwise in progress. In some embodiments, the media nodes 104, 106 execute a “self-discovery” protocol that learns about each other's presence and the ability to function as a source, sink, broadcaster, or renderer, for example. The exchanged device status information includes, for example, information such as whether the device is currently active in the virtual media network, identification information of such a virtual media network, whether the device is currently functioning as a gateway, etc. You can also.

  At step 204, media source 102 is paired with gateway media node 104. One media node 104 acting as a gateway can be specifically selected by the user or can be determined automatically without user intervention. For example, the user of the smart phone 102a selects the media node 104a in the living room as the primary listening device, so that the media node 104a can be the gateway. In some embodiments, the gateway media node 104 is selected as the currently active output device for the media source node 102 based on its status. In some embodiments, the gateway media node 104 also functions as an active output device of the media source node 102 while functioning as a gateway, thus rendering media data for at least some channels. In some embodiments, the gateway media node 104 reports device information or status information to the media source 102.

  In step 206, a virtual media network is formed. Step 206 may be formed in response to the user selecting media node 104, 106. For example, the user can access a software program on the media source 102 that allows the user to select the media node 104, 106. Note that if the media nodes 104, 106 are already part of a different virtual media network, the media nodes 104, 106 may be shown as unavailable through the media source 102. In addition, or alternatively, the user may request that in-use media nodes 104, 106 be released for inclusion in the current virtual media network. In various embodiments, step 206 results in the gateway media node 104 being instructed to forward the media signal to other media nodes 104, 106 in the virtual media network.

  In step 208, media can be transferred from the media source 102 to the gateway media node 104. This step 208 may begin in response to the user selecting to present media on an output device associated with the media source. For example, the user can cause one of the applications to be executed on the smartphone 102a that plays back the media. The user can then select the gateway media node 104 as an output device so that media can be transferred to the gateway media node 104. This media transfer can also be performed at the operating system (OS) level. This transfer has the implication that the user can select any media application as the media source of the virtual media network.

  In step 210, the gateway media node 104 can broadcast the media signal to other media nodes 104, 106 in the virtual media network. For example, the living room gateway 104a may broadcast the media signal received from the smartphone 102a to the office renderer 106a, the kitchen renderer 106b, and the stereo receiver 108. In some embodiments, each media node 104, 106 plays media at its own user-controllable level (eg, volume). Accordingly, several commands can be sent from the media source 102 to the gateway media node 104. However, most of the processing can be done by the gateway. Therefore, the media source 102 does not get stuck with a heavy processing load.

  FIG. 3 illustrates an exemplary virtual media network 300. As shown, media nodes 320, 330 have sinks 322, 332 for receiving media signals and broadcasters 324, 334 for providing media signals to other media nodes 320, 330, 340. Thus, these two media nodes can function as gateways. For illustration purposes, there is an access point 350 that is separate from the media nodes 320, 330, 340. Note that one of the media nodes 320, 330, and 340 can function as an access point.

  Some of the media nodes 320, 330 include broadcasters 324, 334. In this specification, such a node can be referred to as a broadcasting node. Broadcasters 324 and 334 can be implemented by any combination of hardware or software. In various embodiments, broadcasters 324, 334 transmit media in an airtime broadcast format understood by other media nodes 320, 330, 340. Note that this format can be different from the format used to transmit media 360 from media source 310. Broadcasters 324, 334 and renderers 326, 336 can coexist in the same media node 320, 330 so that local playback can be synchronized with playback on a remote renderer. The introduction of the source can be done via a source-sink link. Unlike source-to-sink transmission, airtime broadcast can be used for point-to-multipoint media transmission using synchronized playback.

  As described above, the gateway-capable media nodes 320 and 330 have a combination of sinks 322 and 332 and broadcasters 324 and 336. In some embodiments, the gateways 320, 330 receive media from the media source 310 and rebroadcast the media in a format compatible with other media nodes 320, 330, 340 in the virtual media network. The gateways 320, 330 also include renderers 326, 336. In various embodiments, gateway media nodes 320, 330 are considered endpoints.

  There can be a plurality of gateway-compatible media nodes 320, 330 on the network. In some embodiments, the gateway media nodes 320, 330 utilize a selection method for determining the best gateway for the media source 310 to use. For example, if only one of the media nodes 320, 330 with renderers 326, 336 is active for the media source 310, this rendering node may also be the optimal gateway that saves the network bandwidth of the other source. On the other hand, if multiple renderers are active for media source 310, the gateway with the strongest or best network connection may be the optimal gateway. A selection scheme for identifying the best candidate can be performed, and if necessary, a stream handoff can be performed for different gateways 320, 330, where the original gateway 320, 330 becomes a sink for the source 310. be able to. This stream handoff can be done during the construction of the stream or during the stream. If the active gateway fails, the network can self-heal and select a new gateway to reconfigure the airtime broadcast stream.

  Some of the media nodes 320, 330, 340 include renderers 326, 336, 346. In this specification, such media nodes 320, 330, and 340 may be referred to as rendering nodes. The renderers 326, 336, 346 can be implemented by any combination of hardware or software. When using the audio example for a media signal, the renderers 326, 336, 346 decode the media stream through internally powered speakers or via analog or digital output to another amplifier / speaker device. Can be played. In the case of video, the renderers 326, 336, 346 decode and play the media stream through an internally powered display, or through an analog or digital output to another display or device having or driving the display. can do. In various embodiments, media nodes 320, 330, 340, including renderers 326, 336, 346, support the creation, maintenance, and distribution of virtual wall clocks. The renderers 326, 336, 346 can use this wall clock to accurately render the stream in the air time stream format at the specified time stamp.

  In the example of FIG. 3, a connection exists between the media source 310 and the sink 322 in the gateway media node 320. Media 360 is played by renderer 326 in gateway media node 320. The user may have selected the gateway media node 320 as an output device for the media source 310 to establish this connection. For example, media source 310 may be a mobile phone that allows a user to select a speaker to which audio should be transmitted. Any audio being played by the mobile phone can be transmitted to the selected speaker. Thus, audio can be transferred to the gateway media node 320 regardless of which application (eg, Internet radio, MP3, etc.) is providing audio. In order to perform this transfer, it may be considered that there is no need to change the application providing the audio. The connection between the media source 310 and the gateway media node 320 may be wireless or wired. In various embodiments, this connection is a wireless Bluetooth connection. However, wireless protocols other than Bluetooth can also be used.

  In addition to the connection between the media source 310 and the sink 322 of the gateway media node 320, the broadcaster 324 of the media node 320 transmits the media 360 to the renderer 336 of the media node 330 and the renderer 346 of the media node 340. used. In this example, the access point 350 functions as an intermediary device. However, the access point 350 may not be a necessary condition. In various embodiments, the media node 320 functions as an access point. Connections from the media source 310 to the media node 330 and the media node 340 can be established in a manner similar to the connection between the media source 310 and the media node 320. The user can also establish media nodes 330 and 340 as part of the virtual media network 300. Media source 310 may have a software application that allows a user to select which media nodes 320, 330, 340 should be added to the virtual network. The application can send a command to the media node 320 that instructs the media node 320 to forward the media signal to other media nodes 330, 340 that are active portions of the virtual media network. The media node 320 can handle details such as media signal reformatting, forwarding, and synchronized playback between media nodes. Thus, the media source 310 is not burdened with heavy processing.

  It will be understood that the virtual media network 300 is only one possible configuration of one possible set of devices. Various other media networks 300 can include fewer or additional devices and can distribute media in different ways. For example, media source 310 may send media directly to the access point, media node 330 may function as a gateway instead of media node 320, and media node 340 may not participate in the virtual network. Various other configurations will also be apparent.

  As described above, the media source 310 can introduce the media 360 into the virtual media network 300. An example is a PC or a smartphone. Available media introduction methods include cables that support analog or digital transmission, Bluetooth, and WiFi. In some embodiments, the media source 310 is a broadcaster that transmits media data in a format compatible with the virtual media network. In other embodiments, the broadcast capability of the media source 310 is limited by technical limitations. For example, many phone security models can prevent audio drivers from being modified by third parties. Also, the media source 310 device itself may not have available processing bandwidth or network bandwidth. Further, in some embodiments, the media source's initial link QoS level utilizes a higher QoS than other endpoints so that at least one endpoint can render to the highest degree of fidelity.

  Note that many formats and connections can be used for transmission from the media source 310 to the sink 322. Media source 310 may transmit to sink 322 via a specific protocol using wired, BT A2DP, or Wi-Fi, as some non-limiting examples. The WiFi protocol can be designed to provide a trade-off between quality and latency, or to ensure accuracy. For example, the protocol can detect a data error and request a retransmission. In many cases this may not be the purpose of the broadcast, but it is important to ensure that the media arrives before the broadcast. The embodiments disclosed herein maintain compatibility with existing devices.

  In various embodiments, this network is based on a standard Wi-Fi infrastructure. Each media node can connect to an access point 350, which requests an IP address via DHCP. Some nodes 310, 320, 330 may not have a UI (display, keyboard entry, etc.) that allows entry of wireless access keys. In such a case, connection can be performed using WPS-PBC. Another method may be an ad hoc mode in which a user connects directly to an endpoint from a GUI-enabled device and enters network parameters via a web page provided by the node or an application page that communicates directly with the node. . Another way is for an application running on the phone or other device to communicate with the media node via Bluetooth. The application can instruct the user which access point to connect to and the corresponding network access code. In some embodiments, names are provided by the user to the media nodes 320, 330, 340 during this setup phase.

  In the absence of an infrastructure such as access point 350, a node can turn itself into a virtual access point. Other nodes can discover the access point 350 and connect to it to form a virtual network. A secure connection can be made using WPS-PBC and the ad hoc method.

  FIG. 4 is an exemplary component diagram of media source 400. The media source 400 may correspond to either the media source 102a, b of the example environment 100 or the media source 310 of the example virtual media network 300. Media source 400 may include a network interface 410. In various embodiments, the network interface 410 includes a plurality of separate interfaces. For example, the network interface 410 may include a WiFi compatible interface and a Bluetooth compatible interface. In addition or alternatively, the network interface 410 may include an interface corresponding to any other protocol. In this example, a media signal (such as an audio stream or a video stream) can be transmitted using the Bluetooth compatible interface of the network interface 410. A command for controlling the virtual media network can also be transmitted using the WiFi compatible interface of the network interface 410.

  A user can access the virtual network media application 420 to control the virtual media network. As an example, the virtual network media application 420 can present a user interface that allows a user to select media nodes 104, 106 to control their volume, play, and so on. In some embodiments, there is a master volume for the network and an individual volume for each media node 104,106.

  The media source application 430 may be any application that can play audio on the media source 400. For example, the media source application 430 can be an MP3 player, Internet audio, a web browser, or the like. In various embodiments, the media is played on any output device selected by the user. This output device selection can be performed under the control of an operating system (OS) 440. For example, the OS 440 can provide a pop-up window that allows the user to select an output device. One or both of the media nodes 104, 106 are displayed as options. By selecting one of the media nodes 104, 106, a media signal associated with the audio application is transmitted from the media source 400 to the selected media node 104, 106 via the network interface 410. In some embodiments, the media library 450 is used to decrypt the media. The media library transmits the decrypted media to the network media driver 445, and the network media driver 445 transmits the media signal to the selected output device. When the media node 104 or 106 is selected as an output device, a media signal is transmitted via the network interface 450. In some embodiments, the network media driver 445 is a Bluetooth driver. However, the network media driver 445 can support any protocol.

  Note that in the above embodiment, the virtual media application 420 never touches the media signal. This can provide the advantage that any media source 430 can be used in sending media signals to the media nodes 104, 106 by selecting the appropriate output device for the media source 400. Thus, some embodiments of the virtual network media application 420 are compatible with any media source application 430. Further, there is no need to change the media source application 430.

  As described above, some embodiments of the gateway media node 104 can reformat and process the media signal such that the media signal is compatible with the virtual media network. Thus, the gateway media node 104 can release most of the processing from the media source 102.

  It will be appreciated that the media source 400 is one example and that many modifications may be made to the media source 400 with the implementation of the methods and techniques described herein. For example, in some embodiments, the network media driver 445 may include a virtual network media driver and the virtual network media application 420 may not be present. In such embodiments, a user can install a virtual media network driver 445 to assist in sending media signals to the media nodes 104, 106. If the user wishes to send a media signal to the media nodes 104, 106, he simply selects the media node in the interface presented by the OS 440. Thereby, the virtual network media driver 445 is selected. For example, a media signal can be provided from the media library 450 to the virtual network media driver 445. Similar to the above example, the media source application 430 can be any application used to play media.

  The virtual network media application 420 can be as described above. For example, the virtual network media application 420 may provide an interface for a user to select media nodes 104, 106 to add to the virtual network and control the network. In some embodiments, the virtual network media application 420 is optional because the functionality of the virtual network media application 420 can be incorporated into the virtual network media driver 445.

  The command transmission using the command channel and the media signal transmission using the data channel can be performed using the same protocol via the network interface 410. For example, both commands and data can be transmitted according to the WiFi protocol or the Bluetooth protocol. Alternatively, as described above, commands and data can be transmitted according to different protocols.

  Note that by incorporating the driver 445 into the OS 440, media signals from any media source application 430 can be transmitted to the media nodes 104 and 106. The user need only select one of the media nodes 104, 106. In response, the virtual network media driver 445 is used. Thus, this virtual media network can be used with any media source application 430 running on the media source 400.

  In some embodiments, the media source application 430 is incorporated into the virtual network media application 420. In some such embodiments, any media played by the media source application 430 is sent to the media nodes 104,106.

  In various embodiments, media is rendered simultaneously by media source 400 and media nodes 104, 106 belonging to a virtual media network. For example, media source 400 can render a media video channel, and media nodes 104, 106 can render a media audio channel. In some such embodiments, the various media channels are kept in sync. For example, the media source 400 may send a stamp with the media data indicating when the media nodes 104, 106 should render the media data. Similarly, gateway 104 can also include a time stamp associated with this data when transferring the media data to other media nodes 104, 106. In order to allow such time stamp exchange, the various media devices 104, 106, 400 are timed between a common reference clock, such as a virtual wall clock, or various time domains as described in detail below. There can be a method for converting the stamp.

  FIG. 5 is an exemplary component diagram of media node 500. Media node 500 may correspond to one or both of media nodes 104, 106 of exemplary environment 100. The media node 500 can have a network interface 510. The network interface 510 can allow communication according to one or more wireless or wired protocols. In various embodiments, one or more antennas are connected to the network interface 510. In some embodiments, the network interface 510 can support both Wi-Fi and Bluetooth. In addition or alternatively, the network interface 510 may support any other protocol. In some embodiments, the network interface 510 includes one or more wired network interfaces.

  The renderer 520 may be responsible for processing the media signal so that it can be presented on the speaker 530, display 540, or other output device (not shown). It will be understood that various other media nodes may not include the speaker 530 or the display 540 depending on the type of media that the media node is designed to render. Furthermore, media node 500 may not include renderer 520 if it is designed to function only as a gateway or other broadcaster and not as a rendering device. The rendering module can receive media signals from the network interface 510.

  Broadcaster 550 can forward media signals to other suitable media nodes 104, 106 via network interface 510. The auxiliary output 560 can be used to provide a media signal to a device such as a home stereo system. In some embodiments, the broadcaster 550 performs the process of transferring the media signal to the auxiliary output 560. In various embodiments, media node 500 does not include auxiliary output 560. Further, the media node 500 may not include the broadcaster 550 if it is designed to function only as a rendering device and not as a gateway or other broadcasting device.

  Command module 570 can process commands for controlling media signals. These commands can include volume, play, stop, etc. The synchronization module 580 can serve to accurately synchronize the media signal during playback at various media nodes in the network. As described in detail below, the synchronization module 580 can transmit or receive beacon messages for use in setting up initial clock synchronization. Also, after the start of the media stream, the synchronization module 580 can insert or extract timestamps from the media packets for use in improving or correcting clock synchronization during media playback.

  Media nodes 104 and 106 can be controlled through various mechanisms. The controller can include a smartphone application, a tablet application, a UI on a TV or set-top box, a button with or without display on a node, or a PC application. In some embodiments, these devices control whether the renderer 520 renders a particular stream or its particular channel, the renderer 520's volume output and main volume.

  In some embodiments, the media node 500 supports a command protocol. The command protocol is a method of turning audio playback on / off, a method of consolidating audio playback into a synchronous area, transport control such as playback, forward, rewind and search, transmission of metadata to a node, to a device participating in a network Network status notifications, status updates when a device leaves the network, control via a remote user interface, and other messages and methods for maintaining an airtime network.

  The elements of the media node 500 can be implemented using software, hardware, or a combination of software and hardware. Media node 500 may have one or more processors and a computer-readable medium containing instructions that, when executed on the one or more processors, perform the functions of the various elements of media node 500. .

  FIG. 6 is an exemplary hardware diagram of the media device 600. Exemplary media device 600 may correspond to any of media devices 102, 104, 106, media source 400, or media node 500 in exemplary environment 100. As shown, the hardware device 600 may include a processor 610, a memory 620, a user interface 630, a network interface 640, and a storage device 650 interconnected via one or more system buses 660. FIG. 6 is abstract in a sense, and it will be understood that the actual component configuration of the media device 600 can be more complex than that shown. For example, processor 610 and memory 620 can be connected via a local microprocessor bus, and user interface 630, network interface 640 and storage device 650 can be connected via one or more input / output buses. Can do.

  The processor 610 can be any hardware device capable of executing instructions stored in the memory 620 or the storage device 650. Thus, the processor can include a microprocessor, field programmable gate array (FPGA), application specific integrated circuit (ASIC), or other similar device.

  The memory 620 can include various memories such as, for example, an L1 cache, an L2 cache or an L3 cache, or a system memory. The memory 620 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM) or other similar memory device.

  User interface 630 may include one or more devices for enabling communication with the user or for rendering media to the user. For example, the user interface 630 can include a display, speaker, printer, auxiliary output, mouse, keyboard, alphanumeric keypad, trackball, stylus or button.

  The network interface 640 may include one or more devices for enabling communication with other hardware devices. For example, the network interface 640 may include one or more network interface cards (NICs) configured to communicate according to the Ethernet protocol, TCP / IP protocol, WiFi protocol, or Bluetooth protocol. Various alternative or additional hardware or configurations for the network interface 640 will be apparent.

  Storage device 650 includes one or more machine-readable storage media such as read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or similar storage media. Can do. The storage device may also include a portable non-volatile storage medium, such as a floppy disk, for inputting and outputting data and codes to and from the media device 600. In various embodiments, the storage device 650 stores instructions executed by the processor 610 or data that allows the processor 610 to operate. For example, the storage device is sent by an operating system 670 for coordinating the basic functions of the media device 600 and for transmitting information sufficient for another media device to synchronize playback or by another media device. A synchronization command 672 for processing such information and synchronizing reproduction can be stored.

  Storage device 650 can store various additional instructions depending on the role or capability of media device 600. For example, if media device 600 can function as a media source, storage device 650 can receive media or play media on device 600 with media source application instructions 674, media library instructions for decrypting media. 676 or virtual network media application instructions 678 to allow the user to send commands to the virtual media network. Various other functions for these instructions 674, 676, 678 will be apparent in light of the above description, such as a description of an exemplary media source 400. Additionally or alternatively, if the media device 600 can function as a media node, the storage device can use command module instructions 680 to process commands issued by the media source or other controller, other media. Broadcaster instructions 682 for transferring media to the node and renderer instructions 684 for rendering the media in sync with other devices may be stored. Various other functions for these instructions 680, 682, 684 will be apparent in light of the above description, such as the description of the exemplary media node 500.

  The components included in media device 600 are those typically found in computer systems suitable for use with the methods and systems described herein and represent a broad category of such computer components. It is. Thus, the media device 600 can be a mobile phone, smart phone, PDA, tablet computer, personal computer, mobile computer device, workstation, server, minicomputer, mainframe computer, or any other computer device. The computer can also include various bus configurations, network platforms, multiprocessor platforms, and the like. Various operating systems 660 may be used, including Unix, Linux, Windows, Macintosh OS, Palm OS, Android OS, iOS and other suitable operating systems.

  FIG. 7 illustrates an exemplary method 700 for broadcasting a media signal. Method 700 may correspond to step 210 of method 200. In step 710, the gateway media node 104 and other media nodes 102, 104, 106 may set timing parameters. In some embodiments, media nodes 104, 106 are synchronized to a virtual wall clock. The broadcaster can use this virtual wall clock to time stamp the target rendering time in the media stream. The renderer can use this virtual wall clock to render media samples accurately at a given time. The virtual wall clock can help ensure that the media nodes 104, 106 have a common understanding of rendering time. In some embodiments, each rendering device renders a sample at a specified time in the media stream. The stream format can also include other information for rendering the stream, including sampling frequency, word size, number of channels, encoding format, and the like.

  In step 710, a virtual wall clock or some other common timing reference may be set. For example, the gateway media node 104 can initiate a “flood” of beacon messages that includes a time stamp indicating when each beacon message was transmitted. As a result, the receiving media nodes 104 and 106 can calculate the offset value by obtaining the difference between the time stamp of the sender and the time when each beacon message is transmitted. This calculated offset is the actual offset between the sending device's clock and the receiving device's clock, the network propagation time, and the sending device inserts the time stamp until it actually sends the beacon message. Related to fixed delays related to time, such as the time taken to receive the beacon message from the time the receiver device received the beacon message, and network variations common to various wireless network connections It is possible to reflect the sum of three independent values of variable delay. Since the fixed delay is substantially constant, the receiving device is the offset that includes the variable network delay with the smallest calculated offset of these calculated offsets, and therefore fixed with the actual clock offset. One can be confident that this is the closest available estimate to the network delay plus. As a result, the receiving device can adjust its clock based on this minimum offset, or it can persist this offset so that it can be used in the comparison of the time stamp of the subsequent transmitting device with the local clock. be able to. During the rendering process, this fixed network delay can be taken into account to ensure correct synchronization. A similar method can be used in establishing synchronization between the media source 102 and the media gateway 104.

  In step 720, gateway media node 104 receives a media signal from media source 102. In step 730, the gateway media node 104 decrypts the media. The gateway can demultiplex the media signal before decoding.

  In step 740, the gateway media node 104 re-encodes the media for broadcast to other media nodes 104,106. Note that the gateway can use an encoding different from the encoding used by the media source 102. For example, in the media source 102, the media signal may be encoded in a format compatible with Bluetooth. This media signal can be re-encoded in a format compatible with Wi-Fi.

  In step 750, the gateway media node 104 encapsulates the media signal. In various embodiments, the gateway media node 104 compresses the media signal. As an example of compressing an audio media signal, in a high quality network, a light lossless compression technique such as a free audio lossless codec (FLAC) can be used to cut the bandwidth in half with minimal processing overhead. In low quality networks, high compression standards such as OGG or Advanced Audio Coding (AAC) can be used to minimize network bandwidth at the expense of sound quality and processing overhead. The signal can be resampled beyond the compression algorithm itself to a low sampling rate, downmixed to a mono stream, or downsampled to a low sample resolution. By encoding or transcoding the media stream into a compressed format, airtime reliability can be increased by using narrow network bandwidth at the expense of processing overhead. Supported codecs can include lossless and lossy compression techniques of various bit rates, sampling frequencies, channels and sample sizes.

  In some embodiments, all media nodes 104, 106 are aware of supported encoding formats. In some embodiments, all broadcasters can encode to a supported format. In some embodiments, all renderers can decode supported formats. The encoding format used for each stream depends on the media nodes 104, 106 depending on feedback from network quality, available processing resources, number of supported rendering areas, number of supported active streams and maximum allowable latency. Can be determined between.

  In optional step 760, redundant packets are added. If the media signal is compressed, additional packets can be added. In some embodiments, groups of packets are interleaved with groups of redundant packets. For example, using a 2 to 1 compression ratio, 2 seconds of the original media signal can be compressed to 1 second. As an example, data packets corresponding to 1 second can be interleaved with redundant packets of 1 second. The number of packets in the group can be one or more.

  In some embodiments, broadcasting has two options. Option A allows the gateway media node 104 to broadcast media signals to other media nodes 10 at step 770 as shown. Option B (not shown) allows the gateway media node 104 to send a media signal to the wireless access point. The wireless access point can broadcast this media signal to other media nodes.

  Broadcast media is considered the largest consumer of network bandwidth. A typical uncompressed audio stream may exceed 1.5 Mbps. Upstream transmission to the access point 310 consumes 1.5 Mbps per stream, and downstream to the renderer 306 consumes an additional 1.5 Mbps per stream, which may consume a total of 3 Mbps. In point-to-point simulcasting, the typical bandwidth can be 3 Mbps times the number of simulcast streams. This may saturate the network.

  Various embodiments support multiple transmission protocols. In some embodiments, UDP over IP is used. Note that in some embodiments, the receiving media node may not acknowledge the packet reception. For example, UDP over IP may not require reception of a packet. In some embodiments, the receiving media node may request the gateway to retransmit data packets that were not received. Note that this request can be made in embodiments that use UDP over IP. As described above, in some embodiments, redundant data packets are transmitted.

  Media nodes 104, 106 can maintain network statistics. In various embodiments, the selected broadcaster or gateway is responsible for determining the optimal transmission method to balance quality of service, latency, processor usage and network usage. For example, if the network is of good quality with high bandwidth available and strong connection to the individual nodes 104, 106, a guaranteed transmission protocol can be used. If the network is saturated or of poor quality, multicast technology may be preferred. In a further manner, bandwidth savings and transmission error detection, correction or concealment can be assisted. In general, multicast, simulcast, and point-to-point protocols are supported by the most suitable protocol determined at the time of stream construction, with network quality, available processing power and number of streams factoring in the decision process .

  Via the media stream, the media clock can be recovered with respect to the wall clock and synchronized to the media frame or sample group. The media clock can facilitate the formation of a hardware frame clock, a word clock, and a bit clock. From a logical point of view, synchronization via the media stream can ensure accurate clock generation at the media nodes 104, 106. If slight variations associated with crystals or the like are present in the hardware, variations in clock drift and other clock timings may occur. By constantly measuring and comparing the media clock and wall clock, the system can detect drift. In some embodiments, a software-specific media clock recovery mechanism is involved in resynchronizing the media clock across the device by adding or removing media samples from the media rendering buffer. In some embodiments, the rendering buffer is manipulated such that no obvious clicking or skipping effects occur. Based on the drift measurement, a hardware mechanism using a VCXO or voltage controlled oscillator can be controlled from the processor, and this hardware oscillator can be pushed or pulled to closer synchronization.

  As described above, the various systems described herein can synchronize media playback among multiple devices by setting a common timing reference. For example, the media source and the media gateway can cooperate to set such a common timing reference, or the media gateway and the media node can cooperate to set such a common timing reference. In the context of setting timing parameters, the method can be divided between two media devices, i.e. between a sending media device and a receiving media device. In various embodiments, the common timing reference is an estimate of the clock value at the sending media device at the receiving media device.

  FIG. 8 illustrates an exemplary method 800 for a sending media device to synchronize playback with a receiving media device. The example method 800 may be performed by any media device that functions as a sending media device, such as the media source 102 or the media gateway 104 of the example environment 100, for example. Method 800 may be performed as part of step 710 of exemplary method 700 or at any point where timing parameter synchronization between media devices is appropriate.

  Method 800 begins at step 805 and proceeds to step 810 where it can be determined that the sending device should start flooding the “beacon message” used by the receiving device in setting the timing parameters. . For example, the sending device should determine that the receiving device is powered on, determine that the receiving device is participating in the virtual media network, or the sending device should begin sending media to the receiving device. Can be determined. In step 815, the transmitting device generates a new beacon message. This beacon message can be any type of packet or other data message recognized by the receiving device. For example, the beacon message can be formed according to a dedicated protocol implemented by both the sending device and the receiving device. In various embodiments, such as an embodiment in which a beacon message can traverse one or more intermediary devices such as routers or switches en route to a receiving device, Configure the beacon message to carry a flag or other indication that it constitutes high traffic. The various mechanisms for prioritizing beacon messages within the network differ based on the individual prioritization scheme utilized by the various possible networking technologies. Such prioritization of beacon messages helps to optimize routing time through a mixed topology network, which can reduce variable network delay factors.

  Next, in step 820, the sending device timestamps the beacon message with the time currently indicated by the sending device's clock. Such a time stamp can be referred to as a “sender time stamp”. Next, in step 825, the sending device sends a beacon message to one or more receiving devices. The time elapsed between step 820 and step 825 forms part of the fixed delay component of the clock offset value calculated by the receiving device. Accordingly, various implementations of method 800 strive to reduce or minimize the number of operations performed between steps 820 and 825. In various embodiments, the sending device is responsible for setting timing parameters for other media devices. For example, media gateway 104 can send a beacon message to other media nodes 104, 106. In some such embodiments, the sending device, for example, by addressing a copy of the beacon message to each media device individually or by multicasting the message to multiple media devices, at step 825. A beacon message is transmitted to a plurality of media devices. Alternatively or additionally, the sending device performs the method 800 multiple times to accommodate other media devices.

  In step 830, the transmitting device determines whether the transmitting device has finished flooding the beacon message to the receiving device. For example, the transmitting device can continue flooding the beacon message until a predetermined number of beacon messages are transmitted. Alternatively or in addition, in various embodiments, the sending device bases the condition of step 830 on feedback from the receiving device. For example, the receiving device can send a message when sufficient synchronization is achieved, or a message indicating that sufficient synchronization has not been achieved despite a predetermined number of beacon messages being transmitted. Can be sent. As yet another method of determining whether to stop flooding, the sending device monitors network performance during the flooding period and continues flooding until the network performance meets some minimum acceptable threshold. For example, the sending device can send a round trip diagnostic message to the receiving device in addition to the beacon message. Alternatively, the receiving device can be configured to further return a beacon message to the sending device for network diagnostic purposes. When the sending device receives the message from the receiving device, it generates one or more network performance measurements. For example, the sending device can generate a measurement of network delay or jitter over the previous flooding window, and if this measurement is below some minimum acceptable network performance, it can send a predetermined number of beacon messages. Continue to flood the beacon message even if it has been sent. In light of the teachings herein, it will be apparent that various combinations of these and other methods for determining whether beacon message flooding is sufficient can be employed.

  If, at step 830, the sending device determines that it should continue flooding the beacon message, the method 800 returns to step 815 to send additional beacon messages. In various embodiments where the flooding window includes transmission of a predetermined number of beacon messages, the sending device determines that the previous window was insufficient based on network performance or other factors and before returning to step 815. By resetting the beacon message counter to begin sending another set of beacon messages in a new window. On the other hand, if it is determined at step 830 that the sending device should finish flooding, the method 800 proceeds to step 835 and ends. Thereafter, the transmitting device transmits the media to the receiving device that is currently synchronized.

  Note that in various embodiments, the sending device may not receive a return message from the receiving device based on the beacon message, or in an embodiment in which the sending device receives the return message, the sending device may be a timing parameter. The return message is not used for the purpose of setting. In some embodiments, the return message is used only for the purpose of determining whether the flooding period is sufficient. Thus, unlike the other clock synchronization methods, the method described herein can be referred to as a “one-way” synchronization method where most of the synchronization calculations are performed by the receiving device rather than the transmitting device.

  FIG. 9 illustrates an exemplary method 900 for a receiving media device to synchronize playback with a sending media device. The example method 900 may be performed by any media device that functions as a receiving media device, such as the media gateway 104 or other media nodes 104, 106 of the example environment 100, for example. The method 900 may be performed as part of step 710 of the example method 700 or at any time when timing parameter synchronization between media devices is appropriate.

  Method 900 begins at step 905 and proceeds to step 910 where the receiving device uses a minimum offset variable “MinO” that is used to maintain the minimum offset value that is being executed when a new message is received or processed. ”. Next, in step 915, the receiving device receives a beacon message from the transmitting device. Next, in step 920, the receiving device generates a time stamp based on the time currently indicated by the clock of the receiving device. Such a time stamp can be referred to as a “recipient time stamp”, “R (x)”. The time elapsed between step 915 and step 920 forms part of the fixed delay component of the clock offset value calculated by the receiving device. Accordingly, various implementations of method 900 strive to reduce or minimize the number of operations performed between steps 920 and 925.

  In step 925, the receiving device extracts the sender time stamp “S (x)” from the beacon message. As described above, the sender timestamp is inserted into the beacon message immediately before transmission by the sender device, as in step 820 of the exemplary method 800. In step 930, the receiving device determines whether the sending device is a media source of the virtual media network. For example, if the receiving device is operating as a gateway for a virtual media network, the receiving device determines that the sending device is the media source. In such a case, method 900 proceeds to step 935. Next, the receiving device converts the sender time stamp from the time domain of the transmitting device to the time domain of the virtual media network. Such a conversion can add or subtract an offset previously negotiated between the two devices. Such negotiation and conversion between time domains can be performed according to any method known to those skilled in the art. In some other embodiments, the source device and the media node maintain a clock in the same time domain. In some such embodiments, steps 930, 935 are not present.

  The method 900 proceeds to step 940 after converting the sender timestamp to the virtual media network domain in step 935, or after determining in step 930 that the sender is not a media source, and the receiving device sends the sender timestamp. Based on the recipient time stamp, an offset value such as the difference between the two time stamps is calculated. This current offset value “CurO” is the actual offset between the sender clock and the receiver clock and any delay between the creation of the two timestamps S (x) and R (x) encountered by the beacon message. Is equivalent to As described above, this delay includes two components. The first delay component is related to the time taken to traverse the hardware and software components of the network, such as a certain delay associated with the circuit and data path through which the message travels, and the transmission / reception of the message. It is a fixed delay related to the time required by the OS between the time stamp generation. Such a fixed delay has already been taken into account as part of the rendering process. The second delay component is a variable network delay associated with a delay that changes over time. For example, a shared media network such as WiFi may wait for the media to clear before transmission, and thus different delays may be introduced at different times.

  Since this variable delay only introduces additional delay (it does not remove the delay), a better estimate of the actual clock offset is obtained from the message with the least delay. Thus, the method 900 searches for the minimum offset value obtained during flooding as the best available estimate of the actual offset. In step 945, the receiving device compares the current offset CurO with the previously existing minimum offset, or the current iteration of this loop is the first one for the minimum offset value MinO initialized in step 910. Compare if there is. If CurO is smaller than MinO, then CurO is recognized to represent a more accurate estimate of the actual offset between the sender clock and the receiver clock, and in step 950, the receiver device determines the value of MinO. Overwrite the value of CurO.

  In step 955, the recipient device determines whether the sender device is flooding a beacon message. For example, the receiver device can determine whether a timeout has occurred while waiting for a further beacon message, can determine that the sender device has begun to send a media message, and a predetermined number of beacon messages It can be determined that it has been received, or it can be determined that the sending device has sent a special message indicating the end of flooding. In various embodiments, the recipient device determines whether the flooding was sufficient to establish the desired offset accuracy. For example, the receiver device tracks the interval at which beacon messages are received and is sufficiently stable that the network produces an offset value of the desired accuracy based on a comparison of this measured interval with a known time interval. It can be judged whether or not it was. If the network is not sufficiently stable, the receiving device sends a message to the sending device indicating that further flooding should be performed. Various modifications will be apparent. In light of the teachings herein, it will be apparent that various combinations of these and other methods for determining whether beacon message flooding is sufficient can be employed.

  If the receiving device determines that further flooding is or will be performed, the method 900 returns from step 955 to step 915 to process further beacon messages. Otherwise, the method 900 proceeds to step 960 and the receiving device resets the local clock based on the determined minimum offset. For example, the receiving device can subtract MinO from the current clock value and set the local clock to a new value that is estimated to be close to the actual clock value of the transmitting device. In some embodiments where the fixed delay of the network is known or estimated, the receiving device subtracts MinO from the current clock value and re-adds it to the fixed delay value to actually calculate from the calculated offset value. Attempts to isolate the clock offset value of. In some embodiments, the receiving device does not change the local clock at all, but instead maintains a minimum offset value MinO for use in comparing the local clock with the timestamp received from the sender device. For example, the receiving device can add MinO to the time stamp before any such comparison. Various other modifications will be apparent. Method 900 may proceed to step 965 and end.

  In various other embodiments, the receiving device utilizes a previously set lower limit offset to help ensure that an unreasonably large offset calculated during the flooding period is not used to reset the clock. To do. For example, if the flooding period is included in the highly variable network delay period, the calculated offset may be greater than the actual offset value between the sender clock and the receiver clock. In some such embodiments, the receiver first compares the minimum offset calculated in steps 940-950 with a previously set lower limit offset to determine whether the minimum offset is greater than the lower limit offset. judge. If so, the recipient refuses to update the clock based on this minimum offset and continues to use the previously set lower bound. Otherwise, the minimum offset value is less than the lower limit and is therefore a better estimate than the lower limit, so the receiver updates the clock as detailed in step 960. An exemplary method for determining the lower limit is described in detail below in connection with FIG.

  In various embodiments, the receiving device periodically performs method 900 to reestablish synchronization. In some such embodiments, the receiving device resets the clock to its original value, deletes the stored offset value, or any changes made based on previous executions of the method 900. Is “rewinded” differently, and “redo” is performed to determine the clock offset. The receiving device can better consider the clock drift between the clock of the transmitting device and the clock of the receiving device by periodically resetting the clock offset.

  In light of the teachings herein, method 900 is described as a real-time method of processing each beacon message as it is received, while various other embodiments utilize a method of processing beacon messages as a batch. It will be clear. For example, in some such embodiments, the receiving device receives a plurality of beacon messages, time stamps the messages as they are received, and thereafter, similar to the method described with respect to steps 925-960. In this way, the received messages are processed in order to find the minimum offset.

  It will be appreciated that the method described above attempts to generate the best clock offset estimate between the two devices. After this initial flooding period, the network condition may be temporarily improved and a better estimate may be obtained later. Therefore, after the initial timing parameter is set, a method of trying to estimate the clock offset better can be adopted. Such a method may also address the possibility of clock drift where the transmitting device clock and the receiving device clock may operate at slightly different speeds due to differences in crystal, temperature or other parameters.

  FIG. 10 illustrates an example method 1000 for a receiving media device to obtain good playback synchronization during media streaming. The example method 1000 may be performed by any media device that functions as a receiving media device, such as, for example, the media gateway 104 or other media nodes 104, 106 of the example environment 100. The method 1000 may be performed as part of step 780 of the exemplary method 700 or at any time when timing parameter synchronization between media devices is appropriate.

  Method 1000 starts at step 1005 and proceeds to step 1010, where the receiving device receives a media data packet from the sending device. Next, in step 1015, the receiving device generates a time stamp R (x) based on the time currently indicated by the clock of the receiving device. In step 1020, the receiving device extracts the sender timestamp “S (x)” from the media data message. The sender time stamp can be inserted into the media data message immediately before transmission by the sender device. In step 1025, the receiving device determines whether the sending device is a media source of the virtual media network. For example, if the receiving device is operating as a gateway for a virtual media network, the receiving device can determine that the sending device is the media source. In such a case, method 1000 proceeds to step 1030. Next, the receiving device converts the sender time stamp from the time domain of the transmitting device to the time domain of the virtual media network. Such a conversion can add or subtract an offset previously negotiated between the two devices. Such negotiation and conversion between time domains can be performed according to any method known to those skilled in the art. In some other embodiments, the source device and the media node maintain a clock in the same time domain. In some such embodiments, steps 1020, 1030 are present.

  The method 1000 proceeds to step 1035 after converting the sender timestamp to the virtual media network domain in step 1030, or after determining in step 1025 that the sender is not a media source, and the receiving device sends the sender timestamp Based on the recipient time stamp, an offset value such as the difference between the two time stamps is calculated. If the sender time stamp has been converted, this converted time stamp is used in the offset calculation. This offset value “O” is the fixed offset between the actual offset between the sender clock and the receiver clock and the creation of the two timestamps S (x) and R (x) encountered by the media data message and It corresponds to the sum of all delays including both variable delays. In step 1040, the receiving device determines whether this offset value represents an offset estimate between clocks that is better than the previously utilized offset value. For example, in various embodiments that reset the receiving device's clock using a previously determined minimum offset, the receiving device determines whether the current offset O is less than zero. The positive result of this comparison is that some of the previously used minimum offset incorporates some variable network delay, and subtracting this variable network delay from the local clock will “exceed” the ideal setpoint. Indicates that the local clock is set later than the sender's clock. The current offset O can account for this excess by being a negative number by incorporating a variable delay that is less (or zero) than the previously used minimum. In such a case, it is determined that the current offset O represents the new best estimate of the actual clock offset, and in step 1045 the previous clock is exceeded by resetting the local clock again with this offset O. Can be at least partially rectified. Various modifications for other embodiments will be apparent. For example, in an embodiment where the previously determined minimum offset is not used for local clock correction and continues to be used in the timestamp comparison, in step 1040 is the current offset O less than the previous minimum offset MinO? In step 1045, the receiving side apparatus sets MinO to be equal to zero. Various other modifications will be apparent.

  In various other embodiments, the receiving device utilizes a previously set lower limit offset to help ensure that an unreasonably large offset calculated during the flooding period is not used to reset the clock. To do. In some such embodiments, the receiver first compares the offset calculated in step 1035 to a previously set lower limit offset, and estimates that this offset is better than the lower limit offset. Whether or not is represented. If so, the recipient refuses to update the clock based on this minimum offset and continues to use the previously set lower bound. Otherwise, the offset value is a better estimate than the lower limit, so the receiver updates the clock as detailed in step 1045. An exemplary method for determining the lower limit is described in detail below in connection with FIG.

  In step 1050, the receiving device proceeds to process the received media packet, eg, renders the media output at an appropriate time. For example, the receiving device can extract the presentation time and separate the sender time stamp and the receiver time stamp from the media data packet. Such a presentation time indicates a time at which the media data carried by the message should be rendered. The receiving device extracts the presentation time, and then renders the media data at a time that matches the presentation time. For example, the receiving device can store the media data in a buffer for playback by the local playback device, or can forward the message to another media node for playback. The current time that “matches” the presentation time can include equality of the current time and the presentation timestamp, but can also include other forms of matching. For example, in various embodiments, the current time matches when the current time minus the minimum remaining offset value is equal to the presentation timestamp. In addition or alternatively, the comparison for matching adds, subtracts, or otherwise considers a fixed delay value. Various other methods for determining a suitable time for playback based on the local clock, the presentation time stamp, and other potentially available values will be apparent. Furthermore, the notion that the current time matches the presentation time based on the minimum offset is that comparisons using a local clock previously modified with the minimum offset value do not explicitly account for the minimum offset value otherwise. It will be understood to include. Various embodiments make such comparisons immediately before output to ensure that the data is output at the appropriate time. Other embodiments use such a comparison to insert media data at a location in the playback buffer where the media is likely to be played at the presentation time. Such insertion may include inserting “dummy” data before inserting media data to adjust playback timing. Various additional methods for controlling the playback timing of the data in the buffer will be apparent.

  FIG. 11 illustrates an exemplary method 1100 for determining a lower limit offset. As described above, various alternative embodiments further set a lower limit offset prior to beacon flooding and analysis of media packets in order to obtain a better clock offset estimate. The example method 1100 may be performed by any media device that functions as a receiving media device, such as, for example, the media gateway 104 or other media nodes 104, 106 of the example environment 100. The method 1100 may be performed as part of step 710 of the exemplary method 700 or at any point where timing parameter synchronization between media devices is appropriate.

  The method 1100 begins at step 1105 and proceeds to step 1110, where the receiving device calculates from the round trip transport between the receiving device and the sending device so as to calculate a lower bound by receiving a handshake message from the sending device. Start collecting time stamps. In various embodiments, this handshake message is transmitted over a channel that is different from the channel on which the beacon message or media data packet is transmitted. For example, a handshake message can be transmitted over a Bluetooth channel while a beacon message and a media data packet message can be transmitted over a WiFi channel. As a part of the handshake protocol to be used, the sender includes in the handshake message a time stamp t1 indicating the time according to the sender clock at which the sender sent the handshake message. Next, in step 1115, the receiver records a reception time stamp t2 according to the receiver clock at a time close to the reception of the handshake message in step 1110.

  Next, the receiver device prepares to return the handshake message to the sender device by generating in step 1120 a time stamp t3 indicating the time according to the receiver clock at which the receiver resends the handshake message to the sender. . In some embodiments, the recipient can insert the time stamp t3 into a handshake message received from the sender or a newly generated handshake message. Next, in step 1125, the receiver sends a handshake message to the sender at a time close to the generation of the time stamp t4. Next, in step 1130, the recipient can again receive a handshake message from the sender. At this point, the handshake message includes a time stamp t4 according to the sender clock indicating the time when the sender received the handshake message as part of the step of the sender processing the handshake message.

  As will be described later, the lower limit offset can be calculated using these four time stamps t1 to t4. However, in some embodiments, the recipient device first uses the time stamps t1-t4 to calculate the network transit time experienced by the handshake message so that the network delay during the handshake process is accurate or otherwise Determine if it was low enough to provide an acceptable lower limit. Accordingly, in step 1135, the receiver device calculates the transport time using the formula transport time = ((t2−t1) + (t4−t3)) / 2. The recipient then determines whether the calculated transport time is acceptable, for example by determining whether the transport time is shorter than a predetermined threshold. If the calculated transit time is unacceptable, the recipient device instructs the sender to try the handshake process again at step 1145 and returns to step 1110 to retry the process. On the other hand, if the network transit time is acceptable, the recipient uses the time stamp in step 1150 using the lower limit offset = ((t2-t1)-(t4-t3)) / 2 formula. To calculate the lower limit offset. Thereafter, the method proceeds to step 1155 and ends.

  In some embodiments, the sender clock may be behind the receiver clock, so that the timestamp generated by the receiver device is smaller than the timestamp generated by the sender device. Will be clear. It will be appreciated that the above lower limit clock offset equation can indicate the direction of clock adjustment based on the sign of the calculated value. In some embodiments, the receiver device compares the various calculated offsets so that only the magnitude is compared, not the relative adjustment direction, and which offset is a better estimate. The absolute value can be used in the determination.

  In various embodiments, the receiving device periodically performs method 1100 to reestablish synchronization. In some such embodiments, the receiving device resets the clock to its original value, deletes the stored lower offset value, or any that has been done based on a previous execution of the method 1100. “Rewind” the change differently and “redo” to determine the clock offset. The receiving side apparatus can better consider the clock drift between the clock of the transmitting side apparatus and the clock of the receiving side apparatus by periodically resetting the lower limit offset.

  In light of the above, various embodiments allow for synchronization of media playback between media devices belonging to a network exhibiting variable delay. For example, the effect of variable delay on clock synchronization can be reduced by implementing a one-way synchronization method in which the receiving device identifies the minimum clock offset from multiple messages. Further, the receiving device can improve synchronization while considering clock drift by continuously searching for better synchronization parameters after the media stream is started. Various additional advantages will be apparent in light of the above.

  From the foregoing description, it will be apparent that various exemplary embodiments of the invention may be implemented in hardware. Also, the various exemplary embodiments are implemented as instructions stored on a machine-readable storage medium that can be read by at least one processor and executed to perform the operations detailed herein. You can also. A machine-readable storage medium may include any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, server or other computing device. Thus, tangible non-transitory machine readable storage media may include read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices and similar storage media. Further, as used herein, the term “processor” refers to a microprocessor, field programmable gate array (FPGA), application specific integrated circuit (ASIC), or any other device capable of performing the functions described herein. Will be understood to include.

  Those skilled in the art will appreciate that any block diagram herein represents a conceptual diagram of an exemplary circuit embodying the principles of the present invention. Similarly, any flowcharts, flow diagrams, state transition diagrams, pseudo code, etc. may be represented in substantially machine-readable media, and thus, whether or not explicitly depicted a computer or processor. It will be understood that it represents various processes that can be performed by such a computer or processor.

  Although various exemplary embodiments have been described in detail with particular reference to certain exemplary aspects of the invention, other embodiments are possible in the invention, the details of which are not It should be understood that the point can be modified. It will be readily apparent to those skilled in the art that variations and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the foregoing disclosure, description, and figures are for illustrative purposes only and are in no way intended to limit the invention, which is defined solely by the claims.

100 Exemplary Environment 102a Media Source 102b Media Source 104a Living Room Gateway 104b Bedroom Gateway 106a Office Renderer 106b Kitchen Renderer 106c Guest Bedroom Renderer 108 Stereo Receiver 110 Speakers

Claims (15)

A method of synchronizing media playback between a sending media device and a receiving media device, comprising:
Receiving a plurality of messages including a plurality of sender timestamps from the sender media device at the receiver media device;
Generating a plurality of clock offset values based on the plurality of transmitting time stamps and the clock of the receiving media device;
Identifying a minimum clock offset value from the plurality of clock offset values;
Determining first media data for reproduction and a first presentation time associated with the first media data;
Causing the first media data to be rendered at a first time that matches the first presentation time based on the minimum clock offset;
A method comprising the steps of: Identifying the minimum clock offset value comprises:
Generating a first clock offset value of the plurality of clock offset values;
Setting the minimum clock offset value equal to the first clock offset value;
Generating a second clock offset value of the plurality of clock offset values after setting the minimum clock offset value equal to the first clock offset value;
Determining that the second clock offset value is less than the minimum clock offset value;
Setting the minimum clock offset value equal to the second clock offset value based on a determination that the second clock offset value is less than the minimum clock offset value;
The method of claim 1, comprising: Modifying the value of the clock by subtracting the minimum offset value from the value of the clock of the receiving media device;
The first time that matches the first presentation time based on the minimum clock offset value includes the value of the clock that matches the first presentation time;
The method according to claim 1. The plurality of messages includes a first plurality of messages and a second plurality of messages, the method comprising:
Generating a first measurement for the interval at which the first plurality of messages arrive;
Determining that the first measurement for the interval at which the first plurality of messages arrive indicates that the network is unstable;
Instructing the sending media device to send an additional message;
Generating a second measurement for the interval at which the second plurality of messages arrive after instructing the sending media device to send an additional message;
Determining that the second measurement for the interval at which the second plurality of messages arrives indicates that the network is stable;
Further including
Identifying a minimum clock offset value from the plurality of clock offset values comprises utilizing at least one clock offset value of the plurality of clock offset values associated with the second plurality of messages;
The method according to claim 1. The plurality of messages includes a first plurality of messages and a second plurality of messages, the method comprising:
Sending the first plurality of messages by the sending media device;
Generating, by the sending media device, a first measurement for network performance associated with transmission of the first plurality of messages;
Determining that the first measurement of the network performance indicates that the network is unstable;
Transmitting the second plurality of messages by the transmitting media device based on a determination that the first measurement of the network performance indicates that the network is unstable;
Generating, by the sending media device, a second measurement for network performance associated with sending the second plurality of messages;
Determining that the second measurement of the network performance indicates that the network is stable;
The method of claim 1 further comprising: A receiving media device for synchronizing media playback with a sending media device,
A memory configured to store media data for playback;
A network interface configured to communicate with the sending media device;
A processor;
The processor comprises:
Receiving a plurality of messages including a plurality of sender timestamps from the sender media device via the network interface;
Generating a plurality of clock offset values based on the plurality of sender timestamps and the clock of the receiver media device;
Identifying a minimum clock offset value from the plurality of clock offset values;
Defining first media data for reproduction and a first presentation time associated with the first media data;
The first media data is rendered at a first time coinciding with the first presentation time based on the minimum clock offset value;
Configured as
A receiving-side media device. When the processor identifies the minimum clock offset value,
Generating a first clock offset value of the plurality of clock offset values;
Setting the minimum clock offset value equal to the first clock offset value;
After setting the minimum clock offset value equal to the first clock offset value, generating a second clock offset value of the plurality of clock offset values;
Determining that the second clock offset value is less than the minimum clock offset value;
Setting the minimum clock offset value equal to the second clock offset value based on a determination that the second clock offset value is less than the minimum clock offset value;
7. The receiving-side media device according to claim 6, wherein the receiving-side media device is configured as described above. The processor is
Further configured to modify the value of the clock by subtracting the minimum offset value from the value of the clock of the receiving media device;
The first time that matches the first presentation time based on the minimum clock offset value is the value of the clock that matches the first presentation time;
The receiving-side media device according to claim 6. The plurality of messages includes a first plurality of messages and a second plurality of messages, the processor comprising:
Generating a first measurement for the interval at which the first plurality of messages arrive;
Determining that the first measurement for the interval of arrival of the first plurality of messages indicates that the network is unstable;
Instructing the sending media device to send an additional message;
After instructing the sending media device to send an additional message, generating a second measurement for the interval at which the second plurality of messages arrive;
Determining that the second measurement for the interval at which the second plurality of messages arrive indicates that the network is stable;
Further configured as
The processor utilizes at least one clock offset value of the plurality of clock offset values associated with the second plurality of messages when identifying a minimum clock offset value from the plurality of clock offset values. Composed of,
The receiving-side media device according to claim 6. The receiving media device of claim 6;
The transmitting media device;
The plurality of messages includes a first plurality of messages and a second plurality of messages, and the transmitting media device comprises:
A sending media device network interface configured to communicate with the receiving media device;
A sending media device processor;
The transmitting media device processor comprises:
Sending the first plurality of messages;
Generating a first measure of network performance associated with transmission of the first plurality of messages;
Determining that the first measurement of the network performance indicates that the network is unstable;
Sending the second plurality of messages based on a determination that the first measurement of the network performance indicates that the network is unstable;
Generating a second measure of network performance associated with transmission of the second plurality of messages;
Determining that the second measurement of the network performance indicates that the network is stable;
Configured as
A system characterized by that. A non-transitory machine-readable storage medium encoded with instructions for synchronizing media playback between a sending media device and a receiving media device executed by a receiving media device,
Instructions for receiving a plurality of messages including a plurality of sender timestamps from the sender media device at the receiver media device;
An instruction for generating a plurality of clock offset values based on the plurality of transmitting time stamps and the clock of the receiving media device;
An instruction for identifying a minimum clock offset value from the plurality of clock offset values;
Instructions for determining first media data for reproduction and a first presentation time associated with the first media data;
Instructions for causing the first media data to be rendered at a first time that coincides with the first presentation time based on the minimum clock offset value;
A non-transitory machine-readable storage medium. The instruction for identifying the minimum clock offset value is:
An instruction for generating a first clock offset value of the plurality of clock offset values;
An instruction to set the minimum clock offset value equal to the first clock offset value;
An instruction for generating a second clock offset value of the plurality of clock offset values after setting the minimum clock offset value equal to the first clock offset value;
An instruction to determine that the second clock offset value is less than the minimum clock offset value;
An instruction for setting the minimum clock offset value equal to the second clock offset value based on a determination that the second clock offset value is less than the minimum clock offset value;
The non-transitory machine-readable storage medium of claim 11, comprising: Further comprising an instruction to modify the value of the clock by subtracting the minimum offset value from the value of the clock of the receiving media device;
The first time that matches the first presentation time based on the minimum clock offset value is the value of the clock that matches the first presentation time;
The non-transitory machine-readable storage medium of claim 11. The plurality of messages includes a first plurality of messages and a second plurality of messages, and the non-transitory machine-readable storage medium comprises:
Instructions for generating a first measurement for the interval at which the first plurality of messages arrive;
Instructions for determining that the first measurement for the interval at which the first plurality of messages arrive indicates that the network is unstable;
Instructions for instructing the sending media device to send an additional message;
Instructions for generating a second measurement for the interval at which the second plurality of messages arrive after instructing the sending media device to send an additional message;
Instructions for determining that the second measurement for the interval at which the second plurality of messages arrives indicates that the network is stable;
Further including
An instruction for identifying a minimum clock offset value from the plurality of clock offset values is an instruction for using at least one clock offset value of the plurality of clock offset values associated with the second plurality of messages. including,
The non-transitory machine-readable storage medium of claim 11. A set of non-transitory machine-readable media,
The non-transitory machine-readable storage medium of claim 11;
A further non-transitory machine-readable storage medium encoded with instructions to be executed by the sending media device;
The plurality of messages includes a first plurality of messages and a second plurality of messages, and the further non-transitory machine-readable storage medium comprises:
Instructions for transmitting the first plurality of messages;
Instructions for generating a first measurement for network performance associated with transmission of the first plurality of messages;
Instructions for determining that the first measurement of the network performance indicates that the network is unstable;
Instructions for sending the second plurality of messages based on the determination that the first measurement of the network performance indicates that the network is unstable;
Instructions for generating a second measurement of network performance associated with transmission of the second plurality of messages;
Instructions for determining that the second measurement of the network performance indicates that the network is stable;
including,
A set of non-transitory machine-readable media characterized in that

Images