EP3175372A1

EP3175372A1 - System and method for providing and interacting with coordinated presentations

Info

Publication number: EP3175372A1
Application number: EP15828271.5A
Authority: EP
Inventors: Edo Segal; Dmytro Panin
Original assignee: Touchcast LLC
Current assignee: Touchcast LLC
Priority date: 2014-07-30
Filing date: 2015-07-30
Publication date: 2017-06-07
Also published as: WO2016019146A1; EP3175372A4

Abstract

Disclosed is a system and method for generating a coordinated presentation. Content selected by a user is curated for inclusion in the coordinated presentation, and a plurality of images are captured for inclusion in the coordinated presentation. A first data stream associated with the curated content and a second data stream associated with the captured images are received, and a first arrangement of content associated with at least some of the first and second data streams are output. Telemetry information associated with at least one of a gyroscope and accelerometer is processed and at least some of the curated content is modified in accordance with at least some of the telemetry information and/or at least one of the plurality of images comprised in the second data stream. Further, respective content associated the two data streams are integrated to generate the coordinated presentation.

Description

SYSTEM AND METHOD FOR PROVIDING AND INTERACTING WITH

COORDINATED PRESENTATIONS

Cross-Reference To Related Application

[0001] This patent application is based on and claims priority to U.S. Provisional

Patent Application Serial Number 62/031,114, filed on July 30, 2014, and is a continuation- in-part of U.S. Patent Application Serial Number 14/316,536, filed June 26, 2014, the entire contents of each of which are incorporated by reference herein as if expressly set forth in their respective entireties.

Field

[0002] The present application relates, generally, to content presentation and, more particularly, to a system and method for providing and interacting with coordinated presentations.

Background

[0003] Interactive and supplemental content that has been made available to viewers has been done through a decoupled, separate communication channel. For instance, a producer can provide a separate communication channel with data, a video stream, or both over the Internet via a URL associated with the broadcast. For example, a television station can have on-air programming and also provide supplemental Internet-based content available through a website. Apart from sponsoring both sources of information, these communication channels are generally decoupled from one another. In other words, the broadcaster has only an indirect relationship to the viewer with regard to any supplemental content.

[0004] The present application addresses this in a system and method in which the broadcaster, who may be an individual using a portable computer device, provides viewers with the ability to launch supplemental content that has been curated by the broadcaster to the topics and information sources chosen by the broadcaster. As such, a more personal and deeper experience can be had by utilizing the present invention.

[0005] It is with respect to these and other considerations that the disclosure made herein is presented.

Brief Summary of the Invention

[0006] In one or more implementations, herein is a system and method for generating a coordinated presentation. A computing device is provided that includes a processor and a memory, the computing device being configured by code stored in the memory and executed by the processor. Curated content is selected by a user for inclusion in the coordinated presentation, and a plurality of images are captured for inclusion in the coordinated presentation. A first data stream associated with the curated content and a second data stream associated with the captured images are received, and a first arrangement of at least some content associated with the first and second data stream is output. Moreover, telemetry information associated with at least one of a gyroscope and accelerometer is processed, and at least some of the curated content is modified in accordance with at least some of the telemetry information and/or at least one of the plurality of images comprised in the second data stream. Moreover, respective content associated with the two data streams is integrated to generate the coordinated presentation. The coordinated presentation is capable of transmission to and receipt by one or more remote devices, and wherein the coordinated presentation is configured to enable interaction with at least a portion of the curated content at each of the remote devices.

[0007] These and other aspects, features, and advantages of the invention can be understood with reference to the following detailed description of certain embodiments of the invention taken together in conjunction with the accompanying drawing figures. Brief Description Of The Drawings

[0008] Further aspects of the present disclosure will be more readily appreciated upon review of the detailed description of its various embodiments, described below, when taken in conjunction with the accompanying drawings, of which: [0009] Fig. 1 is a diagram illustrating an example hardware arrangement that operates for providing the systems and methods disclosed herein;

[0010] Fig. 2A is a block diagram that illustrates functional elements of a computing device in accordance with an embodiment;

[0011] Fig. 2B is a block diagram representing a plurality of modules that provide functionality shown and described herein;

[0012] Fig. 3 illustrates an example coordinated presentation that illustrates the effect of green screen functionality provided in accordance with one or more implementations of the present application;

[0013] Figs. 4A and 4B illustrate two video frames of an example coordinated presentation;

[0014] Fig. 5 is a simple block diagram that represents a rig that is configured with a camera and computing device;

[0015] Fig. 6 is a block diagram illustrating features of the present application and configured to process two data streams into one single stream; [0016] Fig. 7 illustrates an example representation, which includes a viewing area that includes a foreground element and two background elements, and a virtual background that extends past the viewing area;

[0017] Fig. 8 illustrates an example representation of authoring a coordinated presentation in accordance with the present application; [0018] Fig. 9 illustrates another example representation of authoring a coordinated presentation in accordance with the present application; and [0019] Fig. 10 is a flow diagram showing a routine that illustrates a broad aspect of a method for authoring a coordinated presentation.

Detailed Description of the Embodiments [0020] By way of introduction and overview, in one or more implementations the present application provides systems and methods for authoring and playing video that can be layered with interactive content, including content that is available over one or more data communication networks, such as the Internet. Devices operating, for example, iOS, ANDROID, WINDOWS MOBILE, BLACKBERRY, MAC OS, WINDOWS or other operating systems are configured to provide functionality, such as an authoring tool and interface for developing distributable coordinated presentations including videos that include customizable and interactive features for use by one or more end-users that receive the presentations. Further, the software applications provide a viewing/interactive tool, referred to herein, generally, as a "consuming" interface, for end-users who receive videos that are authored in accordance with the present application. Using the client interface, users may interact with videos as a function of touch and gestures, as well as other suitable interfaces, such as a mouse, trackball, keyboard or other input. Some functionality available for end- users is defined by an author.

[0021] In one or more implementations, a video mixer module can be provided that comprises instructions executing so as to configure a processor to integrate a plurality of images captured by a camera together with a portion of the curated content via a user selection from a touch-screen interface, and thereby to generate a coordinated presentation that is capable of transmission to and receipt by one or more remote devices; and wherein the coordinated presentation is configured to enable interaction with the portion of the curated content at each of the remote devices such that results of a respective interaction at a particular remote device are viewable at the particular remote device but are not viewable at (a) other of the remote devices and (b) the display.

[0022] In one or more implementations, coordinated presentations may be configured with interactive options, which may include images, video content, website content, or computer programs (collectively referred to herein, generally, as "vApps"). An authoring tool can be integrated with a player tool, and the tools enable one or more vApps to be embedded in video or, in one or more implementations, a code is embedded in the video for enhanced functionality. For example, a play/pause button can be provided that enables a user to play or pause playback of a coordinated presentation. A timeline function can be provided that identifies a current time location within a coordinated presentation as well as to navigate therein. VApp icons can be provided that represent vApps that are included with the current coordinated presentation at respective time locations therein. In accordance with one or more implementations, as a respective icon is selected by the user, the coordinated presentation jumps to the corresponding time location, and the user can interact with the respective vApp. Information can be time coded in video, and selectable user interactive elements for navigation/time can be provided.

[0023] In one or more implementations, the present application includes and improves functionality for chroma key composting, often referred to as use of a "green screen" and/or "blue screen." In one or more implementations, a computing device configured with an authoring tool and interface for developing distributable coordinated presentations manipulates background content provided in a coordinated presentation as a function of the movement and angle the camera(s) used during recording of the coordinated presentation. In addition or in the alternative to basing the manipulation of background content on movement and/or the angle of a camera used during recording, one or more foreground elements can be the basis of such background manipulation. Thus, a module executing on a device configured with an authoring tool detects an angle of view, such as a function of camera position and/or angle of view of one or more foreground elements, and manipulates the appearance of the composited background content to eliminate an otherwise static appearance of the background content. The background content can be, for example, a composited in place of background provided in a respective color range (e.g., green). By manipulating the background content in a dynamic way, the background content eliminates a static appearance that otherwise decreases the realism of the imagery, and requires more from the viewer to suspend disbelief. By adjusting and/or manipulating the virtual background in association with movement or visual appearance of foreground elements, the coordinated presentation appears significantly more real to the viewer. [0024] Various embodiments and aspects of the invention(s) will be described with reference to details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments of the present application. [0025] Referring to Fig. 1 a diagram is provided of an example hardware arrangement that operates for providing the systems and methods disclosed herein, and designated generally as system 100. System 100 can include one or more data processing apparatuses 102 that are at least communicatively coupled to one or more user computing devices 104 across communication network 106. Data processing apparatuses 102 and user computing devices 104 can include, for example, mobile computing devices such as tablet computing devices, smartphones, personal digital assistants or the like, as well as laptop computers and/or desktop computers. Further, one computing device may be configured as a data processing apparatus 102 and a user computing device 104, depending upon operations be executed at a particular time. In addition, an audio/visual capture device 105 is depicted in Fig. 1, which can be configured with one or more cameras (e.g., front-facing and rear-facing cameras), a microphone, a microprocessor, and a communications module(s) and that is coupled to data processing apparatus 102. The audio/visual capture device 105 can be configured to interface with one or more data processing apparatuses 102 for producing high- quality audio/video content. [0026] With continued reference to Fig. 1, data processing apparatus 102 can be configured to access one or more databases for the present application, including image files, video content, documents, audio/video recordings, metadata and other information. In addition, data processing apparatus 102 can be configured to access Internet websites and other online content. It is contemplated that data processing apparatus 102 can access any required databases via communication network 106 or any other communication network to which data processing apparatus 102 has access. Data processing apparatus 102 can communicate with devices including those that comprise databases, using any known communication method, including Ethernet, direct serial, parallel, universal serial bus ("USB") interface, and/or via a local or wide area network. [0027] User computing devices 104 communicate with data processing apparatuses 102 using data connections 108, which are respectively coupled to communication network 106. Communication network 106 can be any communication network, but is typically the Internet or some other global computer network. Data connections 108 can be any known arrangement for accessing communication network 106, such as the public internet, private Internet (e.g. VPN), dedicated Internet connection, or dial-up serial line interface

protocol/point-to-point protocol (SLIPP/PPP), integrated services digital network (ISDN), dedicated leased-line service, broadband (cable) access, frame relay, digital subscriber line (DSL), asynchronous transfer mode (ATM) or other access techniques.

[0028] User computing devices 104 preferably have the ability to send and receive data across communication network 106, and are equipped with web browsers, software applications, or other software and/or hardware tools, to provide received data on audio/visual devices incorporated therewith. By way of example, user computing device 104 may be personal computers such as Intel Pentium-class and Intel Core-class computers or Apple Macintosh computers, tablets, smartphones, but are not limited to such computers. Other computing devices which can communicate over a global computer network such as palmtop computers, personal digital assistants (PDAs) and mass-marketed Internet access devices such as WebTV can be used. In addition, the hardware arrangement of the present invention is not limited to devices that are physically wired to communication network 106, and that wireless communication can be provided between wireless devices and data processing apparatuses 102. In one or more implementations, the present application provides improved processing techniques to prevent packet loss, to improve handling interruptions in communications, and other issues associated with wireless technology.

[0029] According to an embodiment of the present application, user computing device 104 provides user access to data processing apparatus 102 for the purpose of receiving and providing information. Examples and description of specific functionality provided by system 100, and in particular data processing apparatuses 102, is described in detail below. [0030] System 100 preferably includes software that provides functionality described in greater detail herein, and preferably resides on one or more data processing apparatuses 102 and/or user computing devices 104. One of the functions performed by data processing apparatus 102 is that of operating as a web server and/or a web site host. Data processing apparatuses 102 typically communicate with communication network 106 across a permanent i.e., un-switched data connection 108. Permanent connectivity ensures that access to data processing apparatuses 102 is always available. [0031] Fig. 2A illustrates, in block diagram form, an exemplary data processing apparatus 102 and/or user computing device 104 that can provide functionality in accordance with interactive conferencing, as described herein. Although not expressly indicated, one or more features shown and described with reference with Fig. 2A can be included with or in the audio/visual capture device 105, as well. Data processing apparatus 102 and/or user computing device 104 may include one or more microprocessors 205 and connected system components (e.g., multiple connected chips) or the data processing apparatus 102 and/or user computing device 104 may be a system on a chip.

[0032] The data processing apparatus 102 and/or user computing device 104 includes memory 210 which is coupled to the microprocessor(s) 205. The memory 210 may be used for storing data, metadata, and programs for execution by the microprocessor(s) 205. The memory 210 may include one or more of volatile and non-volatile memories, such as Random Access Memory ("RAM"), Read Only Memory ("ROM"), Flash, Phase Change Memory ("PCM"), or other type of memory. [0033] The data processing apparatus 102 and/or user computing device 104 also includes an audio input/output subsystem 215 which may include a microphone and/or a speaker for, for example, playing back music, providing telephone or voice/video chat functionality through the speaker and microphone, etc.

[0034] A display controller and display device 220 provides a visual user interface for the user; this user interface may include a graphical user interface which, for example, is similar to that shown on a Macintosh computer when running Mac OS operating system software or an iPad, iPhone, or similar device when running mobile computing device operating system software.

[0035] The data processing apparatus 102 and/or user computing device 104 also includes one or more wireless transceivers 230, such as an IEEE 802.11 transceiver, an infrared transceiver, a Bluetooth transceiver, a wireless cellular telephony transceiver (e.g., 1G, 2G, 3G, 4G), or another wireless protocol to connect the data processing system 100 with another device, external component, or a network. In addition, Gyroscope/Accelerometer 235 can be provided.

[0036] It will be appreciated that one or more buses, may be used to interconnect the various modules in the block diagram shown in Fig. 2A. [0037] The data processing apparatus 102 and/or user computing device 104 may be a personal computer, tablet-style device, such as an iPad, a personal digital assistant (PDA), a cellular telephone with PDA-like functionality, such as an iPhone, a Wi-Fi based telephone, a handheld computer which includes a cellular telephone, a media player, such as an iPod, an entertainment system, such as a iPod touch, or devices which combine aspects or functions of these devices, such as a media player combined with a PDA and a cellular telephone in one device. In other embodiments, the data processing apparatus 102 and/or user computing device 104 may be a network computer or an embedded processing apparatus within another device or consumer electronic product. [0038] The data processing apparatus 102 and/or user computing device 104 also includes one or more input or output ("I/O") devices and interfaces 225 which are provided to allow a user to provide input to, receive output from, and otherwise transfer data to and from the system. These I/O devices may include a mouse, keypad or a keyboard, a touch panel or a multi-touch input panel, camera, network interface, modem, other known I/O devices or a combination of such I/O devices. The touch input panel may be a single touch input panel which is activated with a stylus or a finger or a multi-touch input panel which is activated by one finger or a stylus or multiple fingers, and the panel is capable of distinguishing between one or two or three or more touches and is capable of providing inputs derived from those touches to the data processing apparatus 102 and/or user computing device 104. The I/O devices and interfaces 225 may include a connector for a dock or a connector for a USB interface, Fire Wire, etc. to connect the system 100 with another device, external component, or a network.

[0039] Moreover, the I/O devices and interfaces can include gyroscope and/or accelerometer 227, which can be configured to detect 3-axis angular acceleration around the X, Y and Z axes, enabling precise calculation, for example, of yaw, pitch, and roll. The gyroscope and/or accelerometer 227 can be configured as a sensor that detects acceleration, shake, vibration shock, or fall of a device 102/104, for example, by detecting linear acceleration along one of three axes (X, Y and Z). The gyroscope can work in conjunction with the accelerometer, to provide detailed and precise information about the device's axial movement in space. More particularly, the 3 axes of the gyroscope combined with the 3 axes of the accelerometer enable the device to recognize approximately how far, fast, and in which direction it has moved to generate telemetry information associated therewith, and that is processed to generate coordinated presentations, such as shown and described herein.

[0040] It will be appreciated that additional components, not shown, may also be part of the data processing apparatus 102 and/or user computing device 104, and, in certain embodiments, fewer components than that shown in Fig. 2A may also be used in data processing apparatus 102 and/or user computing device 104. It will be apparent from this description that aspects of the inventions may be embodied, at least in part, in software. That is, the computer-implemented methods may be carried out in a computer system or other data processing system in response to its processor or processing system executing sequences of instructions contained in a memory, such as memory 210 or other machine -readable storage medium. The software may further be transmitted or received over a network (not shown) via a network interface device 225. In various embodiments, hardwired circuitry may be used in combination with the software instructions to implement the present embodiments. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, or to any particular source for the instructions executed by the data processing apparatus 102 and/or user computing device 104.

[0041] Fig. 2B is a block diagram representing a plurality of modules 250 that provide functionality shown and described herein. It is to be appreciated that several of the logical operations described herein can be implemented (1) as a sequence of computer implemented acts or program modules running on the various devices of the system 100 and/or (2) as interconnected machine logic circuits or circuit modules within the system (100). One or more particular implementations can be configured as a function of specifications of a particular device (e.g., size, energy, consumption, performance, etc.). Accordingly, the logical operations described herein are referred to variously as operations, steps, structural devices, acts, or modules. As referenced above, the various operations, steps, structural devices, acts and modules can be implemented in software, in firmware, in special purpose digital logic, and any combination thereof. It should also be appreciated that more or fewer operations can be performed than shown in the figures and described herein. These operations can also be performed in a different order than those described herein. Depending upon a respective implementation, the modules 250 can be provided via a single computing device, such as data processing apparatus 102, or can be distributed via a plurality of computing devices, such as computing devices 102 and/or 104. Moreover, one module can be provided via a single computing device (102 or 104), or can be distributed across a plurality of computing devices.

[0042] Included in the arrangement shown in Fig. 2B, various modules are

programmatically tied by data processing apparatus 102 and/or user computing device 104. In the example shown in Fig. 2B is tracking module 252 that includes functionality for detecting acceleration and movement in space, and/or for providing and/or processing in association with such movement. The tracking module 152 can transform data collected as a resulting movement of the computing device 104, as well as other "passive" information, such as timestamp information, into new information. The time stamp can function as a key for coordinating telemetry information and image/sound content. Information, such as the timestamp or information transformed by tracking module 152 can be embedded into the stream for use in generating and/or viewing coordinated presentations. Chroma key module 254 includes functionality for users to select one or more pixels and define one or more colors, degrees or a range of color or other input that is used as a basis for defining portions of a frame to be removed or replaced by background content (e.g., green screen

functionality). Also illustrated in Fig. 2B is background adjustment module 256 that is configured to modify the appearance of background content, such as to move the background content or change the angle of view in accordance with telemetry information received and processed as a function of tracking module 252. These modules can include tools (e.g., class files, packages or other program resources) that enable software developers to interface with one or more hardware and software systems and to develop and/or access custom video software applications, such as shown and described herein. Further, 360 degree module 258 is configured to display up to a 360° view of content displayed on computing device 102/104, including in response to information received and processed as a function of tracking module 252. For example, the content includes a "stitched" image file having a 360 degree field of view horizontally, and further includes a 180 degree field of view degree. The present application can include interactive capability for providing the content in a 360-degree view.

[0043] Continuing with reference to Fig. 2B, Audio/Visual input module 260 includes functionality to interface with audio/visual devices, such as configured to interface with data processing apparatus 102 and/or user computing device 104. For example, Audio/Visual input module 260 interfaces with a camera and microphone communicatively coupled to data processing apparatus 102 for use in authoring videos that can be layered with interactive content, such as shown and described herein. Such hardware can be used to capture images and sound for use in generating content that is composited with other graphic and/or sonic data.

[0044] In one or more implementations, the present application can include an effects module 262 that enables one or more processing effects on audio and/or video. Among other effects, including those shown and described herein, green screen and white screen functionality can be provided to enable virtual placement of a presenter of video content in respective environments. Other example effects processing associated with effects module 262 is shown and described below. [0045] Continuing with reference to Fig. 2B, Recommender Module 264 can be included that is configured to provide tools that interface with one or more hardware and software systems and to support content sharing and a new form of social networking as a function of accessible video files that can be layered with various interactive content, and that can be authored, distributed and played in accordance with the teachings herein. Further,

Advertisement Module 266 can be included that is usable to interface with one or more hardware and software systems to provide advertisements in particular context and times.

[0046] Fig. 3 illustrates an example interface for authoring a coordinated presentation and that shows the effect of green screen functionality provided in accordance with one or more implementations of the present application. As shown in Fig. 3, green screen portion 302 is shown in connection with an image file of a chain linked fence behind a foreground element, e.g., the people who are being photographed, and thereafter combined in the coordinated presentation.

[0047] As used herein a "presenter" refers, generally, to a person, organization and/or computing device associated therewith that makes and/or distributes coordinated

presentations for others to consume. The present application can operate to provide functionality for substantially real-time compositing, which can include capturing of whatever is being displayed in the video frame, including from the camera or other content source, as well as capturing meta-data that is around or otherwise associated with one or more elements of the content, which is usable to report to a player device at a later point in time. Meta-data can include, for example, XML data. In this way, when a user interacts with a coordinated presentation, hypertext markup language ("HTML") (or other suitable content) may be served from the host server and used as a function in the coordinated presentation. This enables implementation of a fully (or at least partially) interactive platform.

[0048] During development of a new coordinated presentation, the present application provides a plurality of templates (also referred to generally as "Themes") for quickly authoring a coordinated presentation (e.g., a "TouchCast") that is configured with one or more features associated with a particular style or subject matter. For example, options are available for authoring a new coordinated presentation, which may be in a default configuration and absent of any particular theme, as well as for authoring a new coordinated presentation using or based on a particular theme (e.g., a newscast). Moreover, data processing apparatus 102 and/or user computing device 104 can provide one or more options for "Green Screen" functionality in the authoring tool for authoring a coordinated presentation. As noted herein, options are provided for a user to substitute a virtual background in a coordinated presentation in one or more locations that are defined as a function of color (e.g., a "chroma key"). In one or more implementations, a graphical user interface associated with the authoring tool is configured with a color dropper control that, when selected by the user, enables the user to define a chroma key value, such as from one or more pixels, which represents one or more suitable portions of the background of the coordinated presentation to be replaced by other content, such as an image, video, as well as HTML-based or other content. Using the controls provided, for example, in a display screen enables an extremely flexible way to define custom green screen functionality for coordinated presentations. Other controls can be included, such as sensitivity and smoothing slider controls, which operate to impact the relative smoothness and impacts of green-screen content. Such controls can reduce the impact of light reflecting from a chroma (e.g., green) screen onto a subject, which can impact realism and better integrate the subject into the background content. Further, a sound effects control can be included that enables soundboard functionality which can be selected during recording of a coordinated presentation. For example, a car horn, cymbal crash or virtually any other sound effect can be added to a coordinated presentation.

[0049] Figs. 4A and 4B illustrate two video frames 400 and 401 of an example coordinated presentation and that represent changes in the foreground (photographed) element 402 which, in Figs. 4A and 4B, is represented as a human stick figure and in solid lines. The example shown in Figs. 4A and 4B, the foreground element 402 appears to have moved from one portion of the frame to another portion. The change could have occurred for various reasons, such as a result of camera and/or computing device movement, which can be tracked as a function of tracking module 252, and which generates new information as a result of such tracking. In Figs. 4A and 4B, the virtual background elements 404 are represented as trees and in broken lines. In the example shown in Fig. 4A, the background areas 404 are static (e.g., not moving) and, accordingly, the frame 401 does not appear realistic because the movement of the foreground area 402 (or the movement of the angle of view of foreground area 402), such as due to camera panning, is not represented in the background areas 404. On the other hand, the frame 401 shown in Fig. 4B appears more realistic due to appropriate repositioning of the background elements 404 in view of the movement of the foreground area 402 through use of the new information from the tracking module. This eliminates a need for a viewer of the coordinated presentation to suspend disbelief, as the background elements are modified substantially in real-time in response to movements associated with the foreground element 402, the camera 105, computing device 102/104 and/or other virtual or physical element.

[0050] In one or more implementations, information is processed by tracking module 252, including telemetry information associated with an accelerometer and/or gyroscope and associated with image frames in video. This can be processed to affect a selection of a respective chroma key, to select content associated with a chroma key, and/or as well as for the relative placement of a subject in the frame, such as a person being photographed during production of a coordinated presentation. The placement of the subject in the frame, as well as the appearance of background content that is superimposed (e.g., as a function of the chroma key) can be modified and adjusted in response to output from the tracking module 252, as well as in response to internal and/or external parameters, such as GPS-based information, gesture-based user input, timestamp information, and image visual effects.

[0051] Information associated with the telemetry of the camera and/or computing device 104 (including gyroscope and accelerometer 227), including as a function of tracking module 252, is processed to determine the position and/or location of a subject in the foreground, and used to control the orientation and angle of view of the background content substantially in real-time as the coordinated presentation is being viewed. Accordingly, the positioning of the elements being displayed in the coordinated composite image is optimized. In one or more implementations, the processed information is used as input to one or more algorithms that configure a processor to alter the appearance of the background content, such as by providing movement of the background content relative to movement of one or more subjects in the foreground and/or movement of an angle of view of one or more subjects in the foreground.

[0052] In one or more implementations of the present application, information associated with one or more active user events and/or inactive effects, such as zoom, virtual/optical effects (e.g., fisheye), GPS tracking information or the like can be processed to provide input, as well, and that contributes to a determination of suitable movement and/or adjustment of the background content. For example, gyroscope and/or accelerometer information captured in accordance with tracking module 252 is stored in one or more databases, such as in a data log file. The gyroscope and/or accelerometer information can be captured frame by frame, and can represent the position, orientation and/or heading associated with content in each frame. The background adjustment module processes information representing, for example, that pivoting occurs (e.g., by the camera 105) and results in corresponding pivot and/or other adjustment of the appearance of the virtual background. [0053] Furthermore, a single computing device 102/104, such as a mobile computing device configured as a tablet computer or smart phone, can be configured to provide the features shown and described herein. In an alternative implementation represented in a simple block diagram shown in Fig. 5, a rig 500 is provided that is configured with a camera 105 and computing device 102/104. Camera 105 may be a digital single lens reflex camera or other camera that is configured with an image sensor of higher quality or capacity than included with computing device 102/104, and usable to provide high-quality image content. Using rig 500, the standalone camera 105 and camera 105 and the camera that is configured and/or integrated with computing device 102/104 can both be used to photograph

simultaneously from a common reference frame and/or vantage point. Telemetry information from computing device 102/104, such as provided by tracking module 252, is processed to control the appearance of background content that has been added to the coordinated presentation, such as to affect the appearance of the angle, distance, viewpoint, or the like of the background content. Alternatively, the computing device 102/104 can begin recording at a time earlier than the camera 105 which can result in more telemetry information than photographic content being captured from the stand-alone camera 105. Using postproduction methods, such as provided in video editing software, the photographic content captured by camera 105 can be synchronized with the telemetry information captured and processed in connection with computing device 102/104, to adjust virtual background content in the coordinated presentation, accordingly.

[0054] In one or more implementations, a computing device configured in accordance with the present application processes two data streams into one single stream, substantially as illustrated in the block diagrams shown in Fig. 6. One of the two streams can be captured by a camera, such as configured or integrated with a computing device (e.g., a smart phone camera), and the other of the two streams can be generated by a computing device or originate from a data file, such as an image file (e.g., TIFF, JPG, PNG, or BMP format). As one or more elements represented in the first data stream change, such as by a movement of five pixels up and three pixels to the right, a corresponding and opposite adjustment is made to one or more background elements 404 represented in the second stream, such as to move a background element 404 five pixels down and three pixels to the left. This can generate a smooth and realistic appearance of the coordinated presentation, as a function of the background adjustment module 256. In one or more implementations, a mapping module 262 is provided that identifies a set of objects (e.g., foreground elements 402 and background elements 404), and manipulates the appearance of objects and/or set of objects. In addition to movement and relocation of one or more background elements 404, the present application supports complex manipulations, such as via background adjustment module 256, including to account for pitch, yaw, and roll, and to modify the appearance of one or more background elements 404, such as to change the perspective, the skew, lighting and shadows or the like, such as in the context of one or more respective foreground elements. For example, adjustments can be made in connection with brightness, contrast and/or saturation.

[0055] Continuing with reference to the block diagrams shown in Fig. 6, in an alternative implementation, an alternative output stream 608 can be generated, which corresponds to the alternative implementation described below with reference to Fig. 7 and in connection with a 360° file, such as an image file, a 3-D file or some other content capable of supporting a 360° display and that is configurable as a virtual background. In yet another alternative, video can be provided that supports an effect of a 360° environment, including a 3-D environment, that the author of a coordinated presentation can explore ( e.g., walk through, display to a viewer, or the like). Thus, in addition to 360° image files, video and 3-D files as possible background environments. The second alternative data stream 608 can be manipulable by the viewer during playback of the coordinated presentation. For example, as the viewer of the coordinated presentation moves his or her computing device 104, telemetry information is accessed by computing device 104 and the 360° virtual background image can pan accordingly.

[0056] The present application supports a virtual background provided in a coordinated presentation that can include content that extends beyond the viewing area of computing device 102/104. An example representation is shown in Fig. 7, which includes viewing area 702 that includes a foreground element 402 and two background elements 404. In the example shown in Fig. 7, the background comprises a 360° panoramic image and the presenter may wish to enable the viewer to see all 360° of the image within the context of one or more foreground elements 402. For example, a presenter may provide a coordinated presentation associated with an outdoor landscape and holds the computing device 104 that is integrated with camera 105 in front of her face and turns 360°. The virtual background elements 404 in the coordinated presentation appear to display the 360° view of the outdoor landscape behind the presenter, as telemetry information from the computing device 104 and camera 105 is processed by computing device 102 and/or 104.

[0057] Thus, in one or more implementations, such virtual 360° panning can be provided in the coordinated presentation and can respond to movement associated with the presenter. In one or more alternative implementations, such virtual 360° panning can be provided in the coordinated presentation in response to movement by the viewer of the coordinated presentation, for example on a tablet computing device 104. In the former case, the 360° virtual background may comprise a series of individual image frames that are "stitched" in context as information associated with the presenter's device telemetry is referenced, such as by adjustment module 256 and during generation of the coordinated presentation. In such implementations, the information associated with the presenter's computing device telemetry is usable to adjust virtual background elements 404, substantially as described herein, and that maintains a high degree of realism due to the relative adjustments of the background elements 404 to the foreground elements 402. Alternatively in the latter case, as panning occurs of the virtual background 360° image as the viewer of the coordinated presentation moves his or her computing device 104, although such background elements 404 adjustment relative to the foreground element(s) 402 may not occur due to a lack of telemetry information being available to mapping module 262 and/or background adjustment module 256. [0058] In yet another alternative implementation, the viewer's computing device 104 can be configured to adjust the virtual background 360° image, such as shown and described herein, as a function of substantially real-time compiling of a coordinated presentation and elements 402, 404 thereof. As a viewer of a coordinated presentation can manipulate his or her computing device 104, telemetry information associated with a gyroscope and/or accelerometer configured with the device 104, for example, can be processed to modify the appearance of one or more virtual background elements 404, and/or the entire virtual background itself.

[0059] In one or more implementations, the present application supports integrating content comprising HTML, such as a body of an Internet web page, as background in connection with green screen functionality. For example, during the authoring process and after selecting a control selecting for selecting green screen functionality, e.g., via chroma key module 254, an author can select a respective Internet web page to replace content. Fig. 8 illustrates an example implementation in which an interactive web page 802 is provided as a background content during authoring of a coordinated presentation. In addition to the web page 802 being chosen for the background content to replace any green-colored content, a user overlay option, referred generally herein as "silhouette," background author 804 can be virtually placed within the background content (e.g., web page 802), itself. Positioning of the author 804 within the background content can be affected using standard selection and movement gestures, such as swiping, pinching or the like. In addition to author 804 being provided as the foreground element, the author 804 can be switched as the background element. Accordingly, a vApp (e.g., web page 802) effectively becomes the background content, while maintain its interactivity, for example, for author and viewer.

[0060] Fig. 9 illustrates another example implementation of an authoring process in accordance with the present application, in which an interactive webpage 802 is provided as background content with the author 804 integrated therewith. In the example shown in Fig. 9, a social network content feed from a website 802, and is displayed as background content and integrated with author 804.

[0061] Turning now to Fig. 10, a flow diagram is described showing a routine S100 that illustrates a broad aspect of a method for authoring a coordinated presentation 304 in accordance with at least one implementation disclosed herein. It should be appreciated that several of the logical operations described herein are implemented (1) as a sequence of computer implemented acts or program modules running on computing device 102/104 and/or (2) as interconnected machine logic circuits or circuit modules within computing device 102/104. The implementation is a matter of choice, dependent for example on the requirements of the device (e.g., size, mobility, energy, consumption, performance, etc.). Accordingly, the logical operations described herein are referred to variously as operations, steps, structural devices, acts, or modules. As referenced above, various ones of these operations, steps, structural devices, acts and modules can be implemented in software, in firmware, in special purpose digital logic, and any combination thereof. It should also be appreciated that more or fewer operations can be performed than shown in the figures and described herein. These operations can also be performed in a different order than those described herein.

[0062] At step S102, the process starts, for the example, as an app launching on a tablet computing device. At step SI 04, options in the form of graphical screen controls are provided to a user for authoring a coordinated presentation 304, for example, in the form of camera controls, which are selectable and that cause one or more modules to execute instructions associated with the respective control to capture a first data stream that includes content from a camera configured or integrated with the user's tablet computing device. At step SI 06, options are provided for the user to select background image content, such as content that is stored locally on the user's computing device 102 and/or 104, or content that is available via a data communication network. One or more modules execute instructions associated with a respective selected control and to capture a second data stream, which includes the background image content.

[0063] Continuing with the example process S100 shown in Fig. 8, at step S108, telemetry information associated with the user's tablet computing device (e.g., from a gyroscope and/or accelerometer) is accessed and stored. In one or more implementations, the information is stored in a data log file, such as in XML or other suitable format. At step SI 10, the stored telemetry information is processed and at least a portion of the virtual background image is modified to correspond with the processed telemetry information and in association with a respective image frame in accordance with the content from the camera. At step SI 12, a composited video frame is generated that includes at least some of the content from the camera and at least some of the modified virtual background. At step SI 14, the video frame is integrated with a plurality of other video frames and output in a select format as a coordinated presentation, which is distributed in step SI 16, thereafter the process ends.

[0064] Thus, the present application provides significant flexibility and creativity in connection with creating and viewing coordinated presentations. Although many of the examples shown and described herein regard distribution of coordinated presentations to a plurality of users, the invention is not so limited. Although illustrated embodiments of the present invention have been shown and described, it should be understood that various changes, substitutions, and alterations can be made by one of ordinary skill in the art without departing from the scope of the present invention.

Claims

What is Claimed is:

1. A system for generating a coordinated presentation, comprising: a computing device having a processor and a memory, the computing device being configured by code stored in the memory and executed by the processor to:

curate content selected by a user for inclusion in the coordinated presentation; capture, at least via a camera, a plurality of images for inclusion in the coordinated presentation;

receive a first data stream associated with the curated content and a second data stream associated with the captured images;

output a first arrangement of at least some content associated with the first and second data streams;

process telemetry information associated with at least one of a gyroscope and accelerometer;

modify at least some of the curated content in accordance with at least some of the telemetry information and/or at least one of the plurality of images comprised in the second data stream; and

integrate respective content associated with the two data streams to generate the coordinated presentation that is capable of transmission to and receipt by one or more remote devices, and wherein the coordinated presentation is configured to enable interaction with at least a portion of the curated content at each of the remote devices.

2. The system of claim 1, wherein the telemetry information represents movement of the computing device associated with distance, speed and direction in at least one of an X-axis, a Y-axis and a Z-axis.

3. The system of claim 1, wherein the computing device is further configured by code stored in the memory and executed by the processor to alter placement and/or alter an angle of view of the at least some of the first data stream.

4. The system of claim 1 , wherein the computing device is further configured by code stored in the memory and executed by the processor to:

alter a position of content in the second data stream in a frame of the coordinated presentation as a function of input received in an authoring tool interface.

5. The system of claim 1, wherein the computing device is further configured by code stored in the memory and executed by the processor to:

substitute a portion of the second data stream with at least a portion of the first data stream as a function of at least a chroma value appearing in the second data stream.

6. The system of claim 1, wherein the curated content includes a 360 degree horizontal view and a 180 degree vertical view, and

wherein the computing device is further configured by code stored in the memory and executed by the processor to:

control an angle, distance and/or viewpoint appearance of the first stream of data as a function of the telemetry information.

7. The system of claim 1, wherein the angle, distance and/or viewpoint of the first stream of data are is controlled by adjusting the first stream of data to correspond oppositely to appearance of content in the second data stream.

8. The system of claim 1, wherein the curated content includes web-based content and wherein the computing device is further configured by code stored in the memory and executed by the processor to:

position content associated with second data stream to appear virtually integrated into the first data stream.

9. The system of claim 1, wherein the step of modifying at least some of the curated content includes changing perspective, skew, brightness, contrast and/or saturation.

10. The system of claim 1, wherein the telemetry information represents movement associated with a device configured with an authoring tool or a device configured with a player tool.

11. A method for generating a coordinated presentation, comprising: curating, by a computing device being configured by code stored in a memory and executed by a processor, content selected by a user for inclusion in the coordinated presentation; capturing, at least via a camera, a plurality of images for inclusion in the coordinated presentation;

receiving, by the computing device, a first data stream associated with the curated content and a second data stream associated with the captured images;

outputting, by the computing device, a first arrangement of at least some of the first and second data streams;

processing, by the computing device, telemetry information associated with at least one of a gyroscope and accelerometer;

modifying, by the computing device, at least some of the curated content in accordance with at least some of the telemetry information and/or at least one of the plurality of images comprised in the second data stream; and

integrating, by the computing device, respective content associated with the two data streams to generate the coordinated presentation that is capable of transmission to and receipt by one or more remote devices, and wherein the coordinated presentation is configured to enable interaction with at least a portion of the curated content at each of the remote devices.

12. The method of claim 11, wherein the telemetry information represents movement of the computing device associated with distance, speed and direction in at least one of an X-axis, a Y-axis and a Z-axis.

13. The method of claim 11, further comprising:

altering, by the computing device, placement and/or alter an angle of view of the at least some of the first data stream.

14. The method of claim 11, further comprising:

altering, by the computing device, a position of content in the second data stream in a frame of the coordinated presentation as a function of input received in an authoring tool interface.

15. The method of claim 11, further comprising:

substituting, by the computing device, a portion of the second data stream with at least a portion of the first data stream as a function of at least a chroma value appearing in the second data stream.

16. The method of claim 11, wherein the curated content includes a 360 degree horizontal view and a 180 degree vertical view, and further comprising:

controlling, by the computing device, an angle, distance and/or viewpoint appearance of the first stream of data as a function of the telemetry information.

17. The method of claim 11, wherein the angle, distance and/or viewpoint of the first stream of data are is controlled by adjusting the first stream of data to correspond oppositely to appearance of content in the second data stream.

18. The method of claim 11, wherein the curated content includes web-based content and, further comprising:

positioning, by the computing device, content associated with second data stream to appear virtually integrated into the first data stream.

19. The method of claim 11, wherein the step of modifying at least some of the curated content includes changing perspective, skew, brightness, contrast and/or saturation.

20. The method of claim 11, wherein the telemetry information represents movement associated with a device configured with an authoring tool or a device configured with a player tool.