TWI794512B - System and apparatus for augmented reality and method for enabling filming using a real-time display - Google Patents

Abstract

A system for real-time updates to a display based upon the location of a camera or a detected location of a human viewing the display or both is disclosed. The system enables real-time filming of an augmented reality display that reflects realistic perspective shifts. The display may be used for filming, or may be used as a “game” or informational screen in a physical location, or other applications. The system also enables the use of real-time special effects that are centered upon an actor or other human to be visualized on a display, with appropriate perspective shift for the location of the human relative to the display and the location of the camera relative to the display.

Description

System and device for augmented reality and method for shooting using a real-time display

The present invention relates to augmented reality projections for backgrounds in filmmaking and other purposes. More specifically, the present invention relates to enabling real-time capture of a projected screen while properly computing appropriate viewing angle shifts within the projected content to correspond to real-time camera movement or movement of an object relative to the screen.

Various solutions exist for so-called "green screen" shooting. Traditional green screen shooting relies on actors being shot in front of (or sometimes wrapped in) a single color. Then, in post-production, digital objects, scenes, characters, movements, and the like can be added to the scene. For example, in Ironman® and related Marvel® films, Robert Downey Jr. occasionally wears a model Iron Man suit, but also wears a green suit, which allows post-production graphic artists to use physical props and costumes to create Not necessarily feasible or adding difficult or expensive movement and animations to the Ironman suit.

The disadvantage of green screen shooting is that for actors and directors who all have to imagine the characters they are talking to, or the location they are operating in, or even the table they are sitting at The overall immersive feeling. Many times, the effort is negligible in the quality of the performance and film, but it can reduce the quality. Also, it enables the director to It is far more difficult to determine whether the scene is "correct" when shooting. Post-production processes can add unusual things or require a specific perspective that wasn't captured. Consequently, many of these types of scenes end up being corrected in reshoots. Reshoots and additional post-production add to the cost of production and the time it takes to complete a film. The rendering required for green screen shots (if they have to be corrected) can take significant time periods, depending on the length of the scene, in some cases up to days or weeks.

There are also certain filming techniques that attempt to bridge the gap by providing a rendered scene (e.g. a 3D high quality scene) on a large monitor "instead of" a green screen or providing a scene rendered for display this gap. These systems typically photograph individuals and then use computer vision techniques to detect the location, movement, and the like of those individuals. Videos of the individuals can then be digitally overlaid within the scene.

The disadvantage of these techniques is that they usually incorporate a significant delay. Individuals must be captured on film, then computer vision is applied to the video, and the individuals can then be overlaid in an existing 3D digital scene. The delay is usually a factor of seconds. In a preferred embodiment, the delay is approximately 3 to 5 frames of video. This may not sound like much, on the order of seconds, but if a character in a scene wants to react to something happening in the digital scene, it can appear to have a significant delay in the senses of a watching audience, or must be set Special cues to enable actors to respond appropriately. In general, it's easier to add such actions later in post-production, where objects in the scene are cued from the actors' feigned reactions.

There are also augmented reality systems that overlay objects within a "reality" or a video of the reality that is delivered with substantially no lag. These systems rely on trackers to monitor the location of the wearer of an augmented reality headset (most are headsets, but other forms exist) to continuously update the location of the generated objects within the real scene. Less complex systems rely only on motion trackers, while more robust systems rely on external trackers, such as with a fixed infrared spot A camera or a fixed infrared light tracking system (with a tracker on the headset) or an infrared light spot on the headset (and a fixed infrared light tracker in a known position relative to the headset). These points of infrared light may be referred to as beacons or tracking points. Still other systems rely at least in part on infrared depth mapping of a room or space, or LIDAR depth mapping of a space. Other depth mapping techniques are also known. The depth map yields the physical location of the geometry associated with a location in the sensor's field of view. These systems enable the augmented reality system to intelligently place characters or other objects in a space (eg, not inside a table or in a wall) at an appropriate distance from the augmented reality viewer. However, augmented reality systems are typically only presented to an individual viewer from the perspective of that viewer.

Virtual reality systems are similar, but fully embody a person placed in one of the alternate realities. The degree of immersion varies, and the world a user is placed in varies in quality and interactivity. But again, these systems come almost entirely from a single perspective of one user. First-person perspective, like augmented reality and virtual reality, is occasionally used in traditional film and television filming, but it is not commonly used.

In a related field, there are interactive screens where a user can interact with a screen, for example, to play a game in a mall or shopping mall or to search for a store within a mall. These screens typically incorporate limited functionality. Some functionality can physically track an individual's interaction with the screen, such as is commonly used to enable interaction with a game. But these screens generally do not react or alter themselves based on the perspective of an attempt to recreate the scene for a particular individual interacting with the screen and a given scene.

Finally, post-production special effects add lighting, objects, or other elements to actors or individuals. For example, a "lightning bolt" could be projected outward from Thor's hammer in a Marvel movie, or a laser beam could leave Iron Man's hand. However, currently there is no one in which live, real-time effects can be applied to an actor and relative to the actor's presence within the scene. One of these real-time, real-time effects systems for position adjustment.

100: system

110: camera

112:Associated Tracker

120: workstation

130: Display

142:Associated Tracker

144:Associated Tracker

150: Network

200: computing device

210: Processor

212: memory

214: Storage

216: Network interface

218: I/O interface

300: system

310: camera

312: Tracker

313: communication interface

314: Tracking system

315: communication interface

316: Media generation

320: workstation

321: communication interface

322: Position calculation

323: resource storage

324: Image generation

325: Calibration function

326: Administration/User Interface

330: display

335: communication interface

336: Image performance

342: tracker

344:Tracker

346: communication interface

347: Tracking System

348: communication interface

349: Tracking System

410: camera

430: display

434: cross hair

436:Background object

438:Background object

442: tracker

444: tracker

510: camera

530: display

536:Background object

538:Background object

542: tracker

544: tracker

560: actor

562: actor

610: camera

630: display

636:Background object

638:Background object

642: tracker

644: tracker

660: actor

662: actor

710: camera

730: display

736:Background object

738:Background object

739:Touch screen sensor

742:Tracker

744:Tracker

762: Human

805: step

810: step

820: step

825:Step

830: step

840: step

850: step

895:Step

905: Step

910: step

920: step

930: step

935: step

950: step

960: step

965:step

995: step

1005: step

1010: step

1020: Steps

1030: step

1040: step

1050: step

1095:step

1105:step

1110:step

1120: Step

1130: Step

1135:step

1150: step

1160: step

1165:step

1170:step

1175:step

1195:step

1205: step

1210: step

1220: step

1230: step

1235:step

1240: step

1250: step

1260: step

1270: step

1275:step

1295:step

FIG. 1 is a block diagram of a system for generating and capturing augmented reality displays.

FIG. 2 is a block diagram of a computing device.

3 is a functional diagram of a system for generating and capturing augmented reality displays.

Figure 4 is a functional diagram of the calibration of a system for generating and capturing augmented reality displays.

5 is a functional diagram of camera position tracking of a system for generating and capturing augmented reality displays.

6 is a functional diagram of camera position tracking while moving for a system for generating and capturing augmented reality displays.

7 is a functional diagram of human position tracking of a system for dynamically updating an augmented reality screen for interaction with a viewer as the human moves.

Figure 8 is a flowchart of a procedure for camera and display calibration.

Fig. 9 is a flowchart of a procedure for location tracking.

FIG. 10 is a flowchart of a routine for computing camera positions during position tracking.

Figure 11 is a flowchart of a program for human position tracking.

FIG. 12 is a flow chart of a procedure for human position tracking and AR object overlay combined with humans.

Throughout this description, elements appearing in the figures are assigned three-digit reference designators character, where the most significant digit is the drawing number and the two least significant digits are part-specific. It is presumed that an element not described in connection with a figure has the same properties and functions as a previously described element having a reference designator with the same least significant digit.

Related Application Information

This patent claims priority to U.S. Provisional Patent Application No. 62/685,386 filed on June 15, 2018 and entitled "AUGMENTED REALITY BACKGROUND FOR USE IN MOTION PICTURE FILMING".

This patent claims priority to U.S. Provisional Patent Application No. 62/685,388 filed on June 15, 2018 and entitled "AUGMENTED REALITY WALL WITH VIEWER TRACKING AND INTERACTION".

This patent claims priority to U.S. Provisional Patent Application No. 62/685,390 filed on June 15, 2018 and entitled "AUGMENTED REALITY WALL WITH COMBINED VIEWER AND CAMERA TRACKING".

This patent is also a continuation-in-part of U.S. Nonprovisional Patent Application No. 16/210,951, filed on December 5, 2018, and entitled "AUGMENTED REALITY BACKGROUND FOR USE IN LIVE-ACTION MOTION PICTURE FILMING," the U.S. The non-provisional patent application claims priority to U.S. Provisional Patent Application No. 62/595,427 filed on December 6, 2017 and entitled "AUGMENTED REALITY BACKGROUND FOR USE IN LIVE-ACTION MOTION PICTURE FILMING".

The disclosures of each of these applications are incorporated by reference.

Description of the device

Referring now to FIG. 1 , which is a block diagram of a system 100 for generating and capturing augmented reality displays. System 100 includes a camera 110 , an associated tracker 112 , a workstation 120 , a display 130 , associated trackers 142 and 144 , all interconnected by a network 150 .

Camera 110 is preferably a digital film camera, such as those from RED® or other high-end cameras used to capture video content for theatrical release or programmatic distribution for television. Increasingly, digital cameras for consumers are almost as good as these professional grade cameras. Thus, in some cases, lower end video cameras manufactured primarily for capturing still images or movies for home or online consumption may also be used. The camera is preferably multi-digit, but in some cases an actual conventional film camera may be used in conjunction with the display 130, as discussed below. The camera may be or incorporate one of the computing devices discussed below with reference to FIG. 2 .

Camera 110 is incorporated into or attached to a tracker 112 . The physical relationship between the camera 110 and the tracker is such that the position of the tracker relative to the lens (or more precisely, the focal point of the lens) is or can be known. This known distance and relationship allows the overall system to derive an appropriate viewing angle by extrapolating from the viewpoint of the camera lens based on a tracker that is not at the precise viewpoint of the camera lens.

Thus, for example, the tracker 112 may incorporate an infrared LED (or LED array) having a known configuration such that an infrared camera can detect the infrared LED(s) and Thereby a very accurate distance, position and orientation relative to one of the infrared cameras is derived. Other trackers could be fiducial markers, visible LEDs (or other lights), physical properties such as shape or computer-visible images. Tracker 112 may be a so-called inside-out tracker, where tracker 112 is a camera that tracks external LEDs or other markers. Various tracking schemes are known, and essentially any of them can be employed in current systems.

In this document, the term "tracker" is used to refer generally to a component for performing position and orientation tracking. Trackers 142 and 144 discussed below may be counterparts to tracker 112 discussed here. Trackers typically have at least one stationary "tracker" and one mobile "tracker". Use fixed tracker(s) in order to accurately track the location of the mobile tracker. But which of the fixed tracker and the mobile tracker actually does the tracking (eg, noticing movement) varies from system to system. Thus, as used herein, not specifically relevant, is that the camera A set of infrared LED lights is preferably employed which is tracked by a pair of infrared cameras which thereby derive the relative positions of the infrared LED lights (attached to camera 110).

In fact, camera 110 could be multiple cameras, but only one camera 110 is shown. In some cases, for example, camera 110 may be mounted within, behind, or in a known position relative to displays and/or trackers 142 and 144 . Such a camera can be used to track the position of an individual in front of the display 130 . For example, the system may be operable to shift the perspective of a scene or series of images displayed on display 130 in response to positional information detected from a human (e.g., a human head) in front of display 130 viewing content on display 130 Instead of tracking the camera 110 itself. In this case, display 130 may operate less as a background for capturing content, and as an interactive display suitable for operation as a "game" or for presenting other content to a human viewer.

To enable this interaction, camera 110 may be or include an infrared camera coupled with an infrared illuminator or a LIDAR or an RGB camera coupled with suitable programming for tracking a human's face or head. As discussed herein, in such cases where a human being is tracked rather than a camera (like camera 110), more The scene newly rendered on the monitor. In other contexts, discussed more fully below, both humans and an associated camera (like camera 110 ) can be tracked so that camera 100 can capture an augmented reality background and create an augmented reality for individuals only visible on display 130 . Augmented reality augmentation (discussed below). While tracking of camera 110 is important, it may not be used in some particular implementations, or it may be used in conjunction with human tracking in other particular implementations. These are discussed more fully below.

The workstation 120 is the computing device responsible for calculating the position of the camera relative to the display 130 using the trackers 112 , 142 , 144 , discussed below with reference to FIG. 2 . Workstation 120 may be a personal computer or a workstation class of computer that incorporates a relatively high-end processor designed for video game world/virtual reality rendering or for graphics processing (such as designed for rendering for computer-aided design (CAD) or one of the three-dimensional graphics of three-dimensional performance film production computer). These types of computing devices may incorporate specially designed dedicated hardware such as one or more graphics processing units (GPUs), and incorporate graphics processing designed for vectors, shading, ray tracing, applying textures and instruction sets for other capabilities. GPUs typically employ memory that is faster than that of a general-purpose central processing unit, and an instruction set better tailored to the type of mathematical processing that graphics processing routinely requires.

Workstation 120 interacts with at least tracker 112 , trackers 142 , 144 and with display 130 using a network (or other computing system). Workstation 120 may also communicate with camera 110 that is capturing live action data. Alternatively, the camera 110 may store the data it captures on its own system (e.g., a storage capability native to or plugged into the camera 110), or on another remote system (a live digital video storage system), or both .

Display 130 is a large display screen or a display screen capable of filling a scene as a background for filming live action actors in front of the display. A typical display can be as large as About 20 to 25 feet wide by 15 to 20 feet high. Although various aspect ratios can be used, and screens of different sizes (eg, to fill a window of an actual physical set or to fill an entire wall of a warehouse-sized building) are feasible. Although shown as a two-dimensional display, display 130 may be designed to function as a hemispherical or nearly hemispherical "dome" on which a scene completely surrounding actors and a shooting camera may be displayed. The use of hemispheres enables more dynamic captures involving live actors in a fully realized scene, where the camera captures the scene from different angles simultaneously.

Display 130 may be a single large LED or LCD or other format display, such as those used in conjunction with large screens at sporting events. Display 130 may be incorporated as one of many smaller displays placed close to each other so that there are no empty spaces or gaps. Display 130 may be a projector projected onto a screen. Various forms of display 130 may be used.

Display 130 displays a scene (or more than one scene) and any objects therein from the perspective of camera 110 (or a person, discussed below) behind or in conjunction with any live actor operating in front of display 130 . As the camera moves across the tracker's field of view, the workstation 120 can use the trackers 112, 142, 144 to derive the appropriate perspective in real time.

Trackers 142, 144 are trackers oriented in a known relationship to display 130 (discussed above). In a typical setup, two trackers 142 , 144 are employed, each tracker being in a known relationship to a top corner of the display 130 . As can be appreciated, additional or fewer trackers may be employed depending on the overall system configuration. The known relationship of the tracker(s) 142, 144 to the display 130 is used to determine the full range of sizes of the display 130 and derived for the camera 110 on the display based on the positions provided by the trackers 112, 142, 144 and calculated by the workstation 120 Appropriate viewing angles shown on 130. Trackers 112, 142, 144 may be or include as referenced below Figure 2 discusses a computing device.

Network 150 is a computer network that may include the Internet, but may also include other connectivity systems such as Ethernet, wireless Internet, Bluetooth®, and other communication types. For some aspects of network 150, serial and parallel connections, such as USB®, may also be used. Network 150 enables communication between the various components that make up system 100 .

Turning now to FIG. 2 , there is shown a block diagram of a computing device 200 representing camera 110 (in some cases), workstation 120 , and trackers 112 , 142 and 144 (as appropriate) in FIG. 1 . The computing device 200 may be, for example, a desktop or laptop computer, a server computer, a tablet computer, a smart phone, or other mobile devices. Computing device 200 may include software and/or hardware for providing the functionality and features described herein. Therefore, the computing device 200 may include one or more of the following: logic arrays, memory, analog circuits, digital circuits, software, firmware, and processors. The hardware and firmware components of computing device 200 may include various special purpose units, circuits, software and interfaces for providing the functionality and features described herein.

The computing device 200 has a processor 210 coupled to a memory 212 , storage 214 , a network interface 216 and an I/O interface 218 . Processor 210 may be or include one or more microprocessors, special purpose processors for specific functions, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), programmable logic devices (PLDs) ) and Programmable Logic Array (PLA).

Memory 212 may be or include RAM, ROM, DRAM, SRAM, and MRAM, and may include firmware, such as static data or fixed instructions, BIOS, system functions, configuration data, and during operation of computing device 200 and processor 210 Other routines to use. Memory 212 also provides a storage area for data and instructions associated with applications and data processed by processor 210 . As used herein, the term "memory" corresponds to memory 212 and clearly Transient media such as signals or waveforms are definitely excluded.

Storage 214 provides non-volatile, bulk or long-term storage of data or instructions in computing device 200 . Storage 214 may take the form of a magnetic or solid state disk, magnetic tape, CD, DVD, or other reasonably high capacity addressable or serial storage medium. Multiple storage devices may be provided or the computing device 200 may use multiple storage devices. Some of these storage devices may be external to computing device 200, such as network storage or cloud-based storage. As used herein, the terms "storage" and "storage medium" expressly exclude transitory media such as signals or waveforms. In some cases, such as those involving solid-state memory devices, memory 212 and storage 214 may be a single device.

Network interface 216 includes an interface to a network, such as network 150 (FIG. 1). Network interface 216 may be wired or wireless.

I/O interface 218 interfaces processor 210 to peripherals (not shown) such as displays, video and still cameras, microphones, keyboards, and USB® devices.

FIG. 3 is a functional diagram of a system 300 for generating and capturing augmented reality backgrounds for filming. The system 300 includes a camera 310 , a tracker 312 , a tracker 342 , a tracker 344 , a display 330 and a workstation 320 .

Camera 310, tracker 312, workstation 320, display 330, and trackers 342 and 344 each include a communication interface 315, 313, 321, 335, 346, and 348, respectively. Each of communication interfaces 315, 313, 321, 335, 346, and 348 is responsible for enabling each of the devices or components to communicate data with other devices or components. Communication interfaces 315, 313, 321, 335, 346, and 348 may be implemented in software, with some portion of their capabilities implemented using hardware.

Camera 310 also includes media generation 316 responsible for capturing media (eg, a scene) and storing the media to a storage location. The storage location can be the camera 310 (not shown) The end may be remote on a server (or several) or workstation computers (also not shown). Any typical procedure for capturing and storing digital or conventional film images can be used. However, in cases where the camera itself is incorporated into the tracker 312 , communication of data associated with tracking may be communicated using the workstation 320 . Alternatively, in some cases, visual data captured by media generation 316 of camera 310 may be used to augment tracking data provided by tracker 312 and may provide this data to workstation 320 .

Trackers 312 , 342 and 344 each include a tracking system 314 , 347 , 349 . As discussed above, a tracking system can take many forms. And one device can track another device or vice versa. A point of interest may track tracker 312 attached to camera 310 in a known relative position relative to display 330 with reference to trackers 342 , 344 . In some cases, more or fewer trackers may be used. The trackers 312 , 342 , 344 are operable to track the camera 310 but also a human being in front of the display 330 .

Display 330 includes video presentation 336 . This is a functional description intended to cover many things, including instructions for producing images on the screen, memory for those instructions, one or more frame buffers (which in some cases may be used for speed disabled) and any screen refresh system in communication with workstation 320. The display 330 displays images provided by the workstation 320 for display on the display 330 . As directed by the workstation 320 , based on the trackers 312 , 347 , 349 the image displayed on the display is updated to correspond to the current position of the camera 310 lens.

Workstation 320 includes position calculation 322 , resource storage 323 , image generation 324 , calibration function 325 and management/user interface 326 .

Position calculation 322 uses data generated by tracking systems 314, 347, 349 in each of trackers 312, 342, 344 to be based on known relationships between trackers 342, 344 and displays and between trackers 312 and camera 310 lenses while generating the camera 310 (or human or both) location information. In most typical cases, the relative distance can be used geometrically to derive the distance and height of the camera 310 (actually, the tracker 312 on the camera 310 ) relative to the display 330 . Location calculation 322 uses this data to derive a location. Details of a typical calculation are presented below with respect to FIGS. 4 and 5 .

Resource storage 323 is a storage medium, and potentially a database or data structure, that is used to store data used to generate images on display 330 . Asset storage 323 can store 3D maps of locations, associated textures and colors, any animation data, any characters (including their own 3D characters and texture and animation data), and what a director or art director wishes to incorporate into a live action Any special effects or other elements in the background. These resources are used by image generation 324 discussed below.

Image generation 324 is basically a modified video game graphics engine. Image generation 324 can be more complex, and can incorporate functions and elements not present in a video game graphics engine, but in general, video game graphics engines are designed to present a three-dimensional image to a viewer on a two-dimensional display Software of the world. The world is made up of elements stored in resource storage 323, such as described by a map file or other file format suitable for defining elements and any actions within an overall background scene. Image generation 324 may include a script language that enables image generation 324 to cause events to occur or trigger events or time events involving other resources or animations or backgrounds. The script language can be designed in a way that makes triggering events relatively simple for a non-computer savvy person. Alternatively, a technical director may be employed to ensure smooth operation of command files.

The calibration function 325 operates to set a baseline position of the camera 310 and a baseline characteristic of the display 330 . Initially, image generation 324 and position calculation 322 do not determine the actual size and dimensions of the display. Typically the system 300 must be calibrated. There are various ways of calibrating a system as such. For example, a user can hold the trackers at the corners of the display and make changes to the software. In which corner is one of the "remarks". This is time consuming and especially user-friendly. Film crews will hate this cumbersome setup procedure every time a scene changes.

An alternative procedure involves just setting the tracker with a known position relative to the top (two display corners for any size display). Image generation 324 may then be instructed to enter a calibration mode and display a crosshair or other symbol on the center of display 330 . The tracker 312 can then be held in the center of the display and at that location mentioned in the software. By finding the exact center (or close enough within tolerance), the calibration function 325 can extrapolate the full size of the display. The three points of tracker 342, tracker 344, and tracker 312 define a plane so that calibration function 325 can determine the angle and placement of the display plane. Also, the distance from the center of the display to the upper left corner is the same as the distance from the center of the display to the lower right corner. The same is true for relative corners. Accordingly, the calibration function 325 can determine the full size of the display. Once these two components are known, the display can be defined in terms that can be easily translated to conventional video game engines and to image generation 324 .

Management/user interface 326 may be a more traditional user interface of display 330 or a stand-alone display of workstation 320 . The management/user interface 326 may enable the administrator of the system 300 to set certain settings, switch between different scenes, cause actions to occur, design and trigger script actions, add or remove objects or background characters, or recreate Start the system. Other functions are also possible.

Figure 4 is a functional diagram of the calibration of a system for generating and capturing augmented reality backgrounds for filming. FIG. 4 includes a camera 410 (associated tracker not shown), a display 430 , trackers 442 , 444 . The display 430 incorporates various background objects 436,438.

Camera 410 is brought close to reticle 434 shown on display 430 . The distance from trackers 442 and 444 can be noted by the system. As discussed above with respect to Figure 3, with photographic The camera is moved away from the display 430, which enables the calibration to take into account the position of the display relative to the camera 410 as a two-dimensional plane. Also, the center position enables the system to determine the overall height and width of the display 430 without manual input from a user. If anything goes wrong, recalibrating is relatively simple, just re-entering calibration mode and putting the camera 410 back at the reticle 434 .

In the case of human tracking, calibration can be avoided entirely by knowing the absolute position of the human tracking camera 410 (or other sensor) relative to the display 430 itself. In such cases, calibration may not be required at all, at least for cameras or other sensors that track the user's position relative to the display 430 .

5 is a functional diagram of camera position tracking of a system for generating and capturing augmented reality backgrounds for filming. Here, the same camera 510 is shown (associated tracker not shown), display 530 and trackers 542, 544. Camera 510 is shown moved away from display 530 as the system has been calibrated. Trackers 542 and 544 can calculate the distance and any direction (e.g., angle down or up from the calibration point) of their equidistant cameras and use geometry to derive the distance and angle from the lens of camera 510 to a center point of display 530 (eg, viewing angle). Common to this information is the appropriate viewing angle for the display. This point of view can be used to move in a way that convincingly simulates the effect of an individual moving into the point of view of a particular scene (e.g., as if the camera were on a person and as the person's position changes, the background changes appropriately based on that position). bit background.

As shown here, actors 562 and 560 exist in front of the screen. Background objects 536 and 538 are present, but from the perspective of camera 510 , background object 538 is "behind" actor 560 . The crosshairs shown may not be visible during a show, but are shown to show the relative position of the camera and the center of the display.

FIG. 6 is a functional diagram of camera position tracking while moving for a system for generating and capturing augmented reality backgrounds for filming. This is the same scene as that shown in Figure 5 scene, but after camera 610 has been shifted to the right relative to display 630 . Here, actors 662 and 660 have remained relatively fixed, but the position of camera 610 has changed. From the calculated angle of view of camera 610 , background object 638 has moved out "behind" actor 660 . This is because the position of the camera 610 (the viewer) has shifted to the right and objects that were slightly behind the actor 660 from that view angle have now moved out from behind the actor. In contrast, background object 636 has now moved "behind" actor 662, again based on the shift in perspective.

Tracker 642 and tracker 644 may be used by the system in conjunction with a tracker (not shown) associated with camera 610 to derive the appropriate new perspective on the fly and alter the display accordingly. As live actors operate in front of the display 630, the workstation computer (FIG. 3) can update the image displayed on the display to reflect the viewing angle appropriately. The crosshairs shown may not be visible during a show, but are shown here to demonstrate the relative position of the camera to the center of the display.

FIG. 6 is shown with only a single display 630 . There are situations where multiple displays can be used with multiple cameras to produce more than a single view (e.g. for a frame capture covering a scene), where the same view of the same scene can be captured on one or more displays or a different perspective. For example, a single display typically has a refresh rate as high as 60Hz. Film shooting is usually done at 24 frames per second, with some more modern options using 36 frames per second or 48 frames per second. Thus, a 60Hz display can reset itself up to 60 times per second, more than covering the necessary 24 frames and almost covering 36 frames.

In this case, two cameras may be used, each camera having a shutter synchronized with every other frame of the image displayed on display 430 . Using this, the trackers 442 and 444 can actually track the positions of the two cameras and the associated workstation can alternate between images intended for a first camera and images intended for a second camera. In this way, different perspectives of the same background can be captured using the same display. Polarized lenses can also be used in cameras (or one, as discussed below) to a similar effect.

Alternatively, multiple displays may be provided, one for each camera angle. In some cases, the multiple displays may be a full sphere or hemisphere in which actors or crew members are placed for filming (or humans are placed for participation in a game). In such cases, the perspective may be based on trackers fixed to cameras pointing in different directions to thereby enable the system to render the same scene from multiple perspectives so that coverage of the scene can be provided from multiple perspectives.

7 is a functional diagram of human position tracking of a system for dynamically updating an augmented reality screen for interaction with a viewer as the human moves. This is similar to the diagrams shown in FIGS. 4-6 , but here, at least one camera 710 is fixed relative to the display and tracks a human 762 . While humans are discussed here, other objects (eg, robots, horses, dogs, and the like) can be tracked and employ similar functionality. Additionally or alternatively, trackers 742 and 744 may track human 762 . These trackers 742 and 744 and camera 710 may rely on LIDAR, infrared light sensors and illuminators, RGB cameras coupled with face or eye tracking, fiducial markers, or other tracking schemes to detect one of the A human is present and the position or relative position of the human relative to the display 730 is updated.

Like the camera tracking system of FIGS. 4-6 , BG objects 736 and 738 can have their associated perspectives updated as the human 762 moves. This may be based on an estimate of the eye position of the human 762, or on the general position of the mass of the human. Using such a display 730, a virtual or augmented reality world can be "shown" to a human 762 that appears to properly track the user's movements as if it were a real window. Movement of human 762 may cause display 730 to update appropriately, including being properly occluded by BG objects 736 and 738 as human 762 moves.

In some cases, a touch screen sensor 739 may be integrated into or form part of the display 730 . The touch screen sensor 739 is described as a touch screen screen sensor, and it can be capacitive or resistive touch screen technology. However, the touch screen sensor may instead rely on motion tracking based on trackers 742 and 744 and/or camera 710 (e.g., raising an arm or pointing at display 730) to enable the "touch" functionality of display 730 sex. Using the touch screen sensor 739, interaction with the image displayed on the display 730 can be realized. Alternatively, the touch screen sensor 739 may be an individual's own mobile device, such as a tablet computer or mobile phone. In some cases, for example, a user may use their phone to interact with the display.

Description of the program

Referring now to FIG. 8, there is shown a flowchart of a procedure for camera and display calibration. The flowchart has both a beginning 805 and an end 895, but the procedure can be repeated as many times as necessary if the system is moved, deviated from calibration, or otherwise desired by a user.

After start 805 , the process begins by enabling calibration mode 810 . Here, a user or supervisor operates the workstation or other control device to enter a mode specifically designed for calibration. As discussed above, there are various options for calibration, but the options used herein are disclosed in this flowchart. In this calibration mode, once the calibration mode is enabled at 810, a crosshair or similar indicator is shown on the display.

Next, the user may be prompted at 820 to bring the tracker to the display. On-screen guidance or prompts can be provided, the display can be more complex than a reticle and can include a camera rig or an outline of a tracker to be brought to the display. In this way, the user can be prompted what to do to complete the calibration procedure.

If the tracker was not brought to the display ("NO" at 825), the user may be prompted again at 820. If the user has brought the tracker to the display (presumably in the correct position), the user can confirm the baseline position at 830 . This can be done by clicking a button, leaving the Calibration mode or through some other confirmation (do not move the camera for 10 seconds while in calibration mode).

Baseline information is then known (eg, the relative position of the tracker to the display and camera to its associated tracker and the position of the center of the display). This information can be stored at 840 .

The system can then generate the relative position and at 850 display the size. At this stage, use this data to define the plane of the display and set the size of the display.

An example of this could be a display that is 10 meters high by 15 meters wide in total. Move the camera tracker to the center point of the display, and the tracker system can detect that the camera tracker is approximately 9.01 meters away from the tracker at a specific angle. Pythagorean theorem can be used to determine that if one of the hypotenuses (the line to the center of the display) of the triangle forming 1/8 of the display area is 9.01 meters, and the two trackers on the display (the top of the display The distance between the sides) is 15 meters, and the other two sides are 7.5 meters (1/2 of the top) and 5 meters respectively. Therefore, the width of the display is 10 meters.

The routine can then end at 895.

Fig. 9 is a flowchart of a procedure for location tracking. The procedure has a start 905 and an end 995, but the procedure can occur many times and be repeated, as shown in the figure itself.

After start 905 , the process begins by performing a calibration at 910 . Calibration is described above with respect to FIG. 8 .

Once the calibration is complete, the position of the camera relative to the display can be detected at 920 . This location is detected using two distances (distances from each tracker). From then on, knowing the plane of the display itself, the relative position of the camera and the display can be detected. Tracking systems are known to perform these functions in various ways.

Next, at 930, the three-dimensional scene is displayed (eg, for use in filming). this scene Scenes that are generated by an art director or director and contain assets and other animation or script motion as the director intends. This is discussed in more detail below.

If no motion is detected ("NO" at 935), the scene remains relatively stationary and remains displayed. There may be animations (eg, wind blowing) or other baseline things happening in the scene, but the perspective of the scene remains the same.

If movement is detected (YES at 935 ), then at 950 the new position of the camera relative to the display is detected. This new position information of the camera is used at 960 to calculate a new viewing angle of the camera relative to the display.

If the process (eg, capture) is not complete ("NO" at 965), then the process continues at 930. If the process is complete ("YES" at 965), the process ends at 995.

FIG. 10 is a flowchart of a routine for computing camera positions during position tracking. The procedure begins at start 1005 and ends at 1095, but may repeat each time the camera moves after calibration.

The procedure is efficient enough that it can be done substantially in real-time so that no visible "lag" relative to the actor or camera is detected in the display. One element of achieving this "lag-free" is the definite disregard of tracking data (as opposed to position data) related to camera orientation. Specifically, tracking systems tend to provide a wealth of information not only about position (e.g., an (x,y,z) position in three-dimensional space (more generally defined as vectors from tracker(s)), And it also provides a large amount of targeted information.

The orientation data indicates the specific orientation the tracker is maintained within that location. This is because most trackers are designed for augmented reality and virtual reality tracking. The tracker is designed to track "head" and "hand". These objects require orientation data as well as position data (eg, the head is "looking up") in order to accurately provide the user with a relevant VR or AR view. in order to The data is generally irrelevant for the purpose of using these trackers in the current system. Therefore, disregard, discard or disregard any such information provided unless it is required for some other purpose. In general, it will be assumed that the camera is always facing substantially, if not actually, the display, somewhere on a parallel plane. This is because moving the camera to a different position will cause the illusion to disappear. This information may be used in situations involving dome or hemispherical settings. But relying on this data can significantly slow down processing and introduce lag. Similarly, other systems that rely on computer vision or detection introduce lag for similarly computationally intensive reasons.

The program displaying this scene is a variation of a representative scene representation over a period of time used in the background content of a three-dimensional graphic presentation for a video game, augmented reality or virtual reality environment. The mathematics used to render real-time 3D computer graphics typically consists of using a perspective projection matrix to map 3D points onto a 2D plane (display). A left perspective projection matrix is usually defined at the center as follows:

The view-volume can be shifted by rendering with an off-center view projection matrix using:

Determining the values of l, r, b, t corresponding to the camera position produces an exact view-dependent perspective shift. The method for determining the view-dependent values of l, r, b, t is as follows : First, a virtual screen to be scaled is placed in the 3D scene at the desired virtual position to represent the screen. Next, the corners of the screen are determined at 1010 using the tracker information and the calibration procedure as discussed above. Next, the left screen axis as used in conventional 3D graphics engines can be calculated. Specifically, the right, top and front (normal) unit vectors from the screen corner position to the camera may be calculated at 1020 as follows:

Next, the extent of the truncated angle of the view volume from the viewer position can be generated at 1030 by calculating the vector from the known camera position to the corner of the display.

Next, the display must be scaled appropriately to account for the camera's distance from the display. To do this, we calculate at 1040 a ratio of the distance between the camera (near plane) and the screen plane. This ratio can be used as a scale factor because the truncation angle range is specified at the near plane as follows:

Finally, using the projected camera view at 1050, the view-dependent range applies the vector and scaling ratio to the scene. To accomplish this, l, r, b, t are calculated as follows:

Computing an exact camera position is important for the trompe l'oeil effect to work correctly. To ensure this happens quickly, the trace system can be updated and executed simultaneously in a single thread (CPU core) independently of other threads to minimize latency. Other systems at the workstation (eg, the rendering itself) can read current telemetry (position, orientation) from the tracking system every frame update. Minimized motion-to-photon latency is achieved by keeping rendering execution at 60Hz or higher. In the inventor's experience, if the rendering system reads the tracking data at 60 FPS, the motion-to-photon (eg, camera-to-display on-screen) latency is approximately 16.66 milliseconds. With this low level of latency, the result is essentially or practically imperceptible to human vision and cameras.

Figure 11 is a flowchart of a program for human position tracking. The routine starts at 1105 and ends at 1195. FIG. 11 is quite similar to the trace described with reference to FIG. 9 . Also, tracking can occur in much the same way as described above. Only the differences relative to human location tracking and their relevance to associated programs will be detailed below.

As in FIG. 9 , calibration is performed at 1110 . This may not be necessary if there are no external cameras. However, an initial calibration may be required to enable accurate human tracking in front of a display to occur. This can be as simple as positively defining the camera and/or tracker position(s) relative to the display so that human tracking can occur accurately.

Once calibration has occurred at 1110, at 1120 the position of a human in front of the display can be detected. This detection can rely on infrared light, LIDAR, fiducial markers, RGB camera combined with image processing software or other similar methods. Regardless, detecting and/or generating an actual detection of a tracked human eye or eye position or an estimate of human eye position is part of this process.

This information is used in much the same way as the tracker used in FIG. 9 for camera detection, specifically, to display a three-dimensional scene at 1130 with the perspective properly linked to the human eye position. In this way, the scene on the display is rendered in such a way that it looks "correct" to the detected human.

Then, if no motion is detected (eg, by camera(s) or tracker(s)) (“NO” at 1135 ), the same scene continues to be displayed. The scene itself can be active (e.g. things can happen on the display, such as a sequence of events, a sunrise, other real or animated actors or actions, gunfire, rain, or any other number of actions), but doesn't handle perspective shifts , which is due to no movement of tracked humans.

If at 1135 movement is detected (eg, by camera(s) or tracker(s)) (“YES” at 1135 ), then at 1150 a new position of the human relative to the display is detected. Here, an updated position of the human eye or an estimated position of the human eye is generated and/or detected by camera(s) and/or tracker(s).

Next, at 1160, a new viewing angle for the human relative to the display is calculated and displayed for the human. Here, changes are altered to reflect viewing angle shifts detected by movement of a human's head, body or eyes. This happens incredibly quickly, allowing the scene to update substantially instantaneously with no discernible lag to a human viewer. In this way, the scene may appear to be a "portal" or "window" to a virtual or augmented reality world.

The system can also track interactions using "touch" sensors or virtual touch sensors, as discussed above with respect to FIG. 7 . If this "touch" is detected (this may only be an interaction in mid-air), action) (“Yes” at 1165 ), the interaction can be processed at 1170 . This processing may be updating the scene (eg, a user selects an option on the display) or interacting with someone shown on the screen (eg, shaking hands), firing a weapon, or causing some other shift in the display.

If there is no interaction ("NO" at 1165) or after any interaction has been processed at 1170, then at 1175 a determination can be made as to whether the procedure is complete. This can happen when the user has stopped in the frame of the camera or tracker(s) tracking the person and a timeout occurs or can be triggered externally by a supervisor. Other reasons for program completion ("YES" at 1175) are also possible. Thereafter, the program will end at 1195.

However, until the process is complete (“Yes” at 1175 ), the scene will continue to be displayed at 1130 , motion is detected at 1135 and the scene is updated as a whole based on the movement and interactions at 1150 to 1175 .

FIG. 12 is a flow chart of a procedure for human position tracking and AR object overlay combined with humans. The procedure starts at 1205 and ends at 1295, but can happen any number of times. As with FIG. 11 , this figure largely corresponds to FIG. 9 . Only those aspects that differ will be discussed here.

After start 1205, the routine begins with calibration 1210. Here, there may be multiple cameras and/or trackers. Thus, a camera tracking a person (if it is fixed relative to a display) may not require any calibration. However, for a camera that captures a scene using a display as an active background, calibration may be required, as described with respect to FIGS. 8 and 9 .

Next, at 1220, the location of the human is detected. Here, tracker(s) and/or camera(s) that track a human's position relative to the display determine where a human is relative to the display. This information is necessary to enable the system to place augmented reality objects convincingly relative to humans.

Next, at 1230, the position of the camera is detected. This system is superimposed on human beings In between the camera shoots the display as a background. Here, the position of the camera is calculated so that the scene shown on the display can be properly rendered.

Although this detection at 1230 is discussed with respect to a camera, it may only be the detection of a human (eg, a viewer) viewing one of the scenes. In this case, a similar human tracking system and method as discussed with respect to FIG. 11 would be employed.

Next, the three-dimensional scene is presented at 1240 using any desired AR objects. This step has at least three subcomponents. First, the perspective of the scene itself (and any associated perspective shift) must be accurately reflected on the display. This is done using the position of the camera alone.

Second, the position of the entity relative to the camera and display must be generated. This is based on the detected position of the human relative to the display, and then the detection of the relative position of the camera. These two pieces of data can be combined to determine the relative position of the camera to the human and the display.

Thereafter, one (or several) augmented reality objects must be rendered. Generally speaking, these will be selected in advance as bits of a "special effect" used in the scene being photographed. In a very basic example, an individual whose position relative to the display is known can be seen to "glow" with a bright light around its body. This glow can appear on the display, but since it is updated in real time and depends on the camera position and human position, it can appear to the camera as the scene is being recorded. Using real-time human tracking, the glow may follow the user, but at this step 1240 the glow is only indicated to "surround" the user or however the special effects artist has indicated it should happen.

Other examples of augmentation could be the self-firing of beams of light (either automatically or upon specific command, e.g., pressing a firing button, by a special effects supervisor or assistant director), above an individual's head as they walk. A halo, distinct "wings" on a human's back that appear only behind that human, and other effects that appear to come from or emanate from a tracked human can also be applied.

In this case, if no motion is detected ("NO" at 1235), the effect continues. The effect itself (eg glow or fog or pulse or beam) may have its own continuing independent animation, but it will not "move" with the human.

If movement is detected ("YES" at 1235), then at 1250 both the new position of the human and the new position of the camera are detected, and the display is updated to reflect the perspective of the scene, and to reflect the new position of the digital effect. Note that if an actor is in front of the display with the correct perspective shift for reflection of, for example, a halo over the human's head, the halo will appear "closer" to the camera than the background. Therefore, the associated viewing angle shift will be smaller for the halo (because it is closer) than for the background.

New view data for the camera and AR object(s) is calculated at 1260 and updated to the new view for the display at 1270 .

At 1275 a determination is made whether the program is complete. If it is not complete ("NO" at 1275), the process continues at 1240 with the display of the three-dimensional scene and any AR objects(s). If the process is complete ("YES" at 1275), then the process ends at end 1295.

end comment

Throughout this description, the embodiments and examples shown should be considered as examples, rather than limitations of the devices or processes disclosed or claimed. Although many of the examples presented herein involve specific combinations of method acts or system elements, it should be understood that such acts and such elements can be combined in other ways to accomplish the same goal. With respect to the flowcharts, additional and fewer steps may be taken, and steps as shown combined or further refined to achieve the methods described herein. Acts, elements and characteristics discussed in connection with only one embodiment are not intended to be excluded from a similar role in other embodiments.

As used herein, "plural" means two or more. As used herein, a "set" of items may comprise one or more of these items. As used herein, regardless of the In the written description or the scope of the patent application, the terms "comprising", "comprising", "with", "having", "containing", "involving" and the like should be understood as open-ended, that is, meaning including but not limited to. Only the transitional phrases "consisting of" and "consisting essentially of" should be closed or semi-closed transitional phrases respectively relative to the patent scope of the invention. Using ordinal numbers such as "first", "second", "third", etc. in a claim to modify a claim element does not by itself mean that one claim element is superior to another claim element Any priority, priority, or sequence of scope elements or chronological order of acts for performing a method is used only to distinguish a claimed scope element bearing a particular name from one having the same name (but using sequential terms) The other element is marked to distinguish the elements within the patent scope of the invention application. As used herein, "and/or" means that the listed items are alternatives, but alternatives also include any combination of the listed items.

100: system

110: camera

112:Associated Tracker

120: workstation

130: Display

142:Associated Tracker

144:Associated Tracker

150: Network

Claims (20)

A system for augmented reality, comprising: a display for displaying images generated by a computing device in communication with the display; a sensor for detecting a position of the display relative to which the camera captures an image of the display from a movable position; the computing device communicates with the display for: the position of the display to display an image on the display so as to correspond to a first perspective of an object shown in the image; and based on the position relative to the display determined using the sensor The image displayed on the display is continuously adjusted once the current position is updated to correspond to additional viewing angles applicable to the objects displayed in the image; and the camera simultaneously captures images contained in the display a composite image of a human, a figure, or a physical object ahead and the image displayed on the display based on the location and the updated current location and display the first viewing angle and the additional viewing angles. The system according to claim 1, further comprising: a second display for displaying an image generated by a computing device communicating with the second display; The computing device communicates with the second display for: displaying a second image on the second display based on the position relative to the second display determined using the sensor so as to correspond to a second image displayed on the image and continuously adjust the second image displayed on the second display based on the updated current position relative to the second display determined using the sensor to correspond to the applicable further additional viewing angles of the objects displayed in the second image; and based on the position and the updated current position, the camera further captures the second image displayed on the second display, the second display The second image above shows the second viewing angle and the further additional viewing angles. The system of claim 1, further comprising: a second sensor for detecting a second position of a second display relative to a second camera; An image generated by the computing device in communication with a second display; the computing device communicates with the second display for: based on the second position relative to the second display determined using the second sensor, on the second display displaying a second image on the second display so as to correspond to a second viewing angle of an object displayed in the image; and based on an updated second current position relative to the second display determined using the second sensor to continuously adjusting the second image displayed on the second display to correspond to further additional views applicable to the objects displayed in the second image and based on the second position and the updated second current position, the second camera captures the second image displayed on the second display, the second image on the second display showing the first Second viewing angles and such further additional viewing angles. The system of claim 1, wherein the position is a position of the camera viewing the display. The system of claim 1, further comprising: a tracker (tracker), which is used to track the movement of a human participant (participant); the viewing angle of the position to alter (altering) the appearance (appearance) of the image displayed on the display to incorporate elements responsive to the human participant's action (action) to continuously adjust the image displayed on the display image. The system of claim 5, wherein the elements positioned relative to the human participant maintain a fixed visual relative to a particular portion of a body of the human participant as the human participant moves in front of the display )element. The system according to claim 1, further comprising: at least one touch sensor for realizing physical interaction with the display; Wherein the computing device is further configured to receive input from a user from the at least one touch sensor; and alter the image displayed on the display in response to the input. An apparatus for augmented reality comprising a non-volatile machine-readable medium storing a program having instructions which, when executed by a processor, cause the processor to: use a sensor , detecting a position of a display relative to a camera used to capture an image of the display from a movable position relative to the display; based on the position relative to the display determined using the sensor to display an image on the display so as to correspond to a first viewing angle of an object displayed in the image; and continuously adjust the display on the display based on an updated current position relative to the display determined using the sensor using the camera to simultaneously capture a composite image containing a person, a figure, or a physical object in front of the display and the image displayed on the display, the image showing the first viewing angle and the additional viewing angles based on the location and the updated current location. The apparatus of claim 8, wherein the instructions further cause the processor to: display a second image on the second display based on the position relative to a second display determined using the sensor so as to correspond to a second viewing angle of the object displayed in the image; and the updated current position relative to the second display based on the determination using the sensor configured to continuously adjust the second image displayed on the second display to correspond to further additional viewing angles applicable to the objects displayed in the second image; and based on the position and the updated current position, using The camera captures the second image displayed on the second display showing the second viewing angle and the further additional viewing angles. The apparatus of claim 8, wherein the instructions further cause the processor to: use a second sensor to detect a second position of a second display relative to a second camera; displaying a second image on the second display at the second position relative to the second display determined by the detector so as to correspond to a second viewing angle of the object displayed in the image; and based on using the second an updated second current position relative to the second display determined by the sensor to continuously adjust the second image displayed on the second display to correspond to the objects suitable for display in the second image and using the second camera to capture the second image displayed on the second display, the second image showing the second angle of view based on the second position and the updated second current position and such further additional perspectives. The apparatus of claim 8, wherein the position is a position of the camera viewing the display. The apparatus of claim 8, wherein the instructions further cause the processor to: use a tracker to track movement of a human viewer relative to the display; and by based on one of the positions from the camera relative to the display The viewing angle is to change the appearance of the image displayed on the display to incorporate elements that respond to the actions of the human participant to continuously adjust the image displayed on the display. The apparatus of claim 12, wherein the elements positioned relative to the human participant are visual elements that remain fixed relative to a particular part of a body of the human participant as the human participant moves in front of the display. The apparatus of claim 8, wherein the instructions further cause the processor to: enable physical interaction with the display using at least one touch sensor; receive input from a user from the at least one touch sensor; and The image displayed on the display is altered in response to the input. The device according to claim 8, further comprising: the processor: a memory; wherein the processor and the memory include circuits and software for executing the instructions on the storage medium. A method for filming using a real-time display, the method comprising: Using a sensor, detecting a position of the display relative to a camera used to capture an image of the display from a movable position relative to the display; based on a relative determination using the sensor displaying an image on the display at the position of the display so as to correspond to a first viewing angle of an object displayed in the image; and based on an updated current position relative to the display determined using the sensor to continuously adjusting the image displayed on the display so as to correspond to additional viewing angles applicable to the objects displayed in the image; and using the camera to simultaneously capture a person, a figure or a person contained in front of the display A composite image of the physical object and the image displayed on the display, the image based on the location and the updated current location to show the first viewing angle and the additional viewing angles. The method of claim 16, further comprising: displaying a second image on a second display based on the position relative to a second display determined using the sensor so as to correspond to the position displayed in the image a second viewing angle of the object; and continuously adjusting the second image displayed on the second display based on the updated current position relative to the second display determined using the sensor so as to correspond to an image suitable for displaying further additional viewing angles of the objects in the second image; and based on the position and the updated current position, using the camera to capture the second image displayed on the second display, on the second display The second image shows the second viewing angle and the further additional viewing angles. The method of claim 16, further comprising: using a second sensor to detect a second position of a second display relative to a second camera; based on the second position relative to the display determined using the second sensor to displaying a second image on the second display so as to correspond to a second viewing angle of an object displayed in the image; and based on an updated second current position relative to the second display determined using the second sensor to continuously adjusting the second image displayed on the second display so as to correspond to further additional viewing angles applicable to the objects displayed in the second image; and based on the second position and the updated second current position , using the second camera to capture the second image displayed on the second display, the second image showing the second viewing angle and the further additional viewing angles. The method of claim 16, wherein the position is a position of the camera viewing the display. The method of claim 16, further comprising: using a tracker to track movement of a human viewer relative to the display; and altering the view displayed on the The appearance of the image on the display continuously adjusts the image displayed on the display to incorporate elements responsive to the actions of the human participant.

Images