WO2013011121A1

WO2013011121A1 - Augmented reality method and device

Info

Publication number: WO2013011121A1
Application number: PCT/EP2012/064280
Authority: WO
Inventors: Vincent Toubiana; Mathieu Boussard
Original assignee: Alcatel Lucent
Priority date: 2011-07-20
Filing date: 2012-07-20
Publication date: 2013-01-24
Also published as: FR2978260A1

Abstract

An augmented reality method on a terminal, particularly a mobile terminal, comprising the sending of a description of at least one object to an application, this description including a semantic description of the object's properties, which method comprises the providing, on the terminal, after the object is selected, of a display of an object in augmented reality in parallel with the displaying of that object.

Description

AUGMENTED REALITY METHOD AND DEVICE

The invention pertains to the technical field of augmented reality, meaning the creation of an augmented real environment using virtual objects.

Augmented reality makes it possible to obtain interaction between people or real objects and virtual objects. For example, augmented reality allows POIs (Points Of Interest) to be superimposed on a mobile terminal's video feed.

More generally speaking, augmented reality aims to give the user additional information that can be perceived, for example, through vision or kinesthetic feedback.

The principles of augmented reality, as applied to mobile terminals, thereby enable directional assistance and guided tours, with possible applications for tourism, culture, and property rental.

The principles of augmented reality, as applied to mobile terminals, also make it possible to add information to media content, in particular during learning, diagnostic, simulation, or prototyping phases.

Augmented reality has current and future applications as varied as interior architecture (see for example the European project Aris), museum environments (virtual tours of archaeological sites using the Archeoguide system, flora and fauna reconstructed by the Lifeplus system), e-commerce (augmented catalogs, such as for example http://www.la-realite-augmentee.fr/videos/my-ikea-et-la-realite- augmentee; http://www.jurawatches.co.uk/virtual-watch.asp), games (hybrids of board games and videogames), industrial maintenance, product design, surgery assistance, and sports (the European projects Piste and Melisa).

The development of mobile terminals equipped with digital cameras, GPS chips, accelerometers, and electronic compasses opens up augmented reality to new applications. Several augmented reality browsers have appeared on the market, including Argon, Wikitude, Layar, as well as various applications (see for example UrbanSpoon, Bionic Eye, Tonchidot).

The application Argon was developed by the Georgia Institute of

Technology, with the support of the applicant. As an implementation of the Kharma project, Argon combines the KML and HTML5 standards and has many advantages, particularly in terms of virtual object interactivity and customization. An overview of Argon is given at the address http://argonbrowser.org.

The application Wikitude developed by the company Mobilizy (http://www. wikitude. org) results from integrating Wikipedia, the Android mobile operating system by Google and the smartphone G1 made by HTC. Wikitude makes it possible to add information from Wikipedia atop the image being filmed by the geo-tagged mobile terminal.

The application Layar enables users of the Android application to add information and points of interest, but does not allow the creation of POIs directly within the source code. The operating principles of Layar are as follows: the user selects a subject from a list, then requests information about that subject from a server (GET request), sending the location information to the server. In response, the server gathers the POIs for the chosen subject in the location's vicinity and sends them to the terminal (in the form of a JSON-format document), with the POIs being overlaid on top, and the virtual objects being correctly oriented on the actual objects particularly by detecting the terminal's orientation (for example, a mobile terminal's compass).

Conventionally, a distinction is made between two types of augmented reality systems: systems related to the terminal's location and systems related to the presence of a marker.

Augmented reality involves recalibrating virtual objects onto a real world. This recalibration requires accurately locating the camera (spatial coherence) and performing that location in "real time" and reducing the latency time due to the propagation of information within the system (temporal coherence).

The implementation of augmented reality on mobile terminals poses several problems related to certain background parameters:

- the technical background, including the mobile terminal's technical specifications, and particularly the sensors (cameras, microphones, GPS, accelerometers), the screen size, and the available network;

- the spatial background, consisting of the device's location and orientation as well as all useful information about the site where the user is located; - the background of the activity, directly related to the user's tasks and intentions, in the short and long terms.

Although applications like Sitelens, Layar, and Wikitude use spatial context data, the same is not true for other contextual dimensions, particularly technical ones.

More generally speaking, the implementation of augmented reality, whether on a mobile terminal or not, may lead to unsuitable results, such as the sending of a noisy audio feed to a terminal whose speakers are at full volume.

The implementation of augmented reality may also be frustrating for a user when multiple actions or combinations are offered to him or her, as configuration by the user is long and tedious, without the user being able to compare the outcomes of each configuration setting. Although in principle it is possible to provide the user with a full description of each object that he or she may interact with, it is not realistic to expect that users will take the time to fully read those descriptions before trying to interact with the objects.

The invention aims to at least partially resolve the drawbacks of the existing devices that have just been presented.

According to a first aspect, the invention pertains to an augmented reality method on a terminal, particularly a mobile terminal, comprising the sending of a description of at least one object to an application, this description including a semantic description of the object's properties, which method comprises the providing, on the terminal, after the object is selected, of a display of an object in augmented reality in parallel with the displaying of that object. The displaying, as a preview, by the augmented reality browser, of the augmented reality object in parallel with the displaying of the object enables the user to assess the interaction's benefit.

The method advantageously comprises a step of composing or aggregating (mashing up) objects.

The description of at least one object advantageously comprises the description of the manner in which the object's functionality must be displayed by the terminal.

The invention relates, according to a first aspect, to an augmented reality system that comprises a terminal equipped with a display unit and a camera, said terminal comprising means for detecting at least one object, said terminal comprising a processor capable of ensuring the receipt of a description of that object, said description comprising a semantic description of the object's properties, the terminal being equipped with means for displaying the object in augmented reality in parallel with the display of that object.

The invention relates, according to a third aspect, to a distributed augmented reality platform comprising a terminal equipped with a client application and a server equipped with a server application, the client application sending the server application some data related to the terminal's environment, the server application sending the client application some data comprising the description of at least one object, this description comprising a semantic description of said object's properties, the terminal being equipped with means of displaying the object in augmented reality in parallel with the displaying of that object.

Other objects and advantages of the invention will become apparent upon reading the description below made with reference to the attached sole figure, which is a functional diagram of one implementation of an augmented reality method.

In the remainder of this description, each "smart" object placed within the user's environment describes its properties using a semantic language understood by the augmented reality browser. The method will be described in a mobile augmented reality context, with the user using a mobile terminal such as a PDA (personal digital assistant), smartphone, tablet computer, or electronic terminal connected to a vehicle.

In a first step 1, the properties of objects placed within the mobile terminal's field are detected. The expression "placed within the mobile terminal's field" particularly means that the objects are placed within the field of the mobile terminal's camera, or that the objects carry chips, for example RFID chips, placed within the mobile terminal's field for detecting those chips.

If the object contains a description of the way in which it must be displayed by the augmented reality browser, that description is sent to the mobile terminal.

In a second step 2, the user selects at least one interaction with an object. At this point, the application advantageously shows multiple combinations 3 of objects. In the following steps 4, user chooses a combination and displays a preview of the chosen combination. This preview step enables the user to check that the proposed augmented reality output complies with the background, particularly in terms of the terminal's capabilities and settings or in terms of the background of activities (the user's short- and medium-term tasks and intentions).

If the user is not enticed by the proposed augmented reality experience depicted by the preview, another combination 3 of objects or another interaction may be chosen.

This way, for example, whenever a client is located in a store in which the items are equipped with RFID chips, the first step of the client's terminal detecting the chips enables him or her to be informed of the presence of items close to him or her. The client may then select at least one direction with an item and the application advantageously shows multiple items and combinations of items, taking into account the client's profile, for example his or her browser's history. The preview may then consist of a virtual try-out of the items, with the client carrying markers, the augmented reality preview may be performed on the client's terminal or on a screen placed within the store.

The inventive method has several advantages, some of which will be illustrated using the example embodiments below.

In a first example embodiment, a first object is a screen, meaning an object capable of displaying a piece of video content; and a second object is a camera (webcam) providing a URL link from which the video may be downloaded. The user has an application enabling the display of the video on any connected screen. During the method's implementation, the augmented reality browser may receive the video stream from the webcam and display that video stream on the screen selected by the user, for example a television screen.

In a second example embodiment, a lamp is identified by the augmented reality browser, this lamp being described as having a light intensity that may vary. In order to simulate the lamp's state changes, the browser may modify the light in the areas that will be lit by the lamp, or for dark environments, change the camera's operating mode to infrared. Preparing the preview increases the chances of a positive augmented reality experience. In a third example embodiment, a landline telephone is identified as a connectable object near the user's mobile terminal. The application offers the user a preview in which, when the landline phone receives a call, in addition to or instead of the landline phone's ring, a light signal appears on the mobile terminal's screen.

The method enables the user to better interact with the connected objects, in smart spaces.

The method particularly makes it possible to offer an alternative to complex contextual browsing methods, while still being compatible with those methods.

In order for augmented reality to be an enriching experience, it is important that the objects offered to the user are consistent with the user's spatial, temporal, technical and activity background. However, the background is fleeting, uncertain, and inaccurate, so that determining it, based particularly on the user's profile, his or her location, the browsing history, cannot be deterministic. When augmented reality is based on location-based services (LBS), not only the environment but also the user profile can change at any time, when the user moves, sometimes by several meters (joining a meeting, leaving a meeting), or when the social background changes (for example, from the professional environment to entertainment).

Claims

1. An augmented reality method on a terminal, particularly a mobile terminal, comprising the sending of a description of at least one object to an application, this description including a semantic description of the object's properties, which method comprises the providing, on the terminal, after the object is selected, of a display of an object in augmented reality in parallel with the displaying of that object.

2. A method according to claim 1, characterized in that it comprises a step of composing or aggregating objects.

3. A method according to claim 1 or claim 2, characterized in that the description of at least one object comprises the description of the way in which the object's functionality must be displayed by the terminal.

4. An augmented reality system comprising a terminal equipped with a display unit and a camera, said terminal comprising means for detecting at least one object, said terminal comprising a processor capable of ensuring the receipt of a description of that object, said description comprising a semantic description of the object's properties, the terminal being equipped with means for displaying the object in augmented reality in parallel with the display of that object.

5. A distributed augmented reality platform comprising a terminal equipped with a client application and a server equipped with a server application, the client application sending the server application some data related to the terminal's environment, the server application sending the client application some data comprising the description of at least one object, this description comprising a semantic description of said object's properties, the terminal being equipped with means of displaying the object in augmented reality in parallel with the displaying of that object.