JP6863902B2 - Augmented reality systems and methods for tracking biometric data - Google Patents

Description

The present disclosure relates to systems and methods for using biometric data to facilitate commerce conducted through augmented reality (AR) devices.

Modern computing and display technology presents digitally reproduced images or parts thereof to users in a manner that appears to be real, or in a manner that can be perceived as such, the so-called "virtual reality". Promotes the development of systems for "augmented reality" experiences. Virtual reality, or "VR" scenarios, typically involves the presentation of digital or virtual image information without transparency to other real-world visual inputs. Augmented reality, or "AR" scenarios, typically involve the presentation of digital or virtual image information as an extension of the visualization of the real world around the user.

For example, with reference to FIG. 1, an augmented reality scene is depicted, where users of AR technology set up a real-world park-like setting featuring concrete platform 1120 against the backdrop of people, trees, and buildings. to see. In addition to these items, users of AR technology also perceive the robot image 1110 standing on the real-world platform 1120 and the flying cartoon-like avatar character 2, but these elements (2, 1110) are , Does not exist in the real world. The human visual perception system is very complex, such augmented reality that promotes the comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. Generating a scene is difficult.

It is envisioned that such an AR device could be used to present any type of virtual content to the user. In one or more embodiments, the AR device can be used in the context of various gaming applications, allowing the user to participate in a single player or multiplayer video / augmented reality game that mimics a real situation. .. For example, rather than playing a video game on a personal computer, an AR user can play the game on a larger scale in conditions that are very similar to the real world (eg, a "full-scale" 3D monster, where the AR user parks. Can appear from behind the actual building while walking around). In fact, this significantly improves the credibility and enjoyment of the gaming experience.

Although FIG. 1 illustrates the potential of AR devices in the context of gaming applications, AR devices can be used in many other applications, such as everyday computing devices such as personal computers, mobile phones, tablet devices, etc. ) Can be expected to replace. By strategically placing virtual content in the user's field of view, an AR device allows the user to simultaneously connect to the user's physical environment while performing various computing tasks (eg, checking email, terms). It can be considered a walking personal computer that allows you to search on the web, conference with other AR users, watch movies, etc.). For example, rather than being constrained by physical devices on the desk, AR users are "on the move" (eg, while taking a walk, on a daily commute, away from the computer in a physical location other than the office, etc. ), And it is still possible to have a video conference with a friend by pulling out a virtual email screen and, for example, checking the email or virtually presetting the screen on the AR device. , Or in another example, it is possible to build a virtual office in a makeshift location. Many similar virtual reality / augmented reality scenarios can be envisioned.

This shift in the nature of lower-tier computing technology has its own advantages and challenges. To present augmented reality scenes such as those mentioned above, taking into account the physiological limitations of the human visual system, AR devices associate desired virtual content with one or more real objects within the user's physical environment. You must be aware of the user's physical surroundings in order to project. To achieve this goal, AR devices typically include various tracking devices (eg, eye tracking devices, GPS, etc.), cameras (eg, field cameras, infrared cameras, depth cameras, etc.), and sensors (eg, field cameras, infrared cameras, depth cameras, etc.). Equipped with, for example, accelerometers, gyroscopes, etc.) to assess the user's position, orientation, distance, etc. in relation to various real objects within the user's surroundings, real-world objects and other such functions. Detects and identifies sex.

If the AR device is configured to track various types of data about the AR user and its surroundings, in one or more embodiments, this data advantageously requires minimal input from the user. It can be utilized to assist various types of transactions of the user while ensuring that it is done and with little or no interruption in the user's AR experience.

More specifically, traditional transactions (financial or other) typically involve the user in some form of monetary token (eg, cash, check, credit card, etc.), in some cases, in order to participate in a commercial transaction. Require physical carrying of identification (eg, driver's license, etc.), and authentication (eg, signature, etc.). Suppose a user goes inside a department store. To make any kind of purchase, the user typically picks up the item, puts the item in the cart, walks to the cashier, lines up in front of the cashier, and has some cashiers. Wait for the item to be scanned, remove the credit card, provide identification, sign the credit card receipt, and keep the receipt for future returns of the item. In traditional financial transactions, these steps are necessary, but time consuming and inefficient, and in some cases these steps deter or prohibit users from making purchases (eg,). The user does not carry a financial token or ID card, etc.). However, in the context of AR devices, these steps are redundant and unnecessary. In one or more embodiments, the AR device may be configured to allow the user to seamlessly perform many types of transactions without requiring the user to perform the tedious steps described above. ..

Therefore, there is a need for better solutions to help AR users participate in routine transactions.

Embodiments of the invention cover devices, systems, and methods for facilitating virtual reality and / or augmented reality interactions for one or more users.

On the one hand, the method of conducting a transaction through an augmented reality device is to capture biometric data from the user and to determine the user's identity based at least in part on the captured biometric data. Includes authenticating the user for a transaction based on the identification made.

In one or more embodiments, the method further comprises transmitting a set of data about the transaction to a financial institution. In one or more embodiments, the biometric data is an iris pattern. In one or more embodiments, the biometric data is a voice recording of the user. In one or more embodiments, the biometric data is a retinal signature. In one or more embodiments, the biometric data is a property associated with the user's skin.

In one or more embodiments, the biometric data is captured through one or more eye tracking cameras that capture the movement of the user's eyes. In one or more embodiments, the biometric data is a pattern of movement of the user's eyes. In one or more embodiments, the biometric data is a blink pattern of the user's eyes.

In one or more embodiments, the augmented reality device is head-mounted and the augmented reality device is individually calibrated for the user. In one or more embodiments, the biometric data is compared to predetermined data about the user. In one or more embodiments, the predetermined data is a known signature transfer in the user's eye.

In one or more embodiments, the predetermined data is a known iris pattern. In one or more embodiments, the predetermined data is a known retinal pattern. In one or more embodiments, the method further detects the user's desire to make a transaction and requests biometric data from the user, at least in part, based on the detected desire. Comparing metric data with predetermined biometric data to produce a result, including that the user is authenticated based on the result, at least in part.

In one or more embodiments, the transaction is a commercial transaction. In one or more embodiments, the method is further to communicate the user's authentication to the financial institution associated with the user, who pays on behalf of the user, at least in part, based on the authentication. Including doing. In one or more embodiments, the financial institution transmits the payment to one or more distributors indicated by the user.

In one or more embodiments, the method further comprises detecting an interruption or trading event associated with an augmented reality device. In one or more embodiments, the method further comprises capturing new biometric data from the user in order to reauthenticate the user, at least in part, based on the detected event. In one or more embodiments, the interruption of activity is detected, at least in part, based on the removal of the augmented reality device from the user's head.

In one or more embodiments, the interruption of activity is detected, at least in part, based on the loss of connectivity of the augmented reality device to the network. In one or more embodiments, the transaction event is detected, at least in part, based on the explicit approval of the transaction by the user. In one or more embodiments, trading events are detected, at least in part, based on the heatmap associated with the user's gaze.

In one or more embodiments, the trading event is detected, at least in part, based on user input received through the augmented reality device. In one or more embodiments, the user input comprises an eye gesture. In one or more embodiments, the user input comprises a hand gesture.

In another aspect, the augmented reality display system is operably coupled to a biometric data tracking device for capturing biometric data from the user and the biometric data tracking device to process the captured biometric data. It includes a processor for determining user identification, at least partially based on captured biometric data, and at least a server for communicating with financial institutions and authenticating users for transactions.

In one or more embodiments, the biometric data is eye movement data. In one or more embodiments, the biometric data corresponds to an image of the user's iris. In one or more embodiments, the server also transmits a set of data about the transaction to the financial institution. In one or more embodiments, the biometric data is an iris pattern.

In one or more embodiments, the biometric data is a voice recording of the user. In one or more embodiments, the biometric data is a retinal signature. In one or more embodiments, the biometric data is a property associated with the user's skin. In one or more embodiments, the biometric tracking device comprises one or more eye tracking cameras to capture the movement of the user's eyes. In one or more embodiments, the biometric data is a pattern of movement of the user's eyes.

In one or more embodiments, the biometric data is a blink pattern of the user's eyes. In one or more embodiments, the augmented reality display system is head-mounted and the augmented reality display system is individually calibrated for the user. In one or more embodiments, the processor also compares biometric data with predetermined data about the user. In one or more embodiments, the predetermined data is a known signature transfer in the user's eye. In one or more embodiments, the predetermined data is a known iris pattern. In one or more embodiments, the predetermined data is a known retinal pattern. In one or more embodiments, the processor detects that the user wants to make a transaction, and the processor requests biometric data from the user, at least in part, based on the detection. Further, the processor compares the biometric data with the predetermined biometric data and authenticates the user based on the comparison, at least in part.

In one or more embodiments, the transaction is a commercial transaction. In one or more embodiments, the processor communicates the user's authentication to the financial institution associated with the user, who makes payments on behalf of the user, at least in part based on the authentication. In one or more embodiments, the financial institution transmits the payment to one or more distributors indicated by the user.

In one or more embodiments, the processor detects interruption or trading events associated with the augmented reality device, and the biometric tracking device reauthenticates the user based at least in part on the detected events. To capture new biometric data from users. In one or more embodiments, the interruption of activity is detected, at least in part, based on the removal of the augmented reality device from the user's head.

In one or more embodiments, the interruption of activity is detected, at least in part, based on the loss of connectivity of the augmented reality device to the network. In one or more embodiments, the transaction event is detected, at least in part, based on the explicit approval of the transaction by the user. In one or more embodiments, trading events are detected, at least in part, based on the heatmap associated with the user's gaze. In one or more embodiments, the trading event is detected, at least in part, based on user input received through the augmented reality device. In one or more embodiments, the user input comprises an eye gesture. In one or more embodiments, the user input comprises a hand gesture.

In one or more embodiments, the biometric tracking device comprises an eye tracking system. In one or more embodiments, the biometric tracking device comprises a tactile device. In one or more embodiments, the biometric tracking device comprises a sensor that measures physiological data about the user's eye.
The present specification also provides, for example, the following items.
(Item 1)
A method of conducting a transaction through an augmented reality device, said method.
Capturing biometric data from users and
Determining the identification of the user based at least in part on the captured biometric data.
To authenticate the user for the transaction based on the determined identification.
Including methods.
(Item 2)
The method of item 1, further comprising transmitting a set of data relating to the transaction to a financial institution.
(Item 3)
The method according to item 1, wherein the biometric data is an iris pattern.
(Item 4)
The method according to item 1, wherein the biometric data is a voice recording of the user.
(Item 5)
The method according to item 1, wherein the biometric data is a retinal signature.
(Item 6)
The method of item 1, wherein the biometric data is a property associated with the user's skin.
(Item 7)
The method of item 1, wherein the biometric data is captured through one or more eye tracking cameras that capture the movement of the user's eyes.
(Item 8)
The method according to item 1, wherein the biometric data is a pattern of eye movement of the user.
(Item 9)
The method according to item 1, wherein the biometric data is a blink pattern of the user's eyes.
(Item 10)
The method of item 1, wherein the augmented reality device is head-mounted and the augmented reality device is individually calibrated for the user.
(Item 11)
The method of item 1, wherein the biometric data is compared with predetermined data about the user.
(Item 12)
The method of item 11, wherein the predetermined data is a known signature movement of the user's eye.
(Item 13)
The method of item 11, wherein the predetermined data is a known iris pattern.
(Item 14)
The method of item 11, wherein the predetermined data is a known retinal pattern.
(Item 15)
To detect the user's desire to make a transaction
Requesting the biometric data from the user, at least in part, based on the detected desire.
By comparing the biometric data with predetermined biometric data and generating a result, the user is authenticated based on the result at least in part.
The method according to item 1, further comprising.
(Item 16)
The method according to item 1, wherein the transaction is a commercial transaction.
(Item 17)
The first item, wherein the user's authentication further comprises communicating the user's authentication with a financial institution associated with the user, the financial institution making payments on behalf of the user, at least in part based on the authentication. Method.
(Item 18)
17. The method of item 17, wherein the financial institution transmits the payment to one or more sellers indicated by the user.
(Item 19)
To detect interruption or trading events associated with the augmented reality device,
To capture new biometric data from the user in order to reauthenticate the user, at least in part, based on the detected event.
The method according to item 1, further comprising.
(Item 20)
19. The method of item 19, wherein the interruption of the activity is detected, at least in part, based on the removal of the augmented reality device from the user's head.
(Item 21)
19. The method of item 19, wherein the interruption in activity is detected, at least in part, based on the loss of connectivity of the augmented reality device with the network.
(Item 22)
19. The method of item 19, wherein the transaction event is detected, at least in part, based on the explicit approval of the transaction by the user.
(Item 23)
19. The method of item 19, wherein the trading event is detected, at least in part, based on a heatmap associated with the user's gaze.
(Item 24)
19. The method of item 19, wherein the trading event is detected, at least in part, based on user input received through the augmented reality device.
(Item 25)
24. The method of item 24, wherein the user input comprises an eye gesture.
(Item 26)
24. The method of item 24, wherein the user input comprises a hand gesture.
(Item 27)
The method of item 1-26, which is implemented as a system having means for implementing the method steps.
(Item 28)
The method of item 1-26, which is implemented as a computer program product comprising a computer-enabled storage medium having executable code for performing the method steps.

Additional and other purposes, features, and advantages of the invention are described in the embodiments, drawings, and claims for carrying out the invention.

The drawings illustrate the design and availability of various embodiments of the invention. Note that figures are not necessarily drawn to exact scale and elements of similar structure or function are represented by similar reference numbers throughout the figure. In order to gain a deeper understanding of the aforementioned and other advantages of the various embodiments of the present invention as well as the method of obtaining the object, embodiments for carrying out the aforementioned invention in a simple manner are illustrated in the accompanying drawings. It will be given by reference to the embodiments. With the understanding that these drawings depict only typical embodiments of the invention and are therefore not considered to be a limitation of its scope, the invention provides additional specificity and detail through the use of accompanying drawings. Will be described and explained with.
FIG. 1 illustrates an exemplary augmented reality scene displayed to the user. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 3 illustrates an augmented reality device that communicates with one or more servers in the cloud, according to one embodiment. 4A-4D illustrate various eye and head measurements made to configure an augmented reality device for a particular user. FIG. 5 shows a plan view of various components of an augmented reality device according to one embodiment. FIG. 6 shows the system architecture of an augmented reality system for carrying out commercial transactions according to one embodiment. FIG. 7 is an exemplary flowchart illustrating how to conduct a commercial transaction through an augmented reality device. 8A and 8B illustrate an exemplary eye identification method for identifying a user according to one embodiment. 8A and 8B illustrate an exemplary eye identification method for identifying a user according to one embodiment. FIG. 9 illustrates an exemplary flow chart illustrating a method of authenticating a user using eye movement according to one embodiment. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device. FIG. 10A-10I illustrates a series of process flow diagrams illustrating an exemplary scenario of conducting a commercial transaction using an augmented reality device.

Various embodiments of the present invention cover methods, systems, and manufactured products that implement a multi-scenario physical cognitive design for electronic circuit design in a single embodiment or multiple embodiments. Other objects, features, and advantages of the invention are described in embodiments, drawings, and claims for carrying out the invention.

Here, various embodiments will be described in detail with reference to the drawings provided as exemplary embodiments of the invention to allow one of ordinary skill in the art to practice the invention. .. It should be noted that the figures and examples below do not imply limiting the scope of the invention. If some elements of the invention can be partially or fully implemented using known components (or methods or processes), such known components (or methods or processes) necessary for the understanding of the invention. ) Only those parts will be described, and detailed description of other parts of such known components (or methods or processes) will be omitted so as not to obscure the present invention. In addition, various embodiments include current and future known equivalents of the components referenced herein by way of example.

Disclosed are methods and systems that track biometric data associated with AR users and utilize the biometric data to assist in commerce. In one or more embodiments, the AR device utilizes eye identification techniques (eg, iris pattern, binocular detachment movement, eye movement, pyramidal and rod cell patterns, eye movement patterns, etc.). Can authenticate the user for purchase. Advantageously, this type of user authentication minimizes the frictional cost of conducting a commercial transaction and allows the user to make a seamless purchase with minimal effort and / or interruption (eg). , In response to physical stores, online stores, advertisements, etc.). The following disclosure focuses primarily on authentication based on eye-related biometric data, but understand that other types of biometric data may also be used for authentication purposes in other embodiments. I want to be. Various embodiments discuss new conceptual schemes for conducting buying and selling in the context of augmented reality (AR) systems, as described below, but the techniques disclosed herein are optional. It should be understood that it can be used independently of existing and / or known AR systems. Therefore, the examples discussed below are for illustrative purposes only and the present invention should not be read as limited to AR systems.

With reference to FIG. 2A-2D, some common component options are illustrated. FIG. 2A-2D is a display system in which various systems, subsystems, and components are perceived as high quality and comfortably for human VR and / or AR in some of the embodiments for which the discussion continues. Is presented to address the purpose of providing.

As shown in FIG. 2A, the AR system user 60 is depicted wearing a frame 64 structure coupled to an AR display system 62 located in front of the user's eyes. The speaker 66 is coupled to the frame 64 in the configured configuration and is positioned adjacent to the user's ear canal (in one embodiment, another speaker (not shown) is positioned adjacent to the user's other ear canal. And provides stereo / moldable sound control). The display 62 is operably coupled to the local processing and data module by wired leads, wireless connectivity, etc. 68, and the local processing and data module is fixedly attached to the frame 64 or shown in the embodiment of FIG. 2B. Whether it is fixedly attached to the helmet or hat 80, embedded in headphones, or removably attached to the fuselage 82 of the user 60 in a rucksack configuration as shown in the embodiment of FIG. 2C. Alternatively, it can be mounted in various configurations, such as being removably attached to the waist 84 of the user 60 in a belt-coupled configuration as shown in the embodiment of FIG. 2D.

The local processing and data module 70 may include a power efficient processor or controller and digital memory such as flash memory, both of which may be utilized to assist in the processing, caching, and storage of the data. a) Captured from a sensor that can be operably coupled to a frame 64 such as an image capture device (camera, etc.), microphone, inertial measurement unit, accelerometer, compass, GPS unit, wireless device, and / or gyroscope. And / or b) Perhaps after such processing or reading, it is acquired and / or processed using the remote processing module 72 and / or the remote data repository 74 for passage to the display 62. The local processing and data module 70 provides a wired or wireless communication link, etc. so that these remote modules (72, 74) are operably coupled to each other and available as resources to the local processing and data module 70. Via, it can be operably coupled (76, 78) to the remote processing module 72 and the remote data repository 74.

In one embodiment, the remote processing module 72 may include one or more relatively sophisticated processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 74 may include relatively large digital data storage equipment that may be available through the Internet or other networking configurations within a "cloud" resource configuration. In one embodiment, all data is stored locally and in a data module, where all calculations are done, which allows for fully autonomous use without any remote module.

As described with reference to FIGS. 2A-2D, the AR system continuously receives input from various devices that collect data about AR users and their surroundings. With reference to FIG. 3, various components of an exemplary augmented reality display device will be described. It should be understood that other embodiments may have additional components. Nevertheless, FIG. 3 provides a basic concept of the types of data that can be collected by various components and AR devices.

Here, with reference to FIG. 3, the schematic illustrates the coordination between the cloud computing asset 46 and the local processing asset (308, 120). In one embodiment, the cloud 46 assets are stored in a structure configured to be coupled directly to the user's head-mounted device 120 or belt 308 via wired or wireless networking (40, 42). Obtaining operably coupled to one or both of local computing assets (120, 308) such as processor and memory configuration (wireless is preferred for mobility, wired is some high bandwidth or high that may be desired. Preferred for data volume transfer). These user-local computing assets can also be operably coupled to each other via wired and / or wireless connectivity configurations 44. In one embodiment, in order to maintain low inertia and small size head-mounted subsystem 120, the primary transfer between the user and the cloud 46 is via the link between the belt-based subsystem 308 and the cloud. Obtained, the head-mounted subsystem 120 mainly uses wireless connectivity such as ultra-wideband (“UWB”) connectivity, as is currently employed in personal computing peripheral connectivity applications, for example. , Data tethered to belt-based subsystem 308. Through the cloud 46, the AR display system 120 may interact with one or more AR servers 110 hosted in the cloud. Various AR servers 110 may have communication links 115 that allow the servers 110 to communicate with each other.

A world related to the user's current real or virtual location by using efficient local and remote processing coordination and the appropriate display device for the user, such as the user interface or the user "display device" or a variant thereof. Aspects can be transferred or "passed" to the user and updated in an efficient manner. In other words, the map of the world is partly resident on the user's AR system and partly continuously updated with storage locations that can be resident on cloud resources. A map (also referred to as a passable world model) can be a large database containing raster images, 3D and 2D points, parameter information, and other information about the real world. Maps become more and more accurate as AR users continuously capture more and more information about their real environment (eg, through cameras, sensors, IMUs, etc.).

In connection with the present disclosure, an AR system similar to that described in FIGS. 2A-2D is advantageously used to uniquely identify a user based on a set of biometric data tracked through the AR system. It provides unique access to the user's eyes that can be used. This unprecedented access to the user's eyes necessarily makes it suitable for a variety of applications. The AR device plays an important interaction with the user's eye to allow the user to perceive 3D virtual content, and in many embodiments, various biometrics associated with the user's eye (eg, eg). If binocular detachment movements, eye movements, rod and cone cell patterns, eye movement patterns, etc.) are tracked, the resulting tracking data are advantageously described in more detail below. As such, it can be used in user identification and authentication for various transactions.

The AR device is typically adapted for a particular user's head and the optical components are aligned with the user's eyes. These configuration steps can be used to ensure that the user is provided with an optimal augmented reality experience without causing any physiological side effects such as headache, nausea, discomfort and the like. Thus, in one or more embodiments, the AR device can be configured for each individual user (both physical and digital) and calibrated for the user. In other scenarios, loosely adapting the AR device can be comfortably used by a variety of users. For example, in some embodiments, the AR device captures the distance between the user's eyes, the distance between the head-worn display and the user's eyes, the distance between the user's eyes, and the curvature of the user's forehead. To do. All of these measurements can be used to provide a suitable head-worn display system for a given user. In other embodiments, such measurements may not be necessary to perform the identification and authentication functions described herein.

For example, referring to FIGS. 4A-4D, the AR device can be customized for each user. The user's head shape 402 may be considered when fitting the head-mounted AR system in one or more embodiments, as shown in FIG. 4A. Similarly, can the eye component 404 (eg, optics, structure for optics, etc.) be rotated or adjusted for user comfort, both horizontally and vertically, as shown in FIG. 4B. Or it can be rotated for user comfort. In one or more embodiments, as shown in FIG. 4C, the head rotation point 406 set with respect to the user's head can be adjusted based on the structure of the user's head. Similarly, the interpupillary distance (IPD) 408 (ie, the distance between the user's eyes) can be compensated as shown in FIG. 4D.

Advantageously, in the context of user identification and authentication, this aspect of the head-worn AR device allows the system to measure the user's physical characteristics (eg, eye size, head size, eye-to-eye distance, etc.). It is important because we already have a set of data and other data that can be used to easily identify the user and allow the user to complete one or more commercial transactions. In addition, the AR system may be readily detectable when the AR system is worn by a different AR user other than the user authorized to use the AR system. This allows the AR system to constantly monitor the user's eyes and therefore perceive the user's identity as needed.

In addition to the various measurements and calibrations made on the user, the AR device can be configured to track a set of biometric data about the user. For example, the system provides users with eye movements, eye movement patterns, blink patterns, binocular detachment movements, eye colors, iris patterns, retinal patterns, fatigue parameters, eye color changes, focal length changes, and an augmented reality experience. Many other parameters that can be used in doing so can be tracked.

Referring to FIG. 5, one simplified embodiment of a suitable user display device 62 is shown to include a display lens 106 that can be mounted on the user's head or eyes by a housing or frame 108. Is done. The display lens 106 is positioned by the housing 108 in front of the user's eye 20 and bounces the projected light 38 into the eye 20 to form a beam while also allowing at least a portion of the light to pass from the local environment. May include one or more transparent mirrors configured to facilitate. In the illustrated embodiment, two wide-angle machine vision cameras 16 are coupled to the housing 108 to image the environment around the user. In one embodiment, these cameras 16 are dual capture visible / infrared light cameras.

The embodiment depicted is also a pair of scanning laser-formed wave planes (ie, for depth) with display mirrors and optics configured to project light 38 into the eye 20, as shown. It includes a module 18 (eg, a spatial light modulator such as DLP, fiber scanning device (FSD), LCD). The illustrated embodiment is also paired with an infrared light source 26, such as a light emitting diode "LED", configured to be capable of tracking the user's eye 20 and supporting rendering and user input. It is equipped with two small infrared cameras 24. The display system 62 further comprises an X, Y, Z axis accelerometer capability and a magnetic compass and an X, Y, Z axis gyroscope capability, preferably capable of providing data at a relatively high frequency such as 200 Hz. It features a sensor assembly 39. The system 62 depicted can also be configured to calculate real or near real-time user head postures from wide-angle image information output from the camera 16, ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and the like. And / or a head attitude processor 36 such as an ARM processor (advanced reduction instruction set machine). The head posture processor 36 is operably coupled to the camera 16 and the rendering engine 34 (90, 92, 94; eg, via wired or wireless connectivity).

Another processor 32 is also shown that is configured to perform digital and / or analog processing and derive the attitude from the gyroscope, compass, and / or accelerometer data from the sensor assembly 39. The embodiments depicted also feature a GPS37 subsystem to assist in posture and positioning.

Finally, the depicted embodiment is configured to provide the user's local rendering information for a view of the user's world, facilitating scanner operation and imaging into the user's eyes. It includes a rendering engine 34 that can be characterized by hardware that executes software programs. The rendering engine 34 includes a sensor orientation processor 32, an image orientation processor 36, and an eye tracking camera 24 so that the rendered light 38 is projected using the scanning laser array 18 in a manner similar to a retinal scanning display. , And operably coupled to the projection subsystem 18 (105, 94, 100/102, 104, i.e., via wired or wireless connectivity). The wave front of the projected light beam 38 can be bent or focused to match the desired focal length of the projected light 38.

The small infrared eye tracking camera 24 can be used to track the eye and support rendering and user input (ie, where the user is looking, depth of focus, etc., as discussed below. , Binocular detachment can be used to estimate the depth of focus of the user). A GPS 37, a gyroscope, a compass, and an accelerometer 39 can be used to provide rough and / or fast attitude estimation. Camera 16 images and pose data, along with data from associated cloud computing resources, can be used to map the local world and share user views with virtual or augmented reality communities.

Much of the hardware in the display system 62 featured in FIG. 5 is depicted as being directly coupled to the display 106 and the housing 108 adjacent to the user's eye 20, although the hardware components depicted may be, for example, , As shown in FIG. 2D, may be mounted on or stored in other components such as the belt-mounted component 70.

In one embodiment, all the components of the system 62 featured in FIG. 5 are directly coupled to the display housing 108, except for the image attitude processor 36, the sensor attitude processor 32, and the rendering engine 34, and the latter three and the system. Communication with the remaining components may be by wireless or wired communication, such as ultra-wideband. The housing 108 depicted is preferably head-mounted and can be worn by the user. It may also feature a speaker, such as one that is inserted into the user's ear and can be used to provide sound to the user.

Although the main components of a standard AR device have been described, it should be understood that an AR device can have many components that are configured to collect data from the user and its surroundings. For example, as mentioned above, some embodiments of AR devices collect GPS information and determine the location of the user. In another embodiment, the AR device comprises an infrared camera to track the user's eyes. In yet another embodiment, the AR device comprises a field camera, which may capture an image of the user's environment, which then maps the user's physical space (as described in FIG. 3, server 110). Can be used to configure (contained within one of), which allows the system to render virtual content in relation to the appropriate real object, as briefly described with respect to FIG. To enable.

With respect to the projection of light 38 into the user's eye 20, in one embodiment a small camera 24 may be utilized to measure where the center of the user's eye 20 is geometrically verged. The location generally coincides with the position of the focal point of the eye 20, i.e. the "depth of focus". The three-dimensional surface of all points that the eye approaches is called a "monoscopic locus". Focal lengths can take a finite number of depths or fluctuate infinitely. The light projected from the binocular divergence distance appears concentrated in the target eye 20, while the light in front of or behind the binocular divergence distance is blurred.

In addition, it has been found that spatially coherent light with a beam diameter of less than about 0.7 mm is correctly decomposed by the human eye, regardless of where the eye focuses. Given this understanding, binocular divergence may be added by the small camera 24 to generate an illusion of appropriate depth of focus, with the rendering engine 34 and projection subsystem 18 on or near the monoscopic trajectory. It can be used to focus on all objects and render all other objects with varying degrees of defocus (ie, using intentionally generated defocus). See-through optical waveguide optics configured to project coherent light into the eye are described by Lumus, Inc. Can be provided by a supplier such as. Preferably, the system 62 renders to the user at a frame rate of about 60 frames per second or higher. As mentioned above, preferably the small camera 24 can be utilized for eye tracking and the software provides clues to the focal location to serve as user input as well as binocular divergent motion geometry. Can be configured to get. Preferably, such a system is configured with brightness and contrast suitable for daytime or nighttime.

In one embodiment, such a system preferably has a latency of less than about 20 milliseconds, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 minute for visual object alignment. It is almost the limit of the human eye. The display system 62 may be integrated with a positioning system that may assist in positioning and orientation determination, including GPS elements, optical tracking, compasses, accelerometers, and / or other data sources. Positioning information can be used to facilitate accurate rendering within the user's view of the relevant world (ie, such information is to facilitate the eyeglasses to figure out where they are relative to the real world. Deaf). Although the general components of an AR device have been described, additional embodiments relating to specific user identification and authentication for conducting commercial transactions will be discussed below.

As discussed in some detail above, traditional models for conducting commercial transactions tend to be inefficient and cumbersome, and often have the effect of preventing users from engaging in transactions. For example, suppose a user is in a department store. In the traditional model, the user is required to physically enter the department store, select a product, line up, wait for a cashier, provide payment information and / or identification, and approve the payment. Will be done. Even online shopping is probably less complicated, but it has its drawbacks. Although users do not have to be physically present at the store location and can easily select products of interest, payments still often require credit card information and authentication. However, with the advent of AR devices, traditional models of payment (eg cash, credit cards, monetary tokens, etc.) have become possible because AR devices can easily verify user identification and authenticate commerce. , Can be unnecessary.

For example, an AR user can hang out in a retail store and pick up a product. The AR device may confirm the user's identification, confirm whether the user wants to purchase, and simply leave the store. In one or more embodiments, the AR device may interface with a financial institution that will transfer the amount from the user's account to the account associated with the retail store based on the confirmed purchase. Alternatively, in another example, an AR user may be viewing an advertisement for a particular brand of shoes. The user may indicate through the AR device that the user wants to buy shoes. The AR device can verify the user's identity and authenticate the purchase. At the back end, the order may be placed with a shoe brand retailer, who may simply ship a pair of desired shoes to the user. As illustrated by the example above, the AR device "knows" the identity of the user (and the AR device is typically built and customized for every individual user). Financial transactions are easily authenticated, thereby significantly reducing the frictional costs typically associated with conducting a sale.

More specifically, in one or more embodiments, the AR device may periodically perform user identification tests for privacy and security purposes. As discussed in some detail above, AR devices are typically customized for each user, but this periodic identification and authentication of users is for security purposes, especially in the context of conducting commercial transactions. It is necessary for this purpose, or for privacy purposes to ensure that the AR device is not being used by an unknown user and is linked to the AR user's account on the cloud. This application describes systems and methods for ensuring security for financial / commercial transactions where user identification and authentication are paramount. Similarly, these steps are equally important to ensure user privacy. In fact, these identification steps can be used prior to opening any personal / private user account (eg, email, social networks, financial accounts, etc.) through an AR device.

Other embodiments described herein can be used in the context of anti-theft management. For example, an AR device can identify a user and ensure that the AR device has not been stolen. If the AR device detects an unknown user, the AR device may immediately send the captured information about the user and the location of the AR device to the AR server. Or, in other embodiments, if it is detected that the AR device is being used by an unidentified person, the AR device will shut down completely and the AR will not be leaked or misused to prevent sensitive information from being leaked or misused. All content in the device's memory can be deleted automatically. These security measures can prevent the theft of the AR device because the AR device can capture many types of information about the wearer of the AR device.

In one or more embodiments, the embodiments described below may allow the AR device to remotely control the shopping robot. For example, once a user of an AR device is identified, the AR device may connect to the shopping robot and communicate transactions with the shopping robot through a network of specific stores or franchise stores. Therefore, even if the user is not physically present in the store, the AR device can execute the transaction through the proxy when the user is authenticated. Similarly, many other security and / or privacy applications can be envisioned.

There are many ways to make an identification through tracked biometric data. In one embodiment, the tracked biometric data can be eye-related biometric data such as eye movement patterns, iris patterns, binocular detachment movement information, and the like. In short, instead of requiring the user to remember the password or present some type of identification / proof, the AR device automatically validates the identification through the use of tracked biometric data. Given that the AR device has constant access to the user's eyes, it is expected that the tracked data will provide very accurate and personalized information about the user's identity, and that information. Can be considered a unique user signature. Track biometric data and use the tracked biometric data to authenticate users with one or more external entities (eg, financial institutions, distributors, etc.) before delving into the details of different methods. A system architecture for interacting AR devices will be provided.

Here, with reference to FIG. 6, the overall AR system architecture is illustrated. The AR system architecture includes a head-mounted AR device 62, a local processing module 660 for the AR device 62, a network 650, an AR server 612, a financial institution 620, and one or more distributors (622A, 622B, etc.). including.

As mentioned above (see Figure 5), the head-worn AR device 62 has many sub-components, some of which capture and / or track information associated with the user and / or around the user. It is configured to do. More specifically, in one or more embodiments, the head-worn AR device 62 may include one or more eye tracking cameras. The eye tracking camera can track the movement of the user's eyes, binocular detachment movement, and the like. In another embodiment, the eye tracking camera may be configured to capture a picture of the user's iris. In other embodiments, the head-worn AR device 62 may include other cameras configured to capture other biometric information. For example, the associated camera may be configured to capture an image of the shape of the user's eye. Alternatively, in another example, the camera (or other tracking device) may capture data about the user's eyelashes. Tracked biometric information (eg, eye data, eyebrows data, eye shape data, eye movement data, head data, sensor data, voice data, fingerprint data, etc.) can be transmitted to the local processing module 660. As shown in FIG. 2D, in one or more embodiments, the local processing module 660 can be part of the belt pack of the AR device 62. Alternatively, in other embodiments, the local processing module may be part of the housing of the head-mounted AR device 62.

As shown in FIG. 6, the user 680 head-worn AR device 62 interfaces with the local processing module 660 to provide captured data. In one or more embodiments, the local processing module 660 comprises a processor 664 and other components 652 (eg, memory, power supplies, telemetry circuits, etc.) that allow the AR system to perform various computing tasks. There is. Significantly in the present disclosure, the local processing module 660 also comprises an identification module 614, which can identify the user based on the information tracked by one or more tracking devices of the head-mounted AR device 62.

In one or more embodiments, the identification module 614 comprises a database 652, stores a set of data, and uses it to identify and / or authenticate the user. For example, in the illustrated embodiment, the database 652 comprises a mapping table 670 capable of storing a given set of data and / or a given authentication detail or pattern. In one or more embodiments, the captured data can be compared to predetermined data stored in the mapping table 670 to determine the identification of the user 680. Similarly, the database 652 may include other data that will be used in identifying / authenticating the user 680. In one or more embodiments, the database 652 may store one or more eye tests and verify user identification, as will be described in more detail below.

In one or more embodiments, the local processing module 660 communicates with the AR server 612 through the cloud network 650. Although not shown in FIG. 6, the AR server 612 includes many components / circuits that are important for providing a realistic augmented reality experience to the user 680. In summary, the AR server 612 comprises a physical world map 690 that is frequently referenced by the local processing module 660 of the AR device 62 to render virtual content in relation to real world physical objects. There is. Therefore, the AR server 612 is built on the information captured through a large number of users to build the ever-growing map 690 in the real world. In other embodiments, the AR server 612 may simply host the map 690, which can be built and maintained by a third party.

In one or more embodiments, the AR server 612 may also host an individual user's account, to which the user's privately captured data is supplied. This captured data may be stored in database 654 in one or more embodiments. In one or more embodiments, the database 654 may store user information 610, historical data 615 for user 680, user preferences 616, and other entity authentication information 618. In fact, other embodiments of the AR system may include many other types of information that are personal to the user. User information 610 may include a set of personal biographies (eg, name, age, gender, address, location, etc.) in one or more embodiments.

Historical data 615 for user 680 may refer to previous purchases and / or transactions made by the user. In one or more embodiments, the user preference 616 presents a set of interests (eg, shopping, activities, travel, etc.) and / or purchase preferences for the user (eg, accessories, brands of interest, shopping categories, etc.). Can include. In one or more embodiments, AR user behavior data can be used to inform the system of user preferences and / or purchase patterns. Other entity credentials 618 may refer to a user's credentials to prove that the user has successfully authenticated access to an external account (eg, bank credentials, account credentials for various websites). etc)
In one or more embodiments, the data captured through the AR device 62, the data tracked through past activity, the buying and selling data associated with the user, etc., to recognize and / or understand the user's behavioral patterns. Can be analyzed. These functions may, in one or more embodiments, be performed by a third party in a privacy and security-friendly manner.

Database 654 may also store other entity credentials 618 that allow the AR server 612 to communicate with a particular financial institution and / or third party entity. For example, the other entity authentication information 618 may point to the user's bank information (eg, bank name, account information, etc.). This information can then be used to communicate with financial information, third party entities, distributors, etc.

In one or more embodiments, the AR server 612 may communicate with one or more financial institutions 620 to complete the transaction. The financial institution may have the user's financial information. In one or more embodiments, the financial institution may perform a second verification of the user's credentials for security purposes. When the user is authenticated, the AR server 612 is authenticated and can communicate with the financial institution 620. If the user is authenticated for a particular purchase of goods 630, the financial institution 620 will communicate directly with one or more distributors (622A, 622B, etc.) and directly charge the amount when the user is authenticated. Can be transmitted to the distributor. In another embodiment (not shown), the AR server 612 may directly communicate with the seller and communicate data about one or more purchases.

It should be appreciated that in some embodiments, the user 680 may not need to connect to the AR server 612 in order to proceed with one or more financial transactions. For example, in some embodiments, the AR device 62 may allow "offline browsing" of multiple e-commerce sites, etc., and the user 680 may select one or more products of interest through an offline ID. Can be possible. The financial institution 620 or distributor 622 may have a random number generator generated for that particular transaction, which can be verified later when the AR device 62 is connected to network 650. In other words, even if the AR device 62 is not connected to the AR server, the system will review the transaction offline and then use additional information (eg, randomly generated numbers) to make a later purchase. Can be enabled. This allows the AR device 62 to participate in the required commerce even if the user is not currently connected to the AR server 612 and / or the financial institution or distributor.

In one or more embodiments, the distributor (622A, 622B, etc.) has a pre-established relationship with the AR server 612 and / or financial institution 620 that allows for this new conceptual scheme of making purchases through the AR device 62. Can have. It should be understood that the aforementioned embodiments are provided for illustrative purposes only, and other embodiments may include more or fewer components.

Here, with reference to FIG. 7, an exemplary flowchart for carrying out a commercial transaction through the AR device 62 is provided. At 702, input regarding the transaction may be received. For example, a user may explicitly indicate that he or she is interested in purchasing an item (eg, through commands, gestures, etc.). In another embodiment, the AR system may propose a purchase to the AR user and receive confirmation from the user based on past purchases, user interests, and the like. In yet another embodiment, the AR system may assess interest based on a "heat map" of various products. More specifically, since the AR device 62 knows where and when the user is looking, the AR device 62 allows the user to determine various virtual and virtual interests in the product. / Or it may be possible to determine how long you have been looking at the real object. For example, if the user is viewing a virtual advertisement of a particular brand, the AR device 62 may measure the user's interest by determining how long the user has been viewing the particular product. In one or more embodiments, the AR system may generate a heatmap based on the amount of time one or more users have been viewing a particular product. If the heatmap indicates an interest in a particular product (eg, the amount of time spent looking at a particular product exceeds a predetermined threshold amount of time), the AR device 62 requests confirmation from the AR user about the purchase of the product. obtain.

At 704, the AR device 62 may perform a user identification protocol. There can be many types of user identification protocols, as described in more detail below. In one or more embodiments, the AR device 62 may simply require a "password" based on eye movement to determine if the user is validated. In another embodiment, the AR device 62 can capture a picture of the user's iris to see if the user is a valid user of the AR device 62 (and the account linked to the AR device 62). In another embodiment, the AR device 62 may monitor the continuity of the AR device 62 that remains on the user's head (eg, if the user has not removed the AR device 62 at all, the user is the same. Probability is high). In one or more embodiments, the AR device 62 periodically captures the iris image based on the user identification protocol or to ensure that the user is a verified user of the AR device 62. The test can be performed periodically. As discussed herein, there are many ways to identify a user through biometric data, and some exemplary methods will be further described below.

In some embodiments, the identification protocol can be a constant identification of the user (eg, eye movement patterns, skin contact, etc.). In other embodiments, the identification protocol may simply be a one-time identification (through any identification method). Thus, in some embodiments, once the AR system identifies a user, the same user will be identified unless an intervention event occurs (eg, the user removes the AR device 62, disrupts network connectivity, etc.). It may not need to be done. At 705, the AR device 62 may determine whether the identification protocol requires the capture of biometric data. If the user protocol requires that the biometric data be captured, the AR device may capture the biometric data. Otherwise, the AR device 62 may proceed with user identification through a non-biometric capture identification method.

At 706, biometric data can be captured based on the user identification protocol. For example, if the user identification protocol is an eye test for detecting known patterns, the AR device 62 may track the movement of the user's eye through one or more eye tracking cameras. The captured movement can be correlated with a "password" or signature eye movement to determine if the user is validated. Alternatively, if the user identification protocol is iris capture, the image of the iris can be captured and correlated with a known image of the user. Alternatively, each time the AR user wears the AR device 62, an iris capture or eye test may be performed to verify the user's identification. It should be understood that the biometric data can be eye-related or other types of biometric data in some embodiments. For example, the biometric data can be voice in one or more embodiments. In one or more embodiments, the biometric data can be eyelash-related data or eye shape data. Any type of biometric data that can be used to uniquely identify a user from other users can be used.

At 708, biometric data can be compared with predetermined user identification data to identify the user. Alternatively, if the user identification does not require biometric data, the AR device 62 determines, for example, that the user has not removed the AR device 62, and thus the user is identical to the previously identified user. Can be shown. If the user is identified, the AR device 62 proceeds to 710 and transmits the information to one or more financial institutions.

If the user is not identified, the AR device 62 may perform another user identification protocol or otherwise prevent the user from conducting a transaction. If the user is identified, data about the desired product may be transmitted to the cloud and financial institutions at 710. For example, according to the above example, information about the desired shoe (eg, product number, desired quantity, information about the user, shipping address, user account, etc.) may be communicated to the seller and / or financial institution.

At 712, the AR system may receive confirmation from the financial institution that the payment has been completed and / or approved. At 714, a confirmation message may be displayed to the user to confirm that the purchase is complete.

As mentioned above, in one or more embodiments, one approach to identifying users (for financial transactions and other purposes) for verification purposes is to perform user identification tests periodically. obtain. In one or more embodiments, the user identification method may utilize eye-related data to complete the user identification test. Since the AR device 62 is equipped with an eye tracking camera that continuously tracks the movement of the user's eyes, known patterns of eye movement can be used as eye tests to recognize and / or identify the user. For example, passwords can be easily copied or stolen, but it is difficult to replicate the eye movements or other physiological properties of other users, making it easy to identify unauthorized users of the AR device 62. obtain.

In one or more embodiments, during the configuration of the AR device 62, the system may use the user's input to configure a known pattern of eye movement unique to the user (ie, similar to an eye password). This known pattern of eye movement can be memorized and correlated each time a user identification protocol is performed.

Here, with reference to embodiment 800 of FIGS. 8A and 8B, an exemplary eye pattern 802 of the user's eye 804 is provided. As shown in FIG. 8A, the AR device 806 may track the user's eye movement through an eye tracking camera (not shown) and correlate the pattern with a known pattern of eye movement (ie, eye password). .. If the eye movement pattern is close to a known pattern (within a threshold), the AR device 806 may allow the user to perform a transaction. As shown in FIG. 8A, the user can move his eyes in the pattern shown. For illustration purposes, a line (802) is drawn to represent the eye pattern. Of course, in reality, there are no lines, and the eye tracking device will simply track such movements and convert it to the desired data format.

In one or more embodiments, a grid 904 similar to that shown in FIG. 8B may be utilized to determine if the tracked eye pattern correlates with a predetermined known pattern of eye movement. It should be understood that other such techniques can be used as well. Through the use of grid 904, by dividing the available space into discretized areas, it can be facilitated to determine if the tracked eye pattern 802 is most similar to a given pattern 902. For example, as shown in FIG. 8B, the tracked eye pattern 802 more or less follows a predetermined pattern 902 (as indicated by a thick line connecting the centers of each grid square associated with the predetermined pattern). .. FIG. 8B represents a fairly simplified version of the grid 904, but it should be understood that the size of each grid can be reduced for more accurate determination.

In the illustrated embodiment, the pattern can be recognized if the tracked eye movement targets an area of a predetermined grid square. In other embodiments, most of the grid squares may need to be hit before the user is considered to have passed the user identification test. Similarly, other such thresholds can be devised for various eye movement tracking protocols. In one or more embodiments, blink patterns can be utilized as well, as described above. For example, rather than utilizing an eye movement pattern, the eye password can be a series of blinks or blinks combined with a movement to track a user's signature.

Here, with reference to FIG. 9, an exemplary process for detecting an eye movement signature is described. At 902, an eye movement test can be initiated. For example, the AR user may indicate that he wants to purchase the goods, or the AR user may remove the AR devices 62, 806 and resume wearing the AR devices 62, 806. In another embodiment, the eye movement test may be performed cyclically for security purposes.

At 904, eye movement patterns can be tracked and received. For example, a virtual display screen may display a "password entry" command, which may trigger the user to form a known pattern with his eyes.

At 906, the tracked eye movements can be transformed into a particular data format. For example, returning to the grid approach and referencing, the data may indicate the coordinates of the grid hit by eye movement. Many other approaches can be used as well.

At 908, the transformed data can be compared with predetermined data of known signature eye movement patterns. At 910, the AR system can determine if the tracked eye movements match a predetermined pattern within a threshold. In 912, if it is determined that the eye pattern does not match the known eye pattern within the threshold, the user may fail the test and be prevented from making a purchase or need to take the test again. possible. At 914, the user may be allowed to pass the test and make a purchase if it is determined that the eye pattern matches a known eye pattern within a threshold.

In yet another approach, rather than performing an eye test, the AR system periodically captures pictures of the AR user's eyes and compares the captured images of the user's eyes with known information. Can perform eye identification. In one or more embodiments, the AR devices 62,806 may require the user to stare at a particular virtual object presented to the user when the user intends to make a purchase. This allows the user's eye to rest and images of the user's eye can be captured and compared. If the photo of the eye correlates with a known photo of the user's eye, the AR user may be allowed to make a purchase. Further details regarding the eye identification technique are provided under the patent attorney reference number ML3002.80, entitled "DEVICES, METHODS AND SYSTEMS FOR BIOMETRIC USER RECOGNITION UTILIZING NEURAL NETWORKS", co-pending application No. 59/62/15. To.

This feature is advantageous because the AR system generally needs to know where the user's eyes are looking (or "seeing") and where the user's eyes are focused. Can be used for identification purposes. Thus, in various embodiments, the head-mounted display (“HMD”) component features one or more cameras that are directed to capture image information about the user's eyes. In one configuration, such as those depicted in FIG. 5, each user's eye has a camera focused on it, as well as three or more with a known offset distance to the camera for inducing a flash onto the surface of the eye. The LED may be (in one embodiment, just below the eye, as shown).

The presence of three or more LEDs with a known offset to each camera makes it possible to determine the distance from the camera to each flash point in 3D space by triangulation. Using at least three flash points and a substantially spherical model of the eye, the system can infer the curvature of the eye. Using known 3D offsets and eye orientations, the system provides accurate (eg, image) or abstract (eg, gradient or other features) templates for the iris or retina for use to identify the user. Can be formed. In other embodiments, other characteristics of the eye, such as patterns of veins in and above the eye, can also be used to identify the user (eg, with an iris or retinal template).

In one approach, the iris image identification approach can be used. The pattern of muscle fibers within the iris of the eye forms a stable and unique pattern for each individual, including spots, grooves, and rings. Various iris features can be easily captured using infrared or near-infrared imaging as compared to visible light imaging. The system can convert captured iris features into identification codes in many different ways. The goal is to extract a sufficiently abundant tissue structure from the eye. With sufficient degrees of freedom in the collected data, the system can theoretically identify a unique user.

In another approach, retinal image identification can be used as well. In one embodiment, the HMD comprises a diffractive display driven by a laser scanner steered by steerable fiber optic cables. This fiber optic cable can also be used to visualize the interior of the eye and image the retina with unique patterns of visual receptors (rod and pyramidal cells) and blood vessels. These rod and pyramidal cells can also form patterns that are unique to each individual and can be used to uniquely identify each individual.

For example, each person's dark and clear blood vessel patterns are unique, such as applying a gradient operator to a retinal image and counting elevation transitions within a standardized grid centered on the retina. It can be converted to a "dark-bright" code by a retinal technique.

Therefore, the system of this subject, with a high degree of accuracy and accuracy, by comparing the user characteristics captured or detected by the system with the known baseline user characteristics for the approved users of the system. Can be used to identify.

In yet other embodiments, eye curvature / size can be used as well. For example, this information may aid in the identification of users because the eyes of different users are similar but not exactly the same. In other embodiments, transient biometric information can be collected when the user is under stress and correlated with known data. For example, the user's heart rate can be monitored, whether the user's eyes are producing a film of water, whether the eyes are approaching and focusing together, breathing patterns, blink rates, pulse rate, etc. Similarly, it can be used to verify and / or invalidate a user's identity.

In yet another embodiment, to confirm the user's identity (eg, if misidentification is suspected), the AR system captures information through the AR device (eg, through the field camera of the AR devices 62,806). It is possible to correlate (images of the surrounding environment) and determine if the user is looking at the same scene that correlates with the location as derived from the GPS and the map of the environment. For example, if the user is considered to be at home, the AR system can verify by correlating known objects at the user's home with what they are seeing through the user's field camera.

The AR / user identification system described above provides a very secure form of user identification. In other words, the system can be utilized to determine the user with a relatively high degree of accuracy and accuracy. The system can be used to identify users with a very high degree of certainty and on a continuous basis (eg, using periodic monitoring), requiring a separate login. It can be used to enable various financial transactions without. One approach to ensuring that the user identification system is very accurate is through the use of neural networks, co-pending application 62/159 under Patent Attorney Reference Number ML3002.80. , 593 for more details.

Here, with reference to FIGS. 10A-10I, an exemplary process flow 1000 using biometric data to carry out a transaction is illustrated. As shown in FIG. 10A, the user 1002 wearing the AR device 1004 walks into the store. While in the store, user 1002 may see a pair of shoes 1006 that may be of interest to purchase.

Here, with reference to FIG. 10B, an exemplary view of the shoe visible to user 1002 through the AR device 1004 is shown. Upon detecting that the user's gaze is focused on the pair of shoes 1006, the AR device 1004 looks up details about the pair of shoes 1006 (eg, through a product catalog synchronized with the pair of shoes 1004). , Details can be displayed as virtual content 1008. With reference to FIG. 10C here, the AR device 1004 may determine whether the user wants to purchase consumption by displaying the virtual content 1010. User 1002 may confirm or reject through any form of user input (eg, gesture, voice, eye control, etc.).

With reference to FIG. 10D here, assuming that the user decides to purchase, the AR device 1004 may request a password through the virtual content 1012. At this point, user 1002 may begin to generate eye signature 1016, as shown in FIG. 10E. In one or more embodiments, the virtual grid 1014 may be presented to the user to assist in moving the eyes in a particular manner.

As shown in FIG. 10F, the input signature 1016 may be received by the AR system 1004 and compared with a predetermined signature to determine if the user is an authenticated user. If the user is authenticated, the AR device 1004 may transmit data about the desired product through network 1018 to AR server 1020, distributor 1024 and financial institution 1022, as shown in FIG. 10G. Based on the confirmation received from the AR server 1020, the financial institution 1022 may transmit the appropriate amount to the distributor 1024.

As shown in FIG. 10H, when the transaction is confirmed, the AR device 1004 may display virtual content 1026 confirming the purchase of shoes 1006. Upon receipt of the confirmation, the user 1002 may leave the store with the desired shoe 1006, as shown in FIG. 10I.

It should be understood that the process flow of FIGS. 10A-10I is presented herein for illustrative purposes only and should not be read as a limitation, but merely represents an exemplary embodiment. Many other embodiments can be envisioned as well. For example, in one or more embodiments, instead of requiring a "password" (eg, FIG. 10D), the AR system requires the user to stare at the virtual dots on the screen and the user's eyes (eg, FIG. 10D). For example, images of the retina, iris, eye shape, etc.) can be captured. This image can then be correlated with a known image of the user's eye, confirming the user's identification. Once the user's identity is confirmed, the AR system may transmit information to distributor 1024 and financial institution 1022, as shown in FIG. 10G. Similarly, many other similar embodiments can be envisioned.

As mentioned above, it should be understood that such authentication and payment systems make trading much easier than traditional payment models. Rather than a long and painstaking round trip to a department store, shopping becomes a "playground" experience, where users simply walk into the store, pick up any number of products, and simply leave the store. I can go. The AR system undertakes most of the payment details, while requiring only simple and unobtrusive identification checks based on easily tracked biometric data. As mentioned above, the identification check does not require any user action at the time of purchase, according to some embodiments.

As discussed in detail above, traditional passwords or sign-up / login codes can be excluded from individual secure transactions using the AR / user identification systems and methods described above. The system of the subject can pre-identify / pre-authenticate the user with a very high degree of certainty. In addition, the system can use periodic monitoring to maintain user identification over time. Thus, the identified user can have instant access to any site after being notified about the terms of that site (which can be displayed to the user as an overlaid user interface item). In one embodiment, the system may create a set of standard provisions pre-determined by the user so that the user instantly understands the conditions on the site. If the site does not comply with a set of these Terms (eg, standard terms), the system in this subject may not automatically allow access or transactions there.

In addition, the AR / user identification system described above can be used to facilitate "small amount transactions". Small transactions that generate only very small debits and credits in a user's financial account are typically about a few cents or less than a cent. On a given site, the system of the subject is not only the content viewed or used by the user, but also the time (short browsing can be free, but beyond a certain amount will be charged). Can be configured to confirm. In various embodiments, news articles can be 1/3 cents, books can be charged 1 penny per page, music can be 10 cents per viewing, and so on. In another embodiment, the advertiser may pay the user 0.5 cents for selecting a banner ad or conducting a survey. The system may be configured to allocate a small percentage of the transaction fee to the service provider.

In one embodiment, the system can be used to create a dedicated small transaction account that can be controlled by the user, and the funds associated with the small transaction are aggregated and the user's more traditional financial account as a predetermined meaningful amount. Distributed to / from (eg, online bank accounts). Small trading accounts can be cleared or deposited at regular intervals (eg quarterly) or in response to certain triggers (eg when a user spends more than a few dollars on a particular website).

Small-value transactions are typically impractical and cumbersome in traditional payment conceptual schemes, but as described in the payment techniques described herein, transactions are near-instantaneous user identification and authentication. The ease that arises through eliminates many of the obstacles typically associated with small-value transactions. This can open up new means of monetization for future businesses. For example, it may be easier to monetize music, books, advertisements and the like. Users can hesitate to pay $ 1 for news articles, but they may not hesitate too much for articles that cost a fraction of a cent. Similarly, given that it is significantly easier to carry out these transactions (small and high), advertisers, publishers, etc. are more likely to develop content for different types of payment schemes. Can increase. Therefore, the AR system facilitates both payment and content delivery, thereby making both the front-end and back-end processes relatively simple.

The system and functionality of the subject can be provided by companies focused on augmented reality, and the user's identity is so reliably and securely known that the user has instant access to that account. , Amounts, spending, spending rates, and 3D views of graphical and / or geographical maps of that spending may be provided. Such users may be able to instantly adjust spending access, such as turning spending (eg, small transactions) off and on.

Due to high spending (ie, a few penny or a few dollars, not a percentage of penny), various embodiments may be facilitated using the system configuration of the subject.

The user may use the system to order perishables for delivery to its tracked location or user-selected map location. The system can also notify the user when a delivery will arrive (eg, by viewing a video of the delivery taking place within the AR system). With AR telepresence, a user is physically located in an office away from their home, but a courier enters the home, appears to the courier by avatar telepresence, and the courier delivers the product. You can see the delivery, then confirm that the courier has left, and lock the door of your home with avatar telepresence.

Optionally, the system may memorize the user's product preferences and alert the user to a sale or other promotion related to the user's preferred product. For these high spending embodiments, the user can review the account overview, all statistics for the account, and purchasing patterns, thereby facilitating a shopping comparison before placing the order.

Since the system can be used to track the eyes, it can also allow "at a glance" shopping. For example, suppose a user looks at an object (for example, a robe in a hotel) and says, "I want this if the amount in my account exceeds $ 3,000." The system will make a purchase when certain conditions (eg, the account balance exceeds $ 3,000) are met.

As mentioned above, in one or more embodiments, the iris and / or retinal signature data can be used to secure communication. In such embodiments, the system of the subject only allows the user to authenticate based on one or more dynamically measured iris and / or retinal signatures. It may be configured to allow it to be selectively transmittable and viewable on a trusted secure hardware device that allows access. Since the AR system display device projects directly onto the user's retina, only the intended recipient (identified by the iris and / or retina signature) may be able to view the protected content, and in addition, the viewing device. Actively monitors the user's eye, so dynamically read iris and / or retinal signatures can be recorded as proof that the content was actually presented to the user's eye (eg, perhaps eye movement). As a form of digital reception performed by verification actions, such as performing the required sequence of.

Spoofing detection can rule out attempts using previous recordings of retinal images, static or 2D retinal images, generated images, etc., based on a model of expected natural variation. Unique criteria / watermarks can be generated and projected onto the retina to generate unique retinal signatures for auditing.

The financial and communication systems described above are provided as examples of various common systems that can benefit from more accurate and precise user identification. Therefore, the use of the AR / user identification system described herein is not limited to the disclosed financial and communication systems, but rather is applicable to any system that requires user identification.

Various exemplary embodiments of the invention are described herein. In a non-limiting sense, these examples are referenced. They are provided to illustrate the broader and applicable aspects of the invention. Various modifications can be made to the invention described and equivalents can be replaced without departing from the true spirit and scope of the invention. In addition, many modifications can be made to adapt a particular situation, material, material composition, process, process action, or step to the object, spirit, or scope of the invention. Moreover, as will be appreciated by those skilled in the art, each of the individual variants described and illustrated herein will be of some other embodiment without departing from the scope or spirit of the invention. It has individual components and features that can be easily separated from or combined with any of the features. All such amendments are intended to be within the claims associated with this disclosure.

The present invention includes methods that can be performed using the subject device. The method may include the act of providing such a suitable device. Such provision may be made by the end user. In other words, the act of "providing" simply acquires, accesses, approaches, positions, configures, launches, powers on, and provides the end user with the essential device in the subject method. Or require it to act differently. The methods described herein can be performed in any order in which the events are described, which is logically possible, and in the order in which the events are described.

Illustrative aspects of the invention are described above, along with details regarding material selection and manufacture. With respect to other details of the invention, they are understood in connection with the patents and publications referenced above, and are generally known or can be understood by those skilled in the art. The same may be true with respect to the method-based aspects of the invention in terms of additional actions that are generally or logically adopted.

In addition, while the invention is described with reference to some examples that optionally incorporate various features, the invention is described and directed as considered for each variant of the invention. It is not limited to things. Various modifications may be made to the invention described and are included for some simplification, as described herein, without departing from the true spirit and scope of the invention. (Whether or not) can be replaced. In addition, if a range of values is provided, all intervening values between the upper and lower bounds of the range, and any other provisions or intervening values within that defined range, are included within the invention. Please understand that.

Also, any optional features of the embodiments of the invention described are described and claimed independently or in combination with any one or more of the features described herein. To get is considered. References to singular items include the possibility that the same plural item exists. More specifically, as used herein and in the claims associated with this specification, "one (" a "," an ")", "said", and " The singular form "the (the)" includes the plural directive reference unless otherwise specified. In other words, the use of articles allows for "at least one" subject item in the above description as well as in the claims associated with the present disclosure. Moreover, it should be noted that such claims may be drafted to exclude any voluntary elements. Therefore, this statement is antecedent for the use of exclusive terms such as "simply", "only", and equivalents, or the use of "negative" restrictions in connection with the description of the elements of the claim. The purpose is to fulfill the function as.

Without using such exclusive terms, the term "equipped" in the claims associated with this disclosure is such that a given number of elements are listed or featured in such claims. It shall be possible to include any additional elements, regardless of whether the addition of may be considered as a transformation of the nature of the elements described in such claims. Except as specifically defined herein, all technical and scientific terms used herein are generally understood as broadly as possible while maintaining the validity of the claims. It is given the meaning of being.

The scope of the present invention is not limited to the specification of the examples and / or the subject matter provided, but rather to the scope of the claims associated with the present disclosure.

Claims (19)

A method of conducting a transaction through an AR system that includes a local processing module, a remote server and an augmented reality (AR) device.
At least one camera of the AR device captures biometric data from the user wearing the AR device.
The local processing module communicating with the AR device is at least partially based on the analysis of the captured biometric data by the local processing module executing an eye movement based user identification protocol. The eye movement-based user identification protocol is to determine the identification of
Displaying an instruction on the AR device to prompt the user to enter an eye movement-based password by moving the gaze according to a predetermined eye signature movement pattern.
The local processing module tracks the eye movement-based password entered by the user in response to the displayed instruction, wherein the AR device is the user of the eye movement-based password. Being captured through at least one camera of the AR device while worn on the head,
In the AR device, the local processing module compares the tracked eye movement with the predetermined eye signature movement pattern , where the tracked eye movement differs from the predetermined eye signature movement pattern. The captured biometric data of the
The local processing module is to transmit communication to a remote server to initiate a transaction to purchase an object when the tracked eye movement matches the predetermined eye signature movement pattern. That the communication includes said identification of the user.
The remote server communicating with the local processing module through the first network determines that the tracked eye movement of the gaze corresponds to the predetermined eye signature movement pattern. A method comprising authenticating the user for the transaction based on the identification. The method of claim 1, wherein the remote server further comprises transmitting a set of data relating to the transaction through a second network to a computer of a financial institution. The method of claim 1, wherein the biometric data further comprises an iris pattern of the user's eye on the AR device. The method of claim 1, wherein the biometric data further comprises a voice recording of the user. The method of claim 1, wherein the biometric data is the retinal signature of the user of the AR device. The method of claim 1, wherein the AR device further comprises capturing the biometric data, including properties associated with the user's skin. The method of claim 1, wherein the biometric data detected by the at least one camera is a pattern of movement of the user's eyes with respect to a virtual grid displayed through the AR device. The method of claim 1, wherein the biometric data further comprises a blink pattern of the user's eyes. The method of claim 1, wherein the AR device is head-mounted and the AR device is individually calibrated for the user. The method of claim 1, wherein determining the identification of the user is at least partially based on a known iris pattern. The method of claim 1, wherein determining the identification of the user is at least partially based on a known retinal pattern. The method according to claim 1, wherein the transaction is a commercial transaction. The remote server further comprises communicating the user's authentication to the computer of the financial institution associated with the user, the computer of the financial institution at least partially to the authentication received from the remote server. The method of claim 1, wherein payment is made on behalf of the user based on. 13. The method of claim 13, wherein the computer of the financial institution transmits the payment to the computer of one or more sellers indicated by the user. When the local processing module detects an interruption event or transaction event associated with the AR device,
Claimed that the AR device captures new biometric data from the user in order to reauthenticate the user based on the detected event and the new biometric data at least in part. Item 1. The method according to item 1. 15. The method of claim 15, wherein the interruption event is detected by the local processing module, at least partially based on the removal of the AR device from the user's head. 15. The method of claim 15, wherein the interruption event is detected by the local processing module, at least in part, based on the loss of connectivity of the AR device to the network. 15. The method of claim 15, wherein the trading event is detected by the local processing module, at least in part, based on a heatmap associated with the user's gaze on an object presented through the AR device. The method of claims 1-18, which is implemented as a computer program product comprising a computer-enabled storage medium having executable code for performing the method steps.

Images