TW202114409A - Method and system for setting presentable virtual object for target - Google Patents

Abstract

A method and system for setting a presentable virtual object for a target, wherein a camera and an optical communication device are installed in the augmented reality environment, and the camera and the optical communication device have a relative pose, the method includes: The camera tracks the target in the augmented reality; obtains the position signal of the target according to the tracking result of the camera; sets a virtual object associated with the target with a spatial position signal, wherein the spatial position signal of the virtual object is based on the location signal of the target is determined; and the related signal of the virtual object is sent to the first device, the signal includes the spatial location signal of the virtual object, wherein the related signal of the virtual object can be used by the first device. The virtual object is presented on the display medium based on the position signal and the posture signal determined by the optical communication device.

Description

Method and system for setting presentable virtual object for target

The present invention belongs to the field of augmented reality technology, and in particular relates to a method and system for setting a presentable virtual object for a target in a real scene.

The statements in this section are only to provide background signals related to the present invention to help understand the present invention, and these background signals do not necessarily constitute prior art.

In recent years, Augmented Reality (AR) technology has made great progress. Augmented reality technology is also known as mixed reality technology. It superimposes virtual objects into the real scene through computer technology, so that the real scene and virtual objects can be presented in the same screen or space in real time, thereby enhancing the user’s perception of the real world. Perception. Because augmented reality technology can enhance the display output of the real environment, it has gained more and more widespread use in medical research and anatomy training, precision instrument manufacturing and maintenance, military aircraft navigation, engineering design, and remote robot control. application.

In an augmented reality application, some data signals can be superimposed on a fixed position in the field of view. For example, when a pilot is flying an airplane, he can view the flight data superimposed on the real scene by wearing a display helmet. They are all displayed at a fixed position in the field of view (for example, always in the upper right corner of the field of view), and therefore lack sufficient flexibility. In particular, the superimposed virtual object cannot move with the movement of the real target in the real scene. In another augmented reality application, some virtual objects can be superimposed near the person photographed by the user's mobile phone, but this method is not accurate enough and is not suitable for AR interaction between multiple persons.

The solution of the present invention provides a method and system for setting a presentable virtual object for a target in a real scene, which uses an optical communication device and a camera to make It is necessary to set the associated virtual object based on the position signal of the target in the real scene, the virtual object can be accurately presented on the display medium of the device, and the presented virtual object can follow the target in the real scene.

One aspect of the present invention relates to a method for setting a presentable virtual object for a target in a real scene, wherein a camera and an optical communication device are installed in the real scene, and there is an optical communication device between the camera and the optical communication device. Relative to the pose, the method includes: using the camera to track a target in the real scene; obtaining a position signal of the target according to the tracking result of the camera; setting a spatial position signal associated with the target A virtual object, wherein the spatial position signal of the virtual object is determined based on the position signal of the target; and a signal related to the virtual object is sent to the first device, and the signal includes the spatial position signal of the virtual object Wherein, the related signal of the virtual object can be used by the first device to present the virtual object on its display medium based on the position signal and the posture signal determined by the optical communication device.

Optionally, wherein the obtaining the position signal of the target according to the tracking result of the camera includes: obtaining the position signal of the target relative to the camera according to the tracking result of the camera, and obtaining the position signal of the target relative to the camera according to the tracking result of the camera; The position signal of the camera and the relative pose between the camera and the optical communication device are used to determine the position signal of the target relative to the optical communication device.

Optionally, wherein the spatial position signal of the virtual object is a spatial position signal relative to the optical communication device.

Optionally, wherein the obtaining the position signal of the target according to the tracking result of the camera includes: obtaining the position signal of the target relative to the camera according to the tracking result of the camera, and obtaining the position signal of the target relative to the camera according to the tracking result of the camera; The position signal of the camera and the pose signal of the camera in the real scene are used to determine the position signal of the target in the real scene.

Optionally, wherein the related signal of the virtual object is configured according to the signal related to the target.

Optionally, the method further includes: obtaining a signal related to the target according to the tracking result of the camera.

Optionally, wherein the configuring the signal related to the virtual object according to the signal related to the target includes: receiving a signal related to the target from a facility; Establishing a connection with a target at a predetermined location of the facility; and configuring the relevant signal of the virtual object of the target according to the signal.

Optionally, wherein the configuring the related signal of the virtual object according to the signal related to the target includes: receiving a signal from a second device, the signal including a position signal of the second device; The position signal of the second device is compared with the position signal of one or more targets determined according to the tracking result of the camera to determine a target that matches with the second device; and according to the signal configuration from the second device and The related signal of the virtual object of the target matched by the second device.

Optionally, wherein the second device determines its position signal at least in part by collecting an image including the optical communication device and analyzing the image.

Optionally, the method further includes: obtaining a posture signal of the target according to the tracking result of the camera; and wherein the setting a virtual object associated with the target with a spatial position signal includes The gesture signal of sets the gesture signal of the virtual object.

Optionally, wherein the posture of the virtual object can be adjusted according to a change in the position and/or posture of the first device relative to the virtual object.

Optionally, wherein the first device at least partially collects an image including the optical communication device and analyzes the image to determine its position signal and attitude signal relative to the optical communication device.

Optionally, wherein the signal related to the virtual object to be sent to the first device is determined according to the signal related to the first device.

Another aspect of the present invention relates to a system for setting a presentable virtual object for a target in a real scene, including: a camera installed in the real scene for tracking the target in the real scene; installation The optical communication device in the real scene, wherein the optical communication device and the camera have relative poses; and a server for executing the above method.

Optionally, the system further includes information that can obtain information related to the target. No. of facilities.

Optionally, the system further includes the first device, configured to: receive a signal related to the virtual object from the server; determine the location of the first device at least in part through the optical communication device A signal and a posture signal; and based on the position signal, the posture signal and the related signal of the virtual object, the virtual object is presented on the display medium of the first device.

Another aspect of the present invention relates to a storage medium in which a computer program is stored, and when the computer program is executed by a processor, it can be used to implement the above method.

Another aspect of the present invention relates to an electronic device, including a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, the computer program can be used to implement the above method.

100: light label

101: The first light source

102: second light source

103: third light source

S301~S304: Step flow

Figure 1 shows an exemplary optical label.

Fig. 2 shows a real-world scene including a system for setting a presentable virtual object for a target according to one embodiment.

Fig. 3 shows a method for setting a presentable virtual object for a target according to an embodiment.

Fig. 4 shows a schematic image observed by the first user in Fig. 2 through a mobile phone screen or AR glasses according to an embodiment.

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail through specific embodiments with reference to the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention.

Optical communication devices are also called optical tags, and these two terms can be used interchangeably in this article. Optical tags can transmit signals through different light-emitting methods, which have the advantages of long recognition distance and relaxed requirements for visible light conditions, and the signals transmitted by optical tags can change over time, which can provide large signal capacity and flexible configuration capabilities. Compared with the traditional two-dimensional bar code, The optical tag has a longer recognition distance and stronger signal interaction capability, which can provide users with great convenience.

The optical tag usually includes a controller and at least one light source, and the controller can drive the light source through different driving modes to transmit different signals to the outside. Fig. 1 shows an exemplary optical label 100, which includes three light sources (respectively a first light source 101, a second light source 102, and a third light source 103). The optical label 100 also includes a controller (not shown in FIG. 1), which is used to select a corresponding driving mode for each light source according to the signal to be transmitted. For example, in different driving modes, the controller can use different driving signals to control the light emitting mode of the light source, so that when a device with imaging function is used to photograph the light label 100, the imaging of the light source therein can show a different appearance. (For example, different colors, patterns, brightness, etc.). By analyzing the imaging of the light source in the optical label 100, the driving mode of each light source at the moment can be analyzed, so as to analyze the signal transmitted by the optical label 100 at the moment.

In order to provide users with corresponding services based on optical tags, each optical tag can be assigned an identification signal (ID), which is used to uniquely identify or identify the optical tag by the manufacturer, manager, or user of the optical tag . Generally, the light source can be driven by the controller in the optical tag to transmit the identification signal outward, and the user can use the device to collect the image of the optical tag to obtain the identification signal transmitted by the optical tag, so that the corresponding identification signal can be accessed based on the identification signal. Services, such as accessing web pages associated with the identification signal, obtaining other signals associated with the identification signal (for example, the position signal of the optical tag corresponding to the identification signal), and so on. The devices mentioned in this article can be, for example, devices carried or controlled by users (for example, mobile phones, tablets, smart glasses, AR glasses, smart helmets, smart watches, etc.), or machines that can move autonomously (for example, drones). , Driverless cars, robots, etc.). The device can collect the image of the optical label through the camera on it to obtain the image containing the optical label, and identify the signal transmitted by the optical label by analyzing the imaging of the optical label (or each light source in the optical label) in the image .

The identification signal (ID) or other signals of each optical label can be saved on the server, such as the service signal related to the optical label, the description signal or attribute related to the optical label, such as the position signal and model signal of the optical label , Physical size signal, physical shape signal, attitude or orientation signal, etc. Optical tags can also have a uniform or default physical size signal and physical Shape signal, etc. The device can use the identified identification signal of the optical label to query the server to obtain other signals related to the optical label. The position signal of the optical tag can refer to the actual position of the optical tag in the physical world, which can be indicated by the geographic coordinate signal. The server can be a software program running on a computing device, a computing device, or a cluster composed of multiple computing devices. The optical tag can be offline, that is, the optical tag does not need to communicate with the server. Of course, it can be understood that online optical tags that can communicate with the server are also feasible.

Figure 2 shows a real scene including a system for setting presentable virtual objects for a target according to an embodiment. The real scene can be, for example, a bank branch, in which there are three users, namely the first user and the second user. The user and the third user, the first user may be a bank branch staff, and the second user and the third user may be bank customers. The first user carries a device capable of recognizing the optical tag, such as a mobile phone or AR glasses.

The system includes a camera, an optical tag, and a server (not shown in FIG. 2), wherein the camera and the optical tag are each installed in a real scene in a specific position and posture (hereinafter collectively referred to as "pose"). In one embodiment, the server may obtain the respective pose signals of the camera and the optical tag, and may obtain the relative pose signals between the camera and the optical tag based on the respective pose signals of the camera and the optical tag. In one embodiment, the server can also directly obtain the relative pose signal between the camera and the optical tag. In this way, the server can obtain a transformation matrix between the camera coordinate system and the optical label coordinate system, and the transformation matrix may include, for example, a rotation matrix R and a displacement vector t between the two coordinate systems. Through the transformation matrix between the camera coordinate system and the optical label coordinate system, the coordinates in one coordinate system can be converted to the coordinates in another coordinate system. In one embodiment, when the camera and the optical tag are installed, the pose signals of the two can be manually calibrated, and the pose signals can be stored in the server. The camera may be a camera that is installed at a fixed position and has a fixed orientation. However, it is understood that the camera may also be a movable (for example, the position or direction can be changed), as long as the current position and attitude signal can be determined. The current pose signal of the camera can be set by the server, and the movement of the camera can be controlled based on the pose signal, or the movement of the camera can be controlled by the camera itself or other devices, and the current pose signal of the camera can be sent to the server. In some embodiments, the system may include more than one camera, or more than one optical tag.

In one embodiment, a scene coordinate system (which may also be referred to as a real world coordinate system) can be established for the real scene, and the camera coordinate system and the scene coordinate system can be determined based on the pose signal of the camera in the real scene The transformation matrix of the light tag and the transformation matrix between the light tag coordinate system and the scene coordinate system are determined based on the pose signal of the light tag in the real scene. In this case, the coordinates in the camera coordinate system or the optical label coordinate system can be converted to the coordinates in the scene coordinate system, instead of transforming between the camera coordinate system and the optical label coordinate system, but it is understandable that the camera and the optical label The relative pose signal or transformation matrix can still be known by the server. Therefore, in this application, having a relative pose between the camera and the light tag means that there is a relative pose between the two objectively, and does not require the system to pre-store the relative pose signal between the two or use The relative pose signal. For example, in one embodiment, only the pose signals of the camera and the light tag in the scene coordinate system may be stored in the system, and the relative poses of the two may not be calculated or used.

The camera is used to track a target in a real scene. The target may be stationary or moving. For example, it may be a person, a stationary object, a movable object, etc. in the scene. For the scene shown in Figure 2, the system can track the positions of the first user, the second user, and the third user through a camera. The camera may be, for example, a monocular camera, a binocular camera, or other forms of camera. The camera can be used to track the position of a person or an object in a real scene through various methods in the prior art. For example, in the case of using a single monocular camera, the scene signal (for example, the signal of the plane where the person or object in the scene is located) can be combined to determine the position signal of the target in the scene. In the case of using a binocular camera, the position signal of the target can be determined according to the position of the target in the field of view of the camera and the depth signal of the target. In the case of using multiple cameras, the position signal of the target can be determined according to the position of the target in the field of view of each camera.

In one embodiment, the process of determining the position signal of the target in the scene can be executed by the camera, and the corresponding result can be sent to the server. In another embodiment, the server may determine the position signal of the target in the scene according to the image taken by the camera. The server can convert the determined position signal of the target into a position signal in the optical label coordinate system or the scene coordinate system. After obtaining the position signal of the target, the server can set the virtual object with spatial position signal associated with the target, and the space of the virtual object The location signal can be determined based on the location signal of the target. For example, the spatial position of the virtual object can be set to be directly above the corresponding target, or to be located in another location near the target.

After the virtual object is set, the server can send the relevant signal of the virtual object to a device that can recognize the optical tag. For the scene shown in FIG. 2, the server may send the virtual object associated with the second user and the virtual object associated with the third user to the optical tag recognition device carried by the first user. The optical label recognition device has a camera and a display medium. The recognition device can determine the position and posture signal relative to the optical label by scanning the optical label. The position and posture signal can also be further converted into the position under the scene coordinate system. And gesture signals. After that, the recognition device can use the relevant signal of the virtual object to present the corresponding virtual object at a suitable position on its display medium based on its position signal and posture signal.

It can be understood that the second user or the third user may also carry the optical tag identification device, and can observe virtual objects associated with other users through the display medium of the optical tag identification device in a similar manner to the first user. The solution of the present invention is particularly suitable for multi-person AR interaction in the scene.

Fig. 3 shows a method for setting a presentable virtual object for a target according to an embodiment, which can be implemented using the above-mentioned system, and can include the following steps:

Step 301: Use the camera to track the target in the real scene.

By using the camera to continuously collect images for the real scene, it is possible to visually track the target existing in the real scene. Camera-based visual tracking technology involves detecting, extracting, identifying, or tracking targets in image sequences to obtain the target's position, posture, velocity, acceleration, or motion trajectory. The visual tracking technology belongs to the existing technology in the field, and will not be repeated here.

In one embodiment, when the camera is tracking a target, it can only perform continuous image collection and provide the collected images to the server as the tracking result. After that, the server can analyze these images and determine the target's Position signal. In another embodiment, the camera can also perform further processing on the collected images, such as image processing, target detection, target extraction, target recognition, determination of target position or posture, etc., and the corresponding processing The result can be provided to the server as the tracking result.

Step 302: Obtain the position signal of the target according to the tracking result of the camera.

The server may receive the tracking result from the camera, and obtain the position signal of the target in the real scene according to the tracking result. In one embodiment, the target position signal finally obtained by the server may be the target position signal in the camera coordinate system, the target position signal in the optical label coordinate system, or the target position signal in the scene coordinate system. The server can realize the conversion of the target position between different coordinate systems according to the transformation matrix between different coordinate systems. For example, the server can first obtain the position signal of the target relative to the camera (that is, the position signal in the camera coordinate system) according to the tracking result of the camera, and then can according to the position signal of the target relative to the camera and the difference between the camera and the optical label. To determine the position signal of the target relative to the light tag (that is, the position signal in the light tag coordinate system), it can also be based on the position signal of the target relative to the camera and the camera’s position in the real scene. The signal is used to determine the position signal of the target in the real scene (that is, the position signal in the scene coordinate system).

In one embodiment, in addition to obtaining the position signal of the target, the server may also obtain the posture signal of the target according to the tracking result of the camera, for example, the orientation of a person or an object.

Step 303: Set up a virtual object with a spatial position signal associated with the target.

After determining the location signal of the target, the server can set an associated virtual object for the target. The virtual object has a spatial location signal, which can be determined based on the location signal of the target. For example, the spatial position of the virtual object may be configured to be located at a predetermined distance above the target position. The spatial position signal of the virtual object can be, for example, its spatial position signal relative to the light tag, or its position signal in the scene coordinate system.

In one embodiment, in addition to configuring the spatial position signal of the virtual object, the server may also configure any other signals related to the virtual object. The signal related to the virtual object is the signal used to describe the virtual object. For example, it can include pictures, text, numbers, icons, etc. contained in the virtual object, and can also include the shape signal of the virtual object, Color signal, size signal, attitude signal, etc. Based on the signal, the device can present the corresponding virtual object. The server can configure the related signal of the virtual object corresponding to the target according to the signal related to the target, so that the corresponding virtual object can be customized for each target.

In one embodiment, the server can obtain the signal related to the target according to the tracking result of the camera. For example, the server can determine whether the target is a person or an object, whether the target is a man or a woman, what the target is, whether the target is moving or stationary, the moving speed of the target, the moving direction of the target, etc., according to the tracking result of the camera. These signals related to the target can be used to configure the related signals of the virtual object corresponding to the target.

In one embodiment, other facilities may also be used to obtain information related to the target. For example, fingerprint or vein collection equipment, face recognition equipment, ID card reading equipment and other facilities can be used to obtain personnel’s identity information, and card readers and other facilities can be used to obtain personnel’s gender information, occupational information, identity information, and membership Card information, etc. After these facilities have obtained personnel information, the server can receive the information from the facilities and place the information at a predetermined location currently tracked by the camera (for example, the front, back, left, right, upper, lower, nearby, etc. of these facilities) ) To establish a connection, so that the person’s information (for example, gender information, occupation information, identity information, membership card information, etc.) can be used to configure the relevant information of the virtual object corresponding to the person. Similarly, the related information of the object can also be obtained through the facilities in the scene, and the related information of the virtual object corresponding to the object can be configured with the information.

In one embodiment, the server can also set the posture signal of the virtual object. The posture signal of the virtual object can be set based on the posture signal of the target, but it can also be set in other ways.

Step 304: Send the relevant signal of the virtual object to a device capable of recognizing the optical tag, and the signal includes the spatial position signal of the virtual object.

After setting the virtual object for the target, the server can send the relevant signal of the virtual object to a device capable of recognizing the optical tag. The device may be a mobile phone, AR glasses, etc., which has a camera and a display medium. The related signal of the virtual object includes the spatial position signal of the virtual object. In one embodiment, it may also include the posture signal of the virtual object.

In one embodiment, the signal to be sent can be determined based on the signal related to the device. The relevant signal of the virtual object sent to the device. The signal related to the device may include the related signal of the device itself, the related signal of the user of the device, and so on. For example, according to different permissions or different levels of the device user, a part or all of the related signals of the virtual object can be selected and sent to the device.

In one embodiment, the optical label identification device can identify the signal (for example, identification signal) transmitted by the optical label by scanning the optical label arranged in the scene, and use the signal to access the server (for example, access via wireless signal) to obtain The server obtains the relevant signal of the virtual object. In one embodiment, the server may use the optical tag to send the relevant signal of the virtual object to the optical tag identification device in optical communication. The related signal of the virtual object can be used by the optical tag recognition device to present the virtual object on its display medium based on its position signal and posture signal determined by the optical tag.

The position and attitude signal of the device can be determined by the optical tag. The position and attitude signal can be the position and attitude signal of the device relative to the optical tag (that is, the position and attitude signal in the optical tag coordinate system), but it can also be It is the position and attitude signal converted to other coordinate system (such as scene coordinate system).

In one embodiment, the device can determine its position signal relative to the optical label by collecting an image including the optical label and analyzing the image. For example, the device can determine the relative distance between the optical label and the device through the image size of the optical label in the image and other optional signals (for example, the actual physical size signal of the optical label, the focal length of the device's camera) (the larger the image, the The closer the distance; the smaller the imaging, the farther the distance). The device may use the identification signal of the optical label to obtain the actual physical size signal of the optical label from the server, or the optical label may have a uniform physical size and store the physical size on the device. The device can determine the direction signal of the device relative to the optical label through the perspective deformation of the optical label imaging in the image including the optical label and optional other signals (for example, the imaging position of the optical label). The device may use the identification signal of the optical label to obtain the physical shape signal of the optical label from the server, or the optical label may have a uniform physical shape and store the physical shape on the device. In an embodiment, the device can also directly obtain the relative distance between the optical tag and the device through a depth camera or binocular camera installed on it. The device can also use any other existing positioning method to determine its position signal relative to the optical tag.

In one embodiment, the device can scan the optical tag and determine its attitude signal relative to the optical tag based on the imaging of the optical tag. When the imaging position or imaging area of the optical tag is located in the center of the imaging field of the second device, it can It is assumed that the second device is currently facing the optical tag. When determining the posture of the device, the imaging direction of the optical tag can be further considered. As the posture of the device changes, the imaging position and/or imaging direction of the optical tag on the device will change accordingly. Therefore, the posture signal of the device relative to the optical tag can be obtained according to the imaging of the optical tag on the device.

In an embodiment, the position and attitude signal of the device relative to the optical tag can also be determined in the following manner. Specifically, a coordinate system can be established based on the optical label, and this coordinate system can be referred to as the optical label coordinate system. Some points on the optical label can be determined as some spatial points in the optical label coordinate system, and the coordinates of these spatial points in the optical label coordinate system can be determined according to the physical size signal and/or physical shape signal of the optical label. Some points on the optical label may be, for example, the corners of the housing of the optical label, the end of the light source in the optical label, some identification points in the optical label, and so on. According to the object structure feature or geometric structure feature of the optical tag, the image points corresponding to these spatial points can be found in the image taken by the device camera, and the position of each image point in the image can be determined. According to the coordinates of each spatial point in the optical label coordinate system and the position of each corresponding image point in the image, combined with the internal reference signal of the device camera, the position of the device camera in the optical label coordinate system can be calculated when the image is taken. Attitude signal (R, t), where R is the rotation matrix, which can be used to represent the posture signal of the device camera in the optical label coordinate system, and t is the displacement vector, which can be used to represent the device camera’s position in the optical label coordinate system Position signal. The method of calculating R and t is known in the prior art. For example, the 3D-2D PnP (Perspective-n-Point) method can be used to calculate R and t. In order not to obscure the present invention, it will not be described in detail here. .

After the device obtains the position and attitude signals through the optical tag, for example, the built-in acceleration sensor, gyroscope, camera, etc. of the device can be used through methods known in the art (for example, inertial navigation, visual odometry, SLAM, VSLAM, SFM, etc.) ) To measure or track changes in the device's position and attitude, so as to obtain a new position and attitude signal of the device.

After obtaining the spatial position signal of the virtual object and the position and posture signal of the device, it can be positioned in the appropriate position in the real scene presented by the display medium of the device. Place the virtual object superimposed. In the case that the virtual object has a posture signal, the posture of the superimposed virtual object can be further determined. The posture of the virtual object can be adjusted with the position and/or posture of the device relative to the virtual object, for example, a certain direction of the virtual object (for example, the front direction of the virtual object) always faces the device. In one embodiment, a direction from the virtual object to the device may be determined in space based on the positions of the device and the virtual object, and the pose of the virtual object may be determined based on the direction. Through the above method, the same virtual object can actually have their own postures for devices in different positions.

In one embodiment, after the virtual object is superimposed, the device user can perform various interactive operations on the virtual object. For example, the device user can click on the virtual object to view its details, change the posture of the virtual object, change the size or color of the virtual object, add annotations on the virtual object, and so on. In one embodiment, after the device user changes the attribute of the virtual object, the modified attribute signal of the virtual object can be uploaded to the server. The server can update the related signal of the virtual object stored in it based on the modified attribute signal.

The method shown in FIG. 3 can be executed continuously or repeatedly executed periodically, so that the virtual object can always track the target in the scene.

In one embodiment, the tracking target in the real scene may be a device or a person holding the device, and a signal related to the corresponding virtual object may be set according to the related signal of the device. Specifically, the device can, for example, use the above-mentioned method to determine its location signal through the optical tag, and send the location signal to the server. The location signal can be the location signal of the device relative to the optical tag. The position signal sent by the device can be the position signal obtained when the device scans the optical tag, or it can be the device after scanning the optical tag using a built-in acceleration sensor, gyroscope, camera, etc. through methods known in the art (for example, inertial Navigation, visual odometer, SLAM, VSLAM, SFM, etc.) measure or track the new position signal obtained. In addition, the device can also send some other signals to the server, such as device identification number, device user name, device owner’s occupational information, device owner’s identity information, device owner’s gender information, and device owner’s age Information, account information of an application on the device, signals related to an operation performed by the device, etc. The server can combine the position signal of the device with one or more targets determined according to the tracking result of the camera. The location signal of the target is compared to determine the target that matches the device. For example, the server may select the position signal of one or more targets that is closest to the position signal of the device, and consider that the corresponding target matches the device. After determining the target matching the device, the server can configure the relevant signal of the virtual object corresponding to the target according to the signal from the device. Through the above method, the equipment in the scene (for example, robots, unmanned vehicles, etc.) or persons with equipment (for example, mobile phones, AR glasses, etc.) can upload relevant signals to the server autonomously, eliminating the need to use special facilities (For example, card swiping devices, fingerprint collection devices, etc.) to obtain signals related to equipment or personnel.

Fig. 4 shows a schematic image observed by the first user in Fig. 2 through a mobile phone screen or AR glasses according to an embodiment, which includes a second user and a third user. As mentioned above, the first user may be a bank branch staff, and the second and third users may be bank customers. As shown in Figure 4, the first user can not only observe the second user and the third user through the mobile phone screen or AR glasses, but also the virtual objects superimposed on each user's head (for example, "VIP", "normal", etc.) , These objects can be used to indicate the user’s customer level, for example, whether it is a VIP customer or an ordinary customer. The customer level of the user can be obtained, for example, by reading the user's identity information, bank card information, etc., or by receiving the user's phone number information, bank account information, etc. from the user's mobile phone.

In an embodiment of the present invention, the present invention can be implemented in the form of a computer program. The computer program can be stored in various storage media (for example, a hard disk, an optical disk, a flash memory, etc.), and when the computer program is executed by a processor, it can be used to implement the method of the present invention.

In another embodiment of the present invention, the present invention may be implemented in the form of an electronic device. The electronic device includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, it can be used to implement the method of the present invention.

References herein to "each embodiment", "some embodiments", "one embodiment", or "an embodiment", etc. refer to the specific features, structures, or properties described in connection with the embodiments that are included in In at least one embodiment. Therefore, the phrases "in various embodiments", "in some embodiments", "in one embodiment", or "in implementation The appearances of "in the example" in various places throughout this text do not necessarily refer to the same embodiment. In addition, specific features, structures, or properties can be combined in any suitable manner in one or more embodiments. Therefore, a combination of one implementation The specific features, structures, or properties shown or described in the examples can be combined in whole or in part with the features, structures, or properties of one or more other embodiments without limitation, as long as the combination is not illogical or impossible. Work. Expressions similar to "according to A", "based on A", "through A" or "using A" appearing in this article mean non-exclusive, that is, "according to A" can cover "only according to A", It can also cover "according to A and B", unless it is specifically stated that its meaning is "only according to A." In this application, for clarity, some illustrative operating steps are described in a certain order, but those skilled in the art can understand Each of these operating steps is not indispensable. Some of these steps can be omitted or replaced by other steps. These operating steps do not have to be performed sequentially in the manner shown. On the contrary, some of these operating steps It can be executed in a different order or in parallel according to actual needs, as long as the new execution method is not illogical or unable to work.

Thus, several aspects of at least one embodiment of the present invention have been described, and it can be understood that various changes, modifications and improvements can be easily made by those skilled in the art. Such changes, modifications and improvements are intended to be within the spirit and scope of the present invention. Although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described here, and also includes various changes and changes made without departing from the scope of the present invention.

Although the present invention is disclosed in the foregoing embodiments as above, it is not intended to limit the present invention. Anyone who is familiar with similar skills, without departing from the spirit and scope of the present invention, makes changes and modifications equivalent to replacements, still belongs to the present invention. Within the scope of patent protection.

S301~S304: Step flow

Claims (17)

A method for setting a presentable virtual object for a target in a real scene, wherein a camera and an optical communication device are installed in the real scene, and the camera and the optical communication device have a relative pose, the method includes : Use the camera to track a target in the real scene; Obtain a position signal of the target according to the tracking result of the camera; Setting a virtual object with a spatial position signal associated with the target, wherein a spatial position signal of the virtual object is determined based on the position signal of the target; and The signal related to the virtual object is sent to a first device, and the signal includes the spatial position signal of the virtual object, wherein the signal related to the virtual object can be used by the first device to pass through the optical communication device A determined position signal and a posture signal present the virtual object on its display medium. The method described in item 1 of the scope of patent application, wherein the obtaining the position signal of the target according to the tracking result of the camera includes: Obtain the position signal of the target relative to the camera according to the tracking result of the camera, and The position signal of the target relative to the optical communication device is determined according to the position signal of the target relative to the camera and the relative pose between the camera and the optical communication device. The method described in item 2 of the scope of patent application, wherein the spatial position signal of the virtual object is relative to the spatial position signal of the optical communication device. The method described in item 1 of the scope of patent application, wherein obtaining the position signal of the target according to the tracking result of the camera includes: Obtain the position signal of the target relative to the camera according to the tracking result of the camera, and The position signal of the target in the real scene is determined according to the position signal of the target relative to the camera and the pose signal of the camera in the real scene. The method according to any one of items 1 to 4 in the scope of the patent application, wherein the signal related to the virtual object is configured according to the signal related to the target. As the method described in item 5 of the scope of patent application, it also includes: according to the tracking result of the camera Obtain a signal related to the target. For example, the method described in item 5 of the scope of patent application, wherein the related signal of configuring the virtual object according to the signal related to the target includes: Receive signals related to the target from the facility; Establishing contact with the signal and the target tracked by the camera at a predetermined position relative to the facility; and According to the signal, the relevant signal of the virtual object of the target is configured. For example, the method described in item 5 of the scope of patent application, wherein the related signal of configuring the virtual object according to the signal related to the target includes: Receiving a signal from a second device, the signal including a location signal of the second device; Comparing the position signal of the second device with the position signal of one or more targets determined according to the tracking result of the camera to determine a target matching the second device; and According to the signal from the second device, the related signal of the virtual object of the target matched with the second device is configured. According to the method described in item 8 of the scope of patent application, the second device determines its position signal at least in part by collecting an image including the optical communication device and analyzing the image. The method described in any one of items 1-4 in the scope of the patent application also includes: Obtain the posture signal of the target according to the tracking result of the camera; And wherein, the setting of the virtual object with the spatial position signal associated with the target includes setting the posture signal of the virtual object according to the posture signal of the target. The method according to any one of items 1-4 in the scope of the patent application, wherein the first device at least partially collects an image including the optical communication device and analyzes the image to determine that it is relative to the optical communication device. The position signal and the attitude signal of the device. A system for setting presentable virtual objects for targets in real scenes, including: A camera installed in the real scene, which is used to track the target in the real scene; An optical communication device installed in the real scene, wherein the optical communication device and the camera have relative poses; and A server used to: Obtain the position signal of the target according to the tracking result of the camera; Set a virtual object associated with the target with a spatial position signal, where the virtual object The spatial position signal of the proposed object is determined based on the position signal of the target; The signal related to the virtual object is sent to the first device, and the signal includes a spatial position signal of the virtual object, wherein the signal related to the virtual object can be used by the first device to determine it based on the optical communication device The position signal and posture signal of the virtual object are displayed on the display medium. For example, the system described in item 12 of the scope of patent application, wherein the obtaining the position signal of the target according to the tracking result of the camera includes: Obtain the position signal of the target relative to the camera according to the tracking result of the camera, and The position signal of the target relative to the optical communication device is determined according to the position signal of the target relative to the camera and the relative pose between the camera and the optical communication device. For example, the system described in item 12 of the scope of patent application, wherein the obtaining the position signal of the target according to the tracking result of the camera includes: Obtain the position signal of the target relative to the camera according to the tracking result of the camera, and The position signal of the target in the real scene is determined according to the position signal of the target relative to the camera and the pose signal of the camera in the real scene. For the system described in any one of items 12-14 in the scope of the patent application, the first device is used for: Receiving the relevant signal of the virtual object from the server; Determining the position signal and the attitude signal of the first device at least in part through the optical communication device; and The virtual object is presented on the display medium of the first device based on the position signal, the posture signal and the related signal of the virtual object. A storage medium in which a computer program is stored. When the computer program is executed by the processor, it can be used to implement the method described in any one of items 1-11 in the scope of the patent application. An electronic device, comprising a processor and a memory, the memory stores a computer program, and when the computer program is executed by the processor, it can be used to implement any one of items 1-11 in the scope of the patent application的方法。 The method.

Images