WO2021227918A1

WO2021227918A1 - Interaction method and augmented reality device

Info

Publication number: WO2021227918A1
Application number: PCT/CN2021/091864
Authority: WO
Inventors: 张志灵; 刘梦婷
Original assignee: 维沃移动通信有限公司
Priority date: 2020-05-09
Filing date: 2021-05-06
Publication date: 2021-11-18
Also published as: CN111580661A

Abstract

An interaction method and an augmented reality device, which belong to the field of communication technology, and which solve the problem of relatively poor AR device interactivity. An interaction method for a first augmented reality device side comprises: detecting a line of sight direction of a first user and a first action of the first user (101); when the line of sight of the first user is projected onto a region in which a target symbol is located, and the first action of the first user satisfies a first preset condition, sending to a second augmented reality device target information corresponding to the first action (102); wherein the target symbol is a symbol within a projection region of the first augmented reality device, and the second augmented reality device is an electronic device associated with the target symbol.

Description

Interactive method and augmented reality device

Cross-references to related applications

This application claims the priority of Chinese Patent Application No. 202010389104.1 filed in China on May 9, 2020, the entire content of which is incorporated herein by reference.

Technical field

This application belongs to the field of communication technology, and specifically relates to an interaction method and an augmented reality device.

Background technique

With the development of Augmented Reality (AR) technology, remote video through AR equipment has gradually been applied in various fields. The AR device can capture the environment where the video participant is located, thereby enabling face-to-face communication between multiple users. For example, a user conducts a multi-person video conference through an AR device, and each participant can view the environment where other participants are located through the AR device.

However, in the process of implementing the above-mentioned technology, the inventor found that the prior art has at least the following problems: AR devices can only display the collected user's video images, but cannot interact with users, and the interactivity of AR devices is poor. .

Summary of the invention

The purpose of the embodiments of the present application is to provide an interaction method and an augmented reality device, which can solve the problem of poor interactivity of the AR device.

In order to solve the above technical problems, this application is implemented as follows:

In the first aspect, an embodiment of the present application provides an interaction method applied to a first augmented reality device, and the method includes:

Detecting the direction of the line of sight of the first user and the first action of the first user;

In the case that the line of sight of the first user is projected on the area where the target identifier is located, and the first action of the first user meets the first preset condition, the second augmented reality device is sent to the second augmented reality device corresponding to the first action Target information

Wherein, the target identifier is an identifier in the projection area of the first augmented reality device, and the second augmented reality device is an electronic device associated with the target identifier.

In the second aspect, an embodiment of the present application provides an interaction method applied to a second augmented reality device, and the method includes:

Receiving target information sent by a first augmented reality device, where the target information corresponds to a first action detected by the first augmented reality device;

If it is detected within the third preset time that the second action of the second user meets the second preset condition, output third prompt information, and send a response corresponding to the target information to the first augmented reality device information;

Or, in the case that the second action of the second user that satisfies the second preset condition is not detected within the third preset time, output fourth prompt information.

In a third aspect, an embodiment of the present application provides an interaction device, which is applied to a first augmented reality device, and the interaction device includes:

The detection module is used to detect the line of sight direction of the first user and the first action of the first user;

The first sending module is configured to send and to the second augmented reality device when the line of sight of the first user is projected on the area where the target identifier is located, and the first action of the first user meets the first preset condition Target information corresponding to the first action;

In a fourth aspect, an embodiment of the present application provides an interaction device, which is applied to a second augmented reality device, and the interaction device includes:

A first receiving module, configured to receive target information sent by a first augmented reality device, where the target information corresponds to a first action detected by the first augmented reality device;

The third output module is configured to output third prompt information when it is detected that the second user's second action satisfies the second preset condition within the third preset time, and send the information to the first augmented reality device Response information corresponding to the target information;

Or, the fourth output module is configured to output fourth prompt information when the second action of the second user that satisfies the second preset condition is not detected within the third preset time.

In the fifth aspect, an embodiment of the present application provides an augmented reality device. The electronic device includes a processor, a memory, and a program or instruction that is stored on the memory and can run on the processor. The program or instruction When executed by the processor, the steps in the method according to the first aspect or the second aspect are realized.

In a sixth aspect, an embodiment of the present application provides a readable storage medium that stores a program or instruction on the readable storage medium, and when the program or instruction is executed by a processor, the implementation is as described in the first or second aspect. Steps in the method.

In a seventh aspect, an embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is used to run a program or an instruction to implement the chip as in the first aspect Or the method described in the second aspect.

In an eighth aspect, an embodiment of the present application provides an interaction device, wherein the interaction device is configured to execute the method according to the first aspect or the second aspect.

In the embodiment of the present application, the direction of the line of sight of the first user and the first action of the first user are detected; the line of sight of the first user is projected on the area where the target identifier is located, and the first action of the first user When the first preset condition is met, the target information corresponding to the first action is sent to the second augmented reality device; wherein the target identifier is the identifier in the projection area of the first augmented reality device, and The second augmented reality device is an electronic device associated with the target identifier. The first augmented reality device can detect the user's actions, so as to realize the interaction between the augmented reality device and the user, and can participate in the interaction between the user and the user, and can improve the interaction effect.

Description of the drawings

FIG. 1 is one of the flowcharts of the interaction method provided by the embodiment of the present invention;

Figures 2 to 6 are images displayed by an augmented reality device provided by an embodiment of the present invention;

FIG. 7 is the second flowchart of the interaction method provided by the embodiment of the present invention;

FIG. 8 is a structural diagram of a first augmented reality device provided by an embodiment of the present invention;

FIG. 9 is a structural diagram of a second augmented reality device provided by an embodiment of the present invention;

FIG. 10 is a structural diagram of a first augmented reality device or a second augmented reality device provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms "first" and "second" in the specification and claims of this application are used to distinguish similar objects, but not to describe a specific sequence or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in a sequence other than those illustrated or described herein. In addition, "and/or" in the specification and claims means at least one of the connected objects, and the character "/" generally means that the associated objects before and after are in an "or" relationship.

The interaction method provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios thereof.

The interaction method of the embodiment of the present invention may be applied to augmented reality (Augmented Reality, AR) devices, which may specifically be electronic devices with augmented reality functions or head-mounted augmented reality devices, etc., which are not limited here.

In scenarios such as multi-person video calls, such as multi-person meetings, multi-person video, etc., each user of the video call can view the scene of other users through the augmented reality device, and each user can also use the augmented reality device Interact with other users.

In the embodiment of the present invention, when the first user makes a first action when using the first augmented reality device, such as shaking hands or waving a hand, the first augmented reality device may collect the first action and determine whether the first preset condition is satisfied , Such as whether it matches with the preset action or whether the intensity of the first action is greater than the preset value. When the first action satisfies the first preset condition, the first augmented reality device can obtain the action parameters of the first action, including the direction of the action, the force generated, the speed of the action, etc., and send it to the second augmented reality device Generate target information including action parameters.

The second augmented reality device may prompt the second user according to the received target information, so that the second user can make an interactive action, thereby realizing the interaction between the first user and the second user. For example, when the second augmented reality device receives the prompt information including the action parameters sent from the first augmented reality device, it can output the prompt sound according to the prompt information to simulate a real stereo effect, the volume of the prompt sound and the size of the action parameters Related, the type of prompt sound corresponds to the type of action. If the second augmented reality device detects that the second user makes an action response within the preset time after receiving the prompt, such as projecting the line of sight on the account identification corresponding to the first augmented reality device, waving a hand, etc., the accounts on the AR devices of both parties The logo can display a green halo, which means that the second user is aware of this interactive action and the interaction is successful. If the response is not received within the preset time, the response is deemed to be missed/not heard, and the account logos on the AR devices of both parties can display a red halo or further output prompts to prompt the second user to respond. If the second user still does not respond, a red halo may be displayed, indicating that the user status of the recipient is busy.

In this way, the first user and the second user can interact through the augmented reality device, and the augmented reality device can prompt the interaction between users, can participate in the interaction between users, and can improve the interaction effect.

When the line of sight of the first user is projected on the file identifier, the first augmented reality device may also send the file to other augmented reality devices according to the user's first action, thereby simulating the actual file sending effect.

In order to facilitate the understanding of the specific implementation manners of the embodiments of the present invention, the following descriptions are made from the perspective of the first augmented reality device side and the second augmented reality device side.

As shown in FIG. 1, the interaction method on the side of the first augmented reality device may include the following steps:

Step 101: Detect the line of sight direction of the first user and the first action of the first user.

In the embodiment of the present invention, when the first user uses the first augmented reality device, the first augmented reality device can recognize the user’s line of sight direction by means of an eye locator or camera. In addition, it can also use a depth camera, infrared Sensors such as cameras and RGB (red, green and blue) cameras recognize the first action of the first user.

The above-mentioned first motion may specifically include gesture motion, head motion, or body motion, etc., such as shaking hands, shaking head, and hugging.

Step 102: In a case where the line of sight of the first user is projected on the area where the target identifier is located, and the first action of the first user meets a first preset condition, send a connection with the first augmented reality device to the second augmented reality device. The target information corresponding to the action.

The first augmented reality device can project the content that needs to be displayed to a specific area, which is convenient for the user to watch. The above-mentioned projection area may be the content displayed by the first augmented reality device. By projecting the content, the displayed content can be presented to the user, and the user using the first augmented reality device can view the content of the projection area. The projection area of the first augmented reality device may include a target identifier. The target identifier may be an account identifier associated with other electronic devices, and may also be a file identifier, an application program identifier, or other identifiers used to implement specific functions.

The above-mentioned first action meeting the first preset condition can be understood as the first action matches the preset action, or the moving speed of the first action is within the preset speed range, or the force generated by the first action is within the preset force range, and many more.

When the line of sight of the first user is projected on the area where the target identifier is located, and the first action satisfies the preset conditions, the target information corresponding to the first action can be sent according to the action type and action parameters of the first action, and the target The information can be used to indicate the type of information output by the second augmented reality device, including tactile information (such as vibration), display information (such as yellow halo), sound information (such as prompt sound), and so on. For example, when the first action is clapping, the strength of the clapping action can be acquired, thereby sending target information including the strength information; when the first action is a handshake, the target information includes prompt information for indicating a handshake. When the first action is a retrieval operation on the file identifier, the target information includes the file corresponding to the file identifier.

In this way, the augmented reality device can participate in the interaction between users and send the interaction information of the first user to the second user, and the first user and the second user can interact through the augmented reality device, which can improve the interaction effect. The present invention acquires the user's line of sight direction and behavior and transmits them to the augmented reality device of the other party, which can bring an interactive communication mode close to the real scene.

Optionally, the target identifier is a second account identifier, and the method further includes:

In the case where the line of sight of the first user is projected on the area where the second account identifier is located, a prompt identifier is displayed in the area where the second account identifier is located.

In this embodiment, the projection area of the first augmented reality device may include one or more account identifiers, and each account identifier may correspond to an electronic device. Wherein, the second account identifier may be an account identifier corresponding to the second augmented reality device.

When the line of sight of the first user is projected on the second account identifier, a prompt identifier may be displayed in the area where the second account identifier is located, so that the user confirms that the user needs to interact with the user corresponding to the account identifier. The prompt indicator can be a flashing display, a color-enhanced display, or a focus indicator as shown in the right figure in FIG. 2.

When the first augmented reality device detects that the line of sight of the first user is projected on the area where the second account identifier is located, and the duration is longer than the preset duration, the first augmented reality device can display the prompt identifier in the area where the second account identifier is located, thereby reducing account identification. The error.

Optionally, after the sending target information corresponding to the first action to the second augmented reality device, the method further includes:

In the case of receiving the response information sent by the second augmented reality device within the first preset time, output the first prompt information;

Or, in the case that the response information sent by the second augmented reality device is not received within the first preset time, output second prompt information.

In this embodiment, after the first augmented reality device sends the target information to the second augmented reality device, if the response information to the target information sent by the second augmented reality device is received within the first preset time, the first augmented reality device The reality device outputs the first prompt message to indicate the success of the interaction, such as displaying a green halo on the target logo; if the response information sent by the second augmented reality device is not received within the first preset time, then the second prompt message is output , Such as showing a red halo on the target logo to indicate that the interaction has failed.

For example, the first user taps the target identifier through the first AR device, and the first AR device sends prompt information corresponding to the tap action to the second AR device. The second AR device outputs prompt information according to the received prompt. If within 5 seconds after outputting the prompt information, the eye locator recognizes that the second user is looking at the account identification of the first user, and the account identification of the first user and the account identification of the second user on both devices display a green halo respectively, It means that the second user is aware of this simulated tapping action and the interaction is successful. If the second user does not look at the account ID of the first user within 5 seconds, it is deemed to have missed the prompt, and the account ID of the first user and the account ID of the second user on the devices of both parties respectively show red halos, indicating the user of the recipient The state is busy.

In this way, the first augmented reality device outputs different prompt messages to prompt the user of the success or failure of the interaction, which is convenient for the user to perform other processing or interaction according to the result of the interaction.

Optionally, in the case that the response information sent by the second augmented reality device is not received within the first preset time, before outputting the second prompt information, the method further includes:

In the case that the response information sent by the second augmented reality device is not received within the second preset time, and the second augmented reality device is connected to the second wearable glove, the second augmented reality device is sent to the second augmented reality device. One information;

Wherein, the first information is used to control the second wearing glove to output tactile sensation.

In this embodiment, before the first augmented reality device outputs the second prompt information without receiving the response information, the first augmented reality device may further prompt the second augmented reality device information, that is, send the first information. After receiving the first information, the second augmented reality device may control the second wearing glove to output tactile sensations according to the first information, so as to prompt the second user to make a response action.

Similarly, the second augmented reality device can also prompt the first augmented reality device in the above-mentioned manner, thereby controlling the first augmented reality device to output a tactile sensation to the first wearable glove connected to the first augmented reality device.

In this way, by sending information to the second augmented device to make the second augmented reality device control the wearing glove to output tactile prompts, the prompts can be made more intuitively and effectively, and the effect of interaction can be improved.

Optionally, the sending target information corresponding to the first action to the second augmented reality device includes:

Acquiring the action parameter of the first action;

When the action parameter satisfies the first parameter condition, send first target information to the second augmented reality device, where the first target information is used to instruct the second augmented reality device to output a prompt in the first manner information;

Or, when the action parameter satisfies the second parameter condition, send second target information to the second augmented reality device, where the second target information is used to instruct the second augmented reality device to use the second method Output prompt information.

In this embodiment, the first action may include actions such as clapping, shaking hands, waving, and hugging, and the action parameters may include at least one of the following: the movement direction of the movement, the movement speed, the strength of the movement, and the like. The first augmented reality device can obtain the movement direction and speed of the action through the depth camera, RGB camera, etc. The strength of the action can be obtained through smart gloves or other sensing devices connected to the first augmented reality device, and can also be calculated by using the movement parameters get.

The first augmented reality device may send different information to the second augmented reality device according to the type and size of the action parameter, so as to instruct the second augmented reality device to output the prompt information in a different manner.

For example, when the action parameter of the first action satisfies the first parameter condition, if the intensity is small, the second augmented reality device can be instructed to prompt with a smaller prompt sound, and the type and size of the prompt sound can simulate the first action The sound produced; when the action parameters of the first action meet the second parameter conditions, such as greater intensity, the second augmented reality device can be instructed to prompt with a larger prompt sound, and the type and size of the prompt sound can simulate the first The sound produced by the action.

For another example, when the first action is an eye action, the second augmented reality device can be instructed to display an image of the first action; when the first action is a gesture action, the second augmented reality device can be instructed to control wearing gloves to output tactile sensations.

The prompts are output in different ways, so that the second user can quickly obtain the interactive actions made by the first user, so as to respond quickly and improve the efficiency and effect of the interaction. In addition, the first augmented reality device also sends the above-mentioned action parameters to the second augmented reality device, so that the second augmented reality device outputs corresponding prompts, for example, the prompt sound is loud when the photo is retaken, and the prompt sound is small when the photo is tapped. , It is convenient for the second user to understand the type of action or other information conveyed by the action according to the prompt.

Optionally, after detecting the first action of the first user, the method further includes:

Displaying a first image used to represent the first action in the projection area;

After the sending the target information corresponding to the first action to the second augmented reality device, the method further includes:

In a case where response information to the target information sent by the second augmented reality device is received, and the response information is used to indicate a second action, display a third image in the projection area according to the response information , The action represented by the third image includes the first action and the second action;

Or, in the case that the response information to the target information sent by the second augmented reality device is not received, the first image in the projection area is switched to a fourth image, and the fourth image The indicated action is the opposite of the first action.

In this embodiment, the first action may include interactive actions such as hugging, shaking hands, blinking, blowing kisses, and taking files. The first augmented reality device may display a first image representing the first action in the projection area, and the image may be a picture or a video image, so that the first user can obtain the interactive action he is making.

After receiving the target information, the second augmented reality device may send response information to the first augmented reality device, for example, the second user has made a second action in response. The response information may include an image used to indicate the second action, or it may be an instruction message, which is used to instruct the first device to display its stored interactive image with the first action in the projection area.

In the case that the first augmented reality device receives the response information within the preset time, the third image may be displayed according to the response information, and the third image may include images of the first action and the second action that match each other. For example, after detecting that the first user makes the handshake action as shown in FIG. 3, the first augmented reality device sends target information to the second augmented reality device. The second augmented reality device displays the image as shown in FIG. 4 according to the target information. After the second augmented reality device detects the second action responded by the second user, the first augmented reality device and the second augmented reality device may simultaneously display the handshake image as shown in FIG. 5, indicating that the interaction is successful.

For another example, the first user sends target information including the file receiving request to the second augmented reality device through the first augmented reality device, and both the first augmented reality device and the second augmented reality device display the image of the file sending request. If the second augmented reality device receives the nodding action of the second user (that is, the nodding action is the response information to the target information), the first augmented reality device and the second augmented reality device can display the image received by the file at the same time, indicating that they agree to receive document.

In the case that the first augmented reality device does not receive the response information sent by the second augmented reality device within the preset time, or the first user withdraws the first action, the action represented by the first image can be withdrawn, that is, switch to The fourth image representing the withdrawal action. For example, when the action represented by the first image is a handshake, the action represented by the fourth image is an action of retracting the hand.

In this way, the first augmented reality device can display the interactive actions of the first user and the second user, and can make a corresponding response according to whether the second user responds, and can further restore the interactive effect in the real scene.

Optionally, the first action is a gesture action, and the first augmented reality device is connected to a first wearing glove worn by the first user, and the method further includes:

In a case where the response information to the target information sent by the second augmented reality device includes tactile information, controlling the first wearing glove to output the tactile feeling corresponding to the tactile information;

Or, when the first action is an action of taking a target item, detect the distance between the first finger cuff and the second finger cuff of the first wearing glove, and if the distance is less than a preset distance Next, the first finger cuff is controlled to output a first contact force, and the second finger cuff is controlled to output a second contact force, wherein the preset distance is determined according to the size of the target item.

In this embodiment, the first augmented reality device is connected with the first wearable glove, so that information interaction can be realized. When the response information sent by the second augmented reality device includes tactile information, such as touch strength, touch position, etc., the first augmented reality device can control the first wearing glove to output the tactile sensation, and the strength and position of the tactile sensation can be compared with the second augmented reality device. It corresponds to the information carried in the tactile information sent by the real device. For example, the response information includes handshake action information and the strength of the handshake, and the first wearable device outputs a tactile sensation to the first user according to the strength. In this way, the real scene where the first user and the second user make gesture actions can be restored, and the interaction effect can be improved. In addition, the first augmented reality device can determine whether to control the first wearing glove to output tactile sensations according to the response information, which improves the flexibility of interactive reminders.

The first augmented reality device may also detect the user's first action and the user's line of sight. For example, when the line of sight is projected on the file identifier and the first action meets a preset condition, it is determined to be an action of taking the target item. It is also possible to determine whether to pick up the target item according to parameters such as the action mode and posture. For example, as shown in FIG. 6, the eye locator 1 can identify the direction of the user's eyeballs, and the wearable sensor and the image recognition system 2 can identify the user's gestures. Combining the recognition of the direction of the line of sight and the gesture action can improve the accuracy of the recognition of the user's taking action.

When the distance between the first finger cuff and the second finger cuff is less than the preset distance, it can be considered that the first finger cuff and the second finger cuff have contacted the target item, and the first finger cuff and the second finger cuff can be controlled to output touch Power to simulate the user’s true feelings when picking up items.

For example, when taking a target item, wearing gloves can detect the pressure of the taking action on each finger cuff, that is, the pressure generated by the taking action. At this time, the target item is located between the first finger and the second finger. . When the distance between the first finger cuff and the second finger cuff is less than the preset distance, the first augmented reality device can output a force that is opposite to the direction of the pressure and equal in strength to the finger according to the pressure of each finger cuff. In order to simulate the reaction force received by the user's finger when the target item is picked up. The aforementioned target item may be a virtual item displayed in the projection area.

In the embodiment of the present invention, the user’s action of taking a virtual object is identified by using the parameters of the line of sight projection combined with the gesture action, which can improve the identification accuracy, and the tactile sensation can be output by wearing gloves, which can display the online interactive actions more intuitively and achieve more realism. Effect.

It should be noted that, for the interaction method provided in the embodiments of the present application, the execution subject may be an interaction device corresponding to the first augmented reality device, or a control module in the interaction device for executing the method of loading interaction. In the embodiment of the present application, the method of loading interaction performed by the interactive device is taken as an example to illustrate the method of interaction provided in the embodiment of the present application.

As shown in FIG. 7, the interaction method on the second augmented reality device side may include the following steps:

Step 701: Receive target information sent by a first augmented reality device, where the target information corresponds to a first action detected by the first augmented reality device.

For the specific implementation process of this step, refer to the description on the side of the first augmented reality device.

Step 702: When it is detected that the second action of the second user satisfies the second preset condition within the third preset time, output third prompt information, and send the target information to the first augmented reality device Corresponding response information.

The second action meeting the second preset condition can be understood as a match between the second action and the first action, or that the parameters of the second action meet a specific condition, and so on. The second augmented reality device can be recognized by devices such as an eyeball locator, a depth camera, an infrared camera, and an RGB camera.

When the target information is received, the second augmented reality device may output third prompt information to prompt the second user that the interaction is successful. For example, the second augmented reality device receives prompt information for prompting a handshake request, and when the second user makes a handshake action within a predetermined time, the second augmented reality device outputs a prompt message indicating that the handshake is successful. In addition, the second augmented reality device may also send response information of the target information to the first augmented reality device, so that the first augmented reality device obtains the interaction result. The response information may include image information, or only prompt information used to indicate the second action, and may also include other information.

Preferably, the first prompt information on the first augmented reality device is the same as the third prompt information on the second augmented reality device, and the second prompt information is the same as the fourth prompt information.

The above step 702 can also be replaced with step 703.

Step 703: If the second user's second action meeting the second preset condition is not detected within the third preset time, output fourth prompt information.

When the second user does not respond as a second action that satisfies the second preset condition, the second augmented reality device may output fourth prompt information to prompt the interaction failure. In addition, the second augmented reality device may also send response information of the target information to the first augmented reality device, so that the first augmented reality device obtains the interaction result.

In this way, the second augmented reality device can obtain the action of the second user to determine whether the interaction is successful, and can send the interaction result to the first augmented reality device, which can improve the interaction effect.

Optionally, the second augmented reality device is connected to a second wearable glove worn by the second user, and before the sending response information corresponding to the target information to the first augmented reality device, the Methods also include:

Receiving the first information sent by the first augmented reality device;

According to the first information, the second wearing glove is controlled to output a tactile sensation.

In this embodiment, after receiving the target information, the third prompt information can be output according to the gesture parameters, and the third prompt information can specifically output a sound that simulates the first action according to the type of the first action, so that the second user can obtain more information. The real effect. For example, if the first action is clapping hands, the second augmented reality device can output the clapping stereo sound, and the color of the account logo can be changed according to the sound; the first action is a handshake, then the second augmented reality device can use the smart glove to move towards the second The user outputs the corresponding grip strength.

In this way, the second augmented reality device can restore the effect of the first action more realistically, thereby improving the interaction effect between the first user and the second user.

Optionally, in a case where the target information includes the first target information, output the third prompt information in a first manner;

Or, in a case where the target information includes the second target information, output the third prompt information in a second manner.

In this embodiment, the second augmented reality device may output the prompt information in different ways according to the type of information included in the target information. The first target information and the second target information may be information for indicating the output mode of the prompt information, and the first mode and the second mode may be voice output, display screen output, or vibration output, respectively. This can be understood in conjunction with the description on the side of the first augmented reality device, and will not be repeated here.

The prompt information is output in different ways, which is convenient for the user to obtain the information of the first action according to the prompt information.

Optionally, after the receiving the target information sent by the first augmented reality device, the method further includes:

Displaying a first target image used to represent the first action in the projection area of the second augmented reality device;

After detecting that the second action of the second user meets the second preset condition within the third preset time, the method further includes:

A third target image is displayed in the projection area of the second augmented reality device, and the action represented by the third target image includes the first action and the second action.

In this embodiment, the first target image may be the same or different from the first image in the foregoing embodiment, and the third target image may be the same or different from the third image in the foregoing embodiment. For example, the first image is a gesture of reaching toward the other party (Figure 3), and the first target image is a gesture of reaching toward oneself (Figure 4).

After the second user makes a response action to the first action, that is, the second action, the second augmented reality device may display the third target image in the projection area to indicate that the interaction is successful.

In this way, participating in user interaction through image presentation can improve the interaction effect, and can prompt the user to improve the interaction efficiency.

In the embodiment of the present invention, the second enhanced display device can make corresponding prompt content according to the response action made by the second user to prompt the user that the interaction is successful. It can simulate real interactive scenes more intuitively and improve the interactive effect.

It should be noted that the execution subject of the interaction method provided by the embodiment of the present application may be the interaction device corresponding to the second augmented reality device, or the control module in the interaction device for executing the method of loading interaction. In the embodiment of the present application, the method of loading interaction performed by the interactive device is taken as an example to illustrate the method of interaction provided in the embodiment of the present application.

As shown in FIG. 8, FIG. 8 is a structural diagram of an interaction device provided by an embodiment of the present invention, which can be applied to a first augmented reality device. As shown in FIG. 8, the interaction device 800 includes:

The detection module 801 is configured to detect the line of sight direction of the first user and the first action of the first user;

The first sending module 802 is configured to send to the second augmented reality device when the line of sight of the first user is projected on the area where the target identifier is located, and the first action of the first user meets the first preset condition Target information corresponding to the first action;

Optionally, the interaction device further includes:

The first output module is configured to output first prompt information when the response information sent by the second augmented reality device is received within the first preset time;

Or, the second output module is configured to output second prompt information when the response information sent by the second augmented reality device is not received within the first preset time.

Optionally, the interaction device further includes:

The second sending module is configured to send the response information sent by the second augmented reality device to the The second augmented reality device sends the first information;

Optionally, the first sending module includes:

An acquiring sub-module for acquiring the action parameter of the first action;

A sending submodule, configured to send first target information to the second augmented reality device when the action parameter satisfies the first parameter condition, where the first target information is used to indicate the second augmented reality device Output prompt information in the first way;

Or, when the action parameter satisfies the second parameter condition, send second target information to the second augmented reality device, where the second target information is used to instruct the second augmented reality device to Two ways to output prompt information.

Optionally, the interaction device further includes:

A first display module, configured to display a first image for representing the first action in the projection area;

The second display module is configured to receive response information to the target information sent by the second augmented reality device, and the response information is used to indicate a second action, according to the response information in the The projection area displays a third image, and the action represented by the third image includes the first action and the second action.

Optionally, the first action is a gesture action, the first augmented reality device is connected to a first wearing glove worn by the first user, and the interaction device further includes:

The first control module is configured to control the first wearing glove to output the tactile sensation corresponding to the tactile sensation information in the case that the response information to the target information sent by the second augmented reality device includes tactile sensation information;

Or, the second control module is configured to detect the distance between the first finger cuff and the second finger cuff of the first wearing glove when the first action is an action of picking up a target item, and When the distance is less than a preset distance, the first finger cuff is controlled to output a first contact force, and the second finger cuff is controlled to output a second contact force, wherein the preset distance is based on the size of the target item The size is ok.

The interaction apparatus 800 can implement the interaction method on the first augmented reality device side in the foregoing method embodiment and achieve the same beneficial effects. To avoid repetition, details are not described herein again.

As shown in FIG. 9, FIG. 9 is a structural diagram of an interaction device provided by an embodiment of the present invention. The interaction device 900 can be applied to a second augmented reality device. As shown in FIG. 9, the interaction device 900 includes:

The first receiving module 901 is configured to receive target information sent by a first augmented reality device, where the target information corresponds to a first action detected by the first augmented reality device;

The third output module 902 is configured to output third prompt information to the first augmented reality device when it is detected that the second action of the second user meets the second preset condition within the third preset time Response information corresponding to the target information;

Or, the fourth output module 903 is configured to output fourth prompt information when the second user's second action that satisfies the second preset condition is not detected within the third preset time.

Optionally, the second augmented reality device is connected to a second wearable glove worn by the second user, and the interaction device further includes:

The second receiving module is configured to receive the first information sent by the first augmented reality device;

The third control module is configured to control the second wearing glove to output tactile sensations according to the first information.

Optionally, the interaction device further includes:

A fifth output module, configured to output the third prompt information in a first manner when the target information includes the first target information;

Or, the sixth output module is configured to output the third prompt information in a second manner when the target information includes the second target information.

Optionally, the interaction device further includes:

A third display module, configured to display a first target image used to represent the first action in the projection area of the second augmented reality device;

The interaction device further includes:

The fourth display module is configured to display a third target image in the projection area of the second augmented reality device, and the action represented by the third target image includes the first action and the second action.

The interaction device 900 can implement the interaction method on the second augmented reality device side in the foregoing method embodiment and achieve the same beneficial effects. To avoid repetition, details are not described herein again.

The interaction device in the embodiments of the present application may be a device, or a component, integrated circuit, or chip in a terminal. The device may be an electronic device. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (personal digital assistant). assistant, PDA), etc., and the embodiments of this application are not specifically limited.

The interaction device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

The interaction device provided by the embodiment of the present application can implement each process implemented by the interaction device in the method embodiments of FIG. 1 to FIG.

The first augmented reality device can detect the user's actions, so as to enhance the interaction between the augmented reality device and the user, and can participate in the interaction between the user and the user, and can improve the interaction effect.

Optionally, an embodiment of the present application further provides an augmented reality device, including a processor, a memory, and a program or instruction that is stored in the memory and can run on the processor, and the program or instruction is executed when the processor is executed. Each process in the foregoing embodiment of the interaction method on the first augmented reality device side and the second augmented reality device side can achieve the same technical effect, and in order to avoid repetition, it will not be repeated here.

FIG. 10 is a schematic diagram of the hardware structure of an augmented reality device implementing an embodiment of the present application. The augmented reality device 1000 includes, but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 And other parts.

Those skilled in the art can understand that the augmented reality device 1000 may also include a power source (such as a battery) for supplying power to various components. The power source may be logically connected to the processor 1010 through a power management system, so that the power management system can manage charging, discharging, and Functions such as power management. The structure of the augmented reality device shown in FIG. 10 does not constitute a limitation on the augmented reality device. The augmented reality device may include more or fewer components than shown in the figure, or some components may be combined, or different component arrangements, here No longer.

When the augmented reality device is the first augmented reality device, the processor 1010 is configured to detect the direction of the first user's line of sight and the first action of the first user; and the first user's line of sight is projected on the target identifier In the area where the first user’s first action satisfies a first preset condition, sending target information corresponding to the first action to the second augmented reality device;

In this way, the first augmented reality device can detect the user's actions, thereby enhancing the interaction between the augmented reality device and the user, and can participate in the interaction between the user and the user, and can improve the interaction effect.

Optionally, the target identifier is a second account identifier, and the processor 1010 is further configured to:

Optionally, the first action is a first gesture action, and the target identifier is a second account identifier; the processor 1010 is further configured to:

Optionally, that the processor 1010 executes the sending of target information corresponding to the first action to the second augmented reality device includes:

Acquiring the action parameter of the first action;

When the action parameter satisfies the second parameter condition, send second target information to the second augmented reality device, where the second target information is used to instruct the second augmented reality device to output a prompt in a second manner information.

Optionally, the processor 1010 is further configured to:

The control display unit 1006 displays the first image representing the first action in the projection area; after receiving the response information to the target information sent by the second augmented reality device, and the response information is used In the case of indicating a second action, a third image is displayed in the projection area according to the response information, and the action represented by the third image includes the first action and the second action.

Optionally, the first action is a gesture action, the first augmented reality device is connected to a first wearing glove worn by the first user, and the processor 1010 is further configured to:

Or, in the case that the first action is an action of taking the target item, the distance between the first finger cuff and the second finger cuff of the first wearing glove is detected, and when the distance is less than a preset distance Next, the first finger cuff is controlled to output a first contact force, and the second finger cuff is controlled to output a second contact force, wherein the preset distance is determined according to the size of the target item.

When the augmented reality device is a second augmented reality device, the processor 1010 is configured to receive target information sent by the first augmented reality device, where the target information corresponds to the first action detected by the first augmented reality device;

Optionally, the second augmented reality device is connected to a second wearable glove worn by the second user, and the processor 1010 is further configured to:

Receiving the first information sent by the first augmented reality device;

Optionally, the processor 1010 is further configured to:

In a case where the target information includes the first target information, output the third prompt information in a first manner;

Optionally, the processor 1010 is further configured to:

In this way, the second user can interact with the first user through the second augmented reality device, which can improve the interaction effect. In addition, the second augmented reality device can make a corresponding prompt according to the received target information, which is convenient for the user to quickly obtain the content of the first action.

The embodiment of the present application also provides a readable storage medium, the readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the first augmented reality device side and the second augmented reality device side are realized. Each process of the embodiment of the interactive method can achieve the same technical effect. In order to avoid repetition, it will not be repeated here.

Wherein, the processor is the processor in the augmented reality device described in the foregoing embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk, or optical disk.

An embodiment of the present application further provides a chip. The chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the first augmented reality device side. Each process of the embodiment of the interaction method with the second augmented reality device side can achieve the same technical effect. To avoid repetition, details are not repeated here.

An embodiment of the present application further provides an interaction device configured to execute the processes of the foregoing interaction method embodiments on the first augmented reality device side and the second augmented reality device side, and can achieve the same technology The effect, in order to avoid repetition, will not be repeated here.

It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as system-level chips, system-on-chips, system-on-chips, or system-on-chips.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, It also includes other elements that are not explicitly listed, or elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or device that includes the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. The functions are performed, for example, the described method may be performed in a different order from the described order, and various steps may also be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to make a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.

The embodiments of the application are described above with reference to the accompanying drawings, but the application is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are only illustrative and not restrictive. Those of ordinary skill in the art are Under the enlightenment of this application, many forms can be made without departing from the purpose of this application and the scope of protection of the claims, all of which fall within the protection of this application.

Claims

An interaction method applied to a first augmented reality device, including:

Detecting the direction of the line of sight of the first user and the first action of the first user;

In the case that the line of sight of the first user is projected on the area where the target identifier is located, and the first action of the first user meets the first preset condition, the second augmented reality device is sent to the second augmented reality device corresponding to the first action Target information

Wherein, the target identifier is an identifier in the projection area of the first augmented reality device, and the second augmented reality device is an electronic device associated with the target identifier.
The method according to claim 1, wherein after the sending the target information corresponding to the first action to the second augmented reality device, the method further comprises:

In the case of receiving the response information sent by the second augmented reality device within the first preset time, output first prompt information, where the first prompt information is used to prompt the success of the interaction;

Or, in the case that the response information sent by the second augmented reality device is not received within the first preset time, output second prompt information, where the second prompt information is used to prompt an interaction failure.
The method according to claim 2, wherein, in the case that the response information sent by the second augmented reality device is not received within the first preset time, before outputting the second prompt information, the method Also includes:

In the case that the response information sent by the second augmented reality device is not received within the second preset time, and the second augmented reality device is connected to the second wearable glove, the second augmented reality device is sent to the second augmented reality device. One information;

Wherein, the first information is used to control the second wearing glove to output tactile sensation.
The method according to claim 1, wherein the sending target information corresponding to the first action to the second augmented reality device comprises:

Acquiring the action parameter of the first action;

When the action parameter satisfies the first parameter condition, send first target information to the second augmented reality device, where the first target information is used to instruct the second augmented reality device to output a prompt in the first manner Information, wherein the first manner is associated with the action parameter that satisfies the first parameter condition;

Or, when the action parameter satisfies the second parameter condition, send second target information to the second augmented reality device, where the second target information is used to instruct the second augmented reality device to use the second method Output prompt information, wherein the second mode is associated with the action parameter that satisfies the second parameter condition.
The method according to claim 1, wherein, after detecting the first action of the first user, the method further comprises:

Displaying a first image used to represent the first action in the projection area;

After the sending the target information corresponding to the first action to the second augmented reality device, the method further includes:

In a case where response information to the target information sent by the second augmented reality device is received, and the response information is used to indicate a second action, display a third image in the projection area according to the response information , The action represented by the third image includes the first action and the second action;

Or, in a case where the response information to the target information sent by the second augmented reality device is not received, the first image in the projection area is switched to a fourth image, and the fourth image The indicated action is the opposite of the first action.
The method according to claim 5, wherein the first action is a gesture action, and the first augmented reality device is connected to a first wearing glove worn by the first user, and the method further comprises:

In a case where the response information to the target information sent by the second augmented reality device includes tactile information, controlling the first wearing glove to output the tactile feeling corresponding to the tactile information;

Or, when the first action is an action of taking a target item, detect the distance between the first finger cuff and the second finger cuff of the first wearing glove, and if the distance is less than a preset distance Next, the first finger cuff is controlled to output a first contact force, and the second finger cuff is controlled to output a second contact force, wherein the preset distance is determined according to the size of the target item.
An interaction method applied to a second augmented reality device, including:

Receiving target information sent by a first augmented reality device, where the target information corresponds to a first action detected by the first augmented reality device;

If it is detected within the third preset time that the second action of the second user meets the second preset condition, output third prompt information, and send a response corresponding to the target information to the first augmented reality device information;

Or, in the case that the second action of the second user that satisfies the second preset condition is not detected within the third preset time, output fourth prompt information.
The method according to claim 7, wherein the second augmented reality device is connected to a second wearable glove worn by the second user, and the sending to the first augmented reality device corresponds to the target information Before the response information, the method further includes:

Receiving the first information sent by the first augmented reality device;

According to the first information, the second wearing glove is controlled to output a tactile sensation.
The method according to claim 7, wherein:

In a case where the target information includes the first target information, output the third prompt information in a first manner;

Or, in a case where the target information includes the second target information, output the third prompt information in a second manner.
The method according to claim 7, wherein after the receiving the target information sent by the first augmented reality device, the method further comprises:

Displaying a first target image used to represent the first action in the projection area of the second augmented reality device;

After detecting that the second action of the second user meets the second preset condition within the third preset time, the method further includes:

A third target image is displayed in the projection area of the second augmented reality device, and the action represented by the third target image includes the first action and the second action.
An interactive device applied to a first augmented reality device, including:

The detection module is used to detect the line of sight direction of the first user and the first action of the first user;

The first sending module is configured to send and to the second augmented reality device when the line of sight of the first user is projected on the area where the target identifier is located, and the first action of the first user meets the first preset condition Target information corresponding to the first action;

Wherein, the target identifier is an identifier in the projection area of the first augmented reality device, and the second augmented reality device is an electronic device associated with the target identifier.
The interactive device according to claim 11, further comprising:

The first output module is configured to output first prompt information when the response information sent by the second augmented reality device is received within the first preset time;

Or, the second output module is configured to output second prompt information when the response information sent by the second augmented reality device is not received within the first preset time.
The interactive device according to claim 12, further comprising:

The second sending module is configured to send the response information sent by the second augmented reality device to the The second augmented reality device sends the first information;

Wherein, the first information is used to control the second wearing glove to output tactile sensation.
The interactive device according to claim 11, wherein the first sending module comprises:

An acquiring sub-module for acquiring the action parameter of the first action;

A sending submodule, configured to send first target information to the second augmented reality device when the action parameter satisfies the first parameter condition, where the first target information is used to indicate the second augmented reality device Output prompt information in the first way;

Or, when the action parameter satisfies the second parameter condition, send second target information to the second augmented reality device, where the second target information is used to instruct the second augmented reality device to Two ways to output prompt information.
The interactive device according to claim 11, further comprising:

A first display module, configured to display a first image for representing the first action in the projection area;

The second display module is configured to receive response information to the target information sent by the second augmented reality device, and the response information is used to indicate a second action, according to the response information in the The projection area displays a third image, and the action represented by the third image includes the first action and the second action.
The interaction device according to claim 15, wherein the first action is a gesture action, the first augmented reality device is connected to a first wearable glove worn by the first user, and the interaction device further comprises:

The first control module is configured to control the first wearing glove to output the tactile sensation corresponding to the tactile sensation information in the case that the response information to the target information sent by the second augmented reality device includes tactile sensation information;

Or, the second control module is configured to detect the distance between the first finger cuff and the second finger cuff of the first wearing glove when the first action is an action of picking up a target item, and When the distance is less than a preset distance, the first finger cuff is controlled to output a first contact force, and the second finger cuff is controlled to output a second contact force, wherein the preset distance is based on the size of the target item The size is ok.
An interaction device applied to a second augmented reality device, including:

A first receiving module, configured to receive target information sent by a first augmented reality device, where the target information corresponds to a first action detected by the first augmented reality device;

The third output module is configured to output third prompt information when it is detected that the second user's second action satisfies the second preset condition within the third preset time, and send the information to the first augmented reality device Response information corresponding to the target information;

Or, the fourth output module is configured to output fourth prompt information when the second action of the second user that satisfies the second preset condition is not detected within the third preset time.
The interaction device according to claim 17, wherein the second augmented reality device is connected to a second wearable glove worn by the second user, and the interaction device further comprises:

The second receiving module is configured to receive the first information sent by the first augmented reality device;

The third control module is configured to control the second wearing glove to output tactile sensations according to the first information.
The interactive device according to claim 17, wherein the interactive device further comprises:

A fifth output module, configured to output the third prompt information in a first manner when the target information includes the first target information;

Or, the sixth output module is configured to output the third prompt information in a second manner when the target information includes the second target information.
The interactive device according to claim 17, further comprising:

A third display module, configured to display a first target image used to represent the first action in the projection area of the second augmented reality device;

The interaction device further includes:

The fourth display module is configured to display a third target image in the projection area of the second augmented reality device, and the action represented by the third target image includes the first action and the second action.
An augmented reality device, comprising a processor, a memory, and a program or instruction stored on the memory and capable of running on the processor, and the program or instruction is executed by the processor to implement claims 1 to 6. Steps in the interactive method according to any one of claims, or implement the steps in the interactive method according to any one of claims 7 to 10.
A readable storage medium, wherein a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps in the interaction method according to any one of claims 1 to 6 are implemented, Or implement the steps in the interactive method according to any one of claims 7 to 10.
A chip, wherein the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is used to run a program or an instruction to implement the method according to any one of claims 1 to 6 The steps in the interactive method, or the steps in the interactive method according to any one of claims 7 to 10 are implemented.
An interaction device, wherein the interaction device is configured to perform the steps in the interaction method according to any one of claims 1 to 6, or to implement the interaction according to any one of claims 7 to 10 Steps in the method.