ES2716012B2 - INTERACTION SYSTEM AND METHOD IN VIRTUAL ENVIRONMENTS USING HAPTIC DEVICES - Google Patents

Description

DESCRIPTION

SYSTEM AND METHOD OF INTERACTION IN VIRTUAL ENVIRONMENTS USING

HAPTIC DEVICES

FIELD OF THE INVENTION

The object of the invention is framed in the field of computing, more specifically in that of haptic devices and virtual simulation.

BACKGROUND OF THE INVENTION

The techniques designed to control the interaction by haptic devices can be classified into three groups according to their mode of operation: based on position control, based on speed control and hybrid techniques. In the first place, techniques based on position control use the position of the end effector of the haptic device - within the physical workspace - to, consequently, position its virtual avatar. Second, techniques based on speed control move the virtual avatar according to the deviation of the end effector and with respect to the center of the workspace, as if it were a joystick. Finally, hybrid techniques combine position control and speed control in the same, adding a mechanism that allows them to be interchanged.

Among the interaction techniques based on position control are clutching [1] and direct or scaled mapping [2]. Clutching is an adaptation to haptic devices of the technique used in desktop mice. It consists of lifting the mouse and putting it back in a more comfortable position when with a single movement it is not possible to reach an area of interest. Applied to haptic devices, it consists of decoupling the movement of the end effector from the movement of the virtual avatar while pressing, for example, a button. In this way, it is possible to repeatedly perform such a maneuver and thus reach any part of the virtual scene with a haptic device having a limited working space. The direct mapping or scaling, performs a correspondence between the workspace of the haptic device and the entire virtual scene, establishing a scale factor which implies that in large scenes a small movement of the haptic device produces a large movement in the virtual avatar, seriously penalizing accuracy.

Another technique based on position control is the one presented by Conti & Khatib [3], consisting of continuously repositioning the end effector towards the center of the workspace without updating the visual feedback. In other words, while interacting with a virtual object, the end effector gradually moves — and with smooth movements — towards the center of the physical workspace, without affecting its visual appearance in any way, and therefore , without the user noticing.

Hybrid techniques comprise two modes of operation: one based on position control and the other based on speed control. These techniques focus on solving the problem of mode change, as is the case in the proposal by Liu et al. [4]. In it, it is proposed to maintain a neutral area around the center of the working space of the haptic device. Thus, when executing the mode change, the position desired by the user is not disturbed, since said change is associated with the pressing of a button — which can lead to unwanted movement of the end effector.

Finally, the so-called bubble technique also falls within the hybrid interaction techniques and consists of virtually dividing the working space of the haptic device into two sections: an inner bubble and the rest of the outer working area. A semitransparent bubble is visually represented in the virtual environment and, within the bubble, good precision is achieved thanks to a 1: 1 mapping; while in order to reach any part of the environment it is necessary to move the bubble. To do this, making force with the virtual avatar out of it, it will move in the direction of the avatar, at a higher or lower speed depending on the applied magnitude [5].

Bibliographic references

[1] Dominjon, L., Perret, J., & Lécuyer, A. (2007). Novel devices and interaction techniques for human-scale haptics. The Visual Computer, 23 (4), 257-266.

[2] Hirzinger, G., Brunner, B., Dietrich, J., & Heindl, J. (1993). Sensor-based space robotics-ROTEX and its telerobotic features. IEEE Transactions on robotics and automation, 9 (5), 649 663.

[3] Conti, F., & Khatib, O. (2005, March). Spanning large workspaces using small haptic devices. In Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2005. World Haptics 2005. First Joint (pp. 183-188). IEEE.

[4] Liu, L., Liu, G., Zhang, Y., & Wang, D. (2014, August). A modified motion mapping method for haptic device based space teleoperation. In Robot and Human Interactive Communication, 2014 RO-MAN: The 23rd IEEE International Symposium on (pp. 449-453). IEEE.

[5] Dominjon, L., Lecuyer, A., Burkhardt, J. M., Andrade-Barroso, G., & Richir, S. (2005, March). The "Bubble" technique: interacting with large virtual environments using haptic devices with limited workspace. In Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2005. World Haptics 2005. First Joint (pp. 639-640). IEEE.

DESCRIPTION OF THE INVENTION

The invention relates to a method and system of interaction in virtual environments using haptic devices. The invention proposes a new interaction model for haptic devices that solves their main problem, their limited work space, allowing the use of this type of device with a limited work area as the main interaction elements, achieving a high level of mobility. and precision.

The present invention allows haptic devices to be used for natural and accurate interaction with virtual scenes. The interaction model is based on making enlargements on an initial virtual scene using a haptic device (so the enlargement carried out can be called “haptic zoom”), which allows the operator to carry out a first phase of approach to the area of interest of the scene to later activate the haptic zoom and have greater precision. Therefore, given a virtual scene in which a haptic device is used, the haptic zoom consists of making one or more enlargements of it to achieve a higher level of haptic precision.

In each of the enlarged scenes, the workspace available for the virtual avatar of the haptic device is mapped directly with the space that represents the virtual scene, thus allowing the avatar to reach anywhere in the scene. If the user requires a higher level of precision over any area of the scene, the haptic zoom is activated. This fact affects the realization of a visual zoom on the area of interest, giving rise to a second scene. With the new scene already determined, it is mapped again the workspace of the haptic device with the space that represents the new scene, once again allowing the virtual avatar to reach any zone.

The method of interaction in virtual environments according to the present invention comprises the following stages:

- Detect a zoom command of an initial virtual scene.

- Generate, by a graphic processing unit, a new virtual scene from the initial virtual scene with a magnification level modified according to the detected zoom order.

- Map the workspace of a haptic device with the space that represents the new virtual scene.

- Represent the new virtual scene on a display device, preferably in a virtual reality viewer (although it could also be represented on a screen).

The zoom command can be an instruction that makes it possible to progressively modify the magnification level of the initial virtual scene or to change the magnification level of the initial virtual scene according to a certain predefined value.

Detection of the zoom command can be performed by the haptic device itself, preferably by detecting a button press on the haptic device. In this case, the method comprises sending the detected zoom command to the graphics processing unit. Additionally, or alternatively, the detection of the zoom command can be carried out by recognizing a voice command. In this case, detection can be performed directly by the graphic processing unit by analyzing the signal captured by a microphone.

A second aspect of the present invention refers to an interaction system in virtual environments that implements the method described above. The system comprises a haptic device for interacting with a virtual environment, a graphic processing unit in charge of generating scenes from the virtual environment, and a display device for representing the virtual scenes generated. The graphics processing unit is configured to receive a zoom command of an initial virtual scene; generate, from the initial virtual scene, a new virtual scene with a magnification level modified by function of the zoom order received; and mapping the workspace of the haptic device with the space representing the new virtual scene.

There are also different ways to apply the haptic zoom depending on the location of the magnification and that of the virtual avatar.

• Zoom centered on the virtual avatar. In this zoom mode the position of the virtual avatar will be taken as the center of the scene after magnification. This zoom mode involves physically repositioning the end effector of the haptic device towards the center of the workspace to match the on-screen representation of the workspace. When making an enlargement of the scene centered on the virtual avatar (haptic pointer), in the resulting scene the virtual avatar will be in the center of it. To match the scene with the physical device (effector), it must be automatically repositioned in the center of the workspace, so that both the virtual avatar is in the center of the scene and the effector is in the center of the scene. work space. This correspondence will allow the movements allowed by the (limited) workspace to reach any point in the virtual scene. Automatic repositioning is accomplished by applying forces from the haptic device toward the center of your workspace.

• Zoom keeping the proportions. In this zoom mode, the scene is magnified by keeping the virtual avatar at the same distances in percentage from the edges of the scene, so a physical repositioning of the end effector will not be necessary.

The present invention solves a problem inherent in haptic devices, their small workspace. Applying the model described by the invention to haptic devices allows them to be used in a more natural and precise way as main interaction devices in different types of simulations from different technical fields (medical sector, automobile industry, aerospace industry, etc.).

BRIEF DESCRIPTION OF THE DRAWINGS

Next, a series of figures are very briefly described that help to better understand the invention and that expressly relate to an embodiment of said invention that is presented as a non-limiting example thereof.

Figures 1A and 1B represent a flow chart of the interaction method in environments virtual according to two possible embodiments of the present invention.

Figures 2A and 2B show, according to a possible embodiment, the interaction system in virtual environments in operation, when the user performs a haptic zoom.

Figure 3 represents the relationship between the images per second ( frames) and the zoom level applied.

PREFERRED EMBODIMENT OF THE INVENTION

The present invention relates to a method and system of interaction in virtual environments using haptic devices.

Figure 1A represents a main diagram of the interaction method in virtual environments according to the present invention, through which a zoom is performed (either to zoom in or out of the image) using a haptic device. This type of zoom enabled by a haptic device can be called "haptic zoom."

A user performs an action that generates a zoom command 102 to enlarge or reduce an initial virtual scene, which is the image displayed to the user through a virtual reality viewer. The action may consist, for example, of pressing a button on a haptic device. The zoom command is detected 104 by the haptic device, which sends 106 the detected zoom command to a graphics processing unit.

Next, the graphic processing unit generates 108 a new virtual scene from the initial virtual scene with a magnification level modified according to the received zoom order, and maps 110 the working space of the haptic device with the space that represents the new virtual scene. Finally, the new virtual scene is represented 112 in a virtual reality viewer (which will normally be carried by the same user).

The zoom order includes at least the information indicating whether to enlarge or reduce the initial image. Furthermore, the zoom order may contain information on the zoom level to be applied on the initial virtual scene, although said information is preferably preset in the graphic processing unit (eg by means of a variable stored in memory and modifiable by the user using an application. ). The zoom order can also include information regarding whether it is a point zoom or a continuous or progressive zoom to be applied to the initial virtual scene. In the case of progressive zoom, the command is preferably sent continuously as long as the haptic device detects the user's action (eg as long as the user holds down a button). In this way, the graphic processing unit will generate, according to a determined refresh rate, successive images (ie "frames") enlarged or reduced from the initial virtual scene, according to a certain level of enlargement between successive images, until it stops to receive the zoom command. Due to the use of a high refresh rate (eg 30, 60 or 90 frames per second) and the accelerated count of successive images, the user fluidly perceives a zoom of the initial virtual scene while performing the zoom command action (eg as long as you hold down a button).

Figure 1B represents a diagram of the interaction method in virtual environments according to another possible embodiment. This embodiment differs from the previous embodiment in the zoom command detection mode. In the case of Figure 1A, detection 104 is performed by the haptic device, by detecting a button press on the haptic device. In the embodiment of Figure 1B detection 105 is performed by the graphic processing unit by detecting a voice command from the user captured by a microphone.

Figure 2A shows the different elements of the interaction system in virtual environments according to a possible embodiment of the present invention. In particular, the system comprises a haptic device 204 for the user to interact with a virtual environment, a graphic processing unit 206 responsible for generating scenes from the virtual environment, and a display device 208 for representing the virtual scenes generated. According to the embodiment shown in Figure 2A, the display device is a virtual reality viewer 208, although it can also be a normal screen (eg a monitor, a television).

According to the embodiment of Figure 2A, the graphic processing unit 206 is a separate and independent entity from the virtual reality viewer 208. Said graphic processing unit can be implemented, for example, by a computer connected by cable 210 (or by connection high-frequency wireless according to recent technologies, eg WiGig) to the virtual reality viewer 208. However, in another embodiment (not shown in the figures) the graphic processing unit 206 and the virtual reality viewer 208 may be the same entity , that is, the graphics processing unit 206 may be integrated into the stand-alone virtual reality headset 208 (eg Oculus Go). In another embodiment, the graphic processing unit 206 and the display device can be implemented in a smart mobile phone that is attached to a support (eg Samsung Gear VR virtual reality headset) for correct use.

The haptic device 204 is configured to detect a zoom command to enlarge or reduce an initial virtual scene 212, shown by way of example in the figure. The zoom command can be detected by the haptic device 204 by detecting the pressing, by a user 202, of a button 205 of the haptic device 204. The haptic device 204 sends 106, preferably wirelessly, the zoom command detected (and suitably processed) to graphics processing unit 206.

As shown in Figure 2B , the graphic processing unit 206 generates, from the initial virtual scene 212, a new virtual scene 216 with a magnification level modified as a function of the received zoom order (in the example, the zoom order is magnification, but it could be reduction instead). As indicated above, the level of enlargement or reduction to be applied (eg 1.5x, 2x, 4x, ...) may be predetermined by the graphic processing unit 206 (eg a data stored in memory) or it may be data included in the zoom order sent 106. The zoom to be performed can be one-off or progressive, depending on the zoom order sent 106 (for example, the zoom order can include a field that determines the type of zoom to apply, if punctual or progressive). Thereafter, the graphics processing unit 206 maps the working space of the haptic device 204 (the spatial coordinates where the haptic device moves) with the space representing the new virtual scene 216.

According to one embodiment, the graphics processing unit 206 may be configured to zoom into the initial virtual scene 212 centered on a virtual avatar, repositioning the haptic device 204 towards the center of the workspace.

In another embodiment, a zoom is applied in which the proportions are maintained from a virtual avatar to the edges of the scene. In this embodiment, the graphic processing unit 206 zooms in on the initial virtual scene, keeping a virtual avatar 214 (represented with a cross) at the same distance (Dx, Dy) from the edges of the virtual scene, as shown. represents in the example of Figures 2A and 2B, where the dotted lines represent the distance in percentage from the avatar 214 to the edges of the scene. Behind the activation of the haptic zoom, it is verified how the avatar 214 is kept at the same distance in percentage from the edges of the scene, which represents that both the end effector (ie the physical device that the user handles with the hand) and the pointer haptic (ie the representation of the end effector in the virtual environment) of the haptic device maintain their position.

Depending on the zoom order received or the configuration of the graphic processing unit 206, the executed haptic zoom can be a point zoom, so that each press of the button 205 of the haptic device 204 leads to a zoom in the image. scene with a certain predefined value, or a continuous zoom, where by holding down the button 205 of the haptic device 204, a progressive zoom will be applied to the scene until the user stops pressing the button, at which point the level is considered reached. desired magnification.

Figure 3 shows, for the execution of a continuous zoom, the calculation of the magnification level on the original scene that each frame should have during 1 second, both at 30 frames per second (upper part of Figure 3) and at 60 frames per second (bottom of Figure 3).

Given an initial scene eo and a position of the virtual avatar p measured in percentage to the edges of the scene, the haptic zoom is activated when the user holds down one of the buttons 205 of the haptic device 204. To achieve an acceptable level of fluidity in enlargement At least 30 small zoom enlargements are made every second, with 60 being the number of updates per second desired. Starting from the initial scene eo with a magnification value of 1.0 and a refresh rate of 30 enlargements per second, the value that the zoom should take in the subsequent scene, scene ei, is:

a x zoom ( e0)

zoom ( e1) = zoom ( e0) --------------------

The general formula for calculating the zoom level for a generic scene e, from the previous scene ei, is:

ax zoom ( ei_1)

zoom ( ei) = zoom ( ei_1) +

fps

In the above formula a is a constant and fps is the frame rate (in frames per second). In a preferred embodiment a = 0.3. By applying this formula consecutively for 30 iterations (one second) on the initial scene, with an initial zoom level of 1.0 ( zoom ( e0) = 1.0 ), a final zoom level of 1.3478 is obtained, so that the initial scene eo in a second you will have experienced a magnification level of 34.78% ( zoom ( e30) = 1.3478 ). For the 60fps example, the final zoom is 34.88% ( zoom ( e60) = 1.3488 ).

Once the zoom value of each scene e¡ has been calculated, that level of zoom will be applied to the previous scene in, in such a way that the resulting scene e ¡keeps the virtual avatar at the same distance in percentage from the edges of the scene than the previous scene e¡-i.

There are different ways to activate the haptic zoom according to the physical possibilities (e.g. buttons) available to the haptic device in question. Given the different availability of buttons on the different desktop haptic devices on the market, the activation of the haptic zoom (either in point zoom mode or in continuous zoom mode) can be done in several ways:

- Single button haptic devices:

• Spot zoom (constant zoom levels): Each press of the button on the haptic device zooms the scene to a certain preset value. Undoing a zoom level can be done with a quick double press of the button.

• Continuous zoom: By holding down the button on the haptic device, a progressive zoom will be applied to the scene until the user stops pressing the button, at which point the desired magnification level is considered reached. The action that will activate the reverse zoom will be a double press of the button on the haptic device while holding the button down after the second press.

- Haptic devices with at least two buttons:

• Spot zoom (constant zoom levels): Each press of one of the buttons on the haptic device will zoom the scene with a certain preset value. Undoing a zoom level can be done by pressing the other button.

• Continuous zoom: By holding down one of the buttons on the haptic device, a progressive zoom will be applied on the scene until the user stops pressing the button, which will be when the magnification level has been reached. wanted. The action that will activate the reverse zoom will be to press and hold the other button on the haptic device.

- Haptic devices without buttons: Using voice commands, the user can apply, constantly or continuously, both increases and decreases to the zoom level. The voice commands are captured through a microphone connected to the graphic processing unit 206 (e.g. a computer or the virtual reality viewer itself) in charge of representing the virtual environment. The graphic processing unit 206 receives, analyzes and executes the voice commands, applying the corresponding action on the virtual scene (i.e. activating the haptic zoom in the virtual representation).

The most suitable haptic zoom implementation is one in which a desktop haptic device with at least two buttons is used as the interaction method applying a continuous zoom in which the proportions from the avatar to the edges of the scene are maintained.

Claims (20)

1. Method of interaction in virtual environments, characterized by comprising: detecting (104, 105) a zoom command of an initial virtual scene (212); generating (108), by a graphic processing unit (206), a new virtual scene (216) from the initial virtual scene (212) with a magnification level modified as a function of the detected zoom order; mapping (110) the workspace of a haptic force feedback device (204) with the space representing the new virtual scene (216); represent (112) the new virtual scene (216). Method according to claim 1, characterized in that the zoom command is an instruction to punctually modify the magnification level of the initial virtual scene (212) according to a certain predefined value. 3. Method according to claim 1, characterized in that the zoom command is an instruction to progressively modify the magnification level of the initial virtual scene (212), where the zoom level applied to a scene ei is calculated from the zoom level applied to a previous scene e ^ according to the following formula: a x zoom {ei_1) zoom ( ei) = zoom {ei_1) ------------------------- where zoom ( e) is the zoom level of the scene e, zoom ( e¡-i) is the zoom level of the previous scene ei, a is a constant, fps is the frame rate. Method according to any of the preceding claims, characterized in that the detection (104) of the zoom command is carried out by the haptic device (204); wherein the method comprises sending (106) the detected zoom command to the graphic processing unit (206). Method according to claim 4, characterized in that the detection (104) of the zoom command comprises the detection of the pressing of a button (205) of the haptic device (204). Method according to any one of claims 1 to 3, characterized in that the detection (105) of the zoom command comprises the detection of a voice command. 7. Method according to any of claims 1 to 6, characterized in that the zoom carried out in the initial virtual scene (212) is centered on a virtual avatar (214), where the method comprises repositioning the haptic device towards the center of space of work. Method according to any one of claims 1 to 6, characterized in that the zoom performed in the initial virtual scene (212) keeps a virtual avatar (214) at the same distance in percentage from the edges of the virtual scene. 9. Method according to any of the preceding claims, characterized in that the new virtual scene (216) is represented (112) in a virtual reality viewer (208). Method according to any of the preceding claims, characterized in that the new virtual scene (216) is represented (112) on a screen. 11. Interaction system in virtual environments, comprising: a haptic force feedback device (204) to interact with a virtual environment, a graphic processing unit (206) in charge of generating scenes of the virtual environment, a display device to represent the virtual scenes generated, characterized in that the graphic processing unit (206) is configured to: receiving a zoom command of an initial virtual scene (212); generating, from the initial virtual scene (212), a new virtual scene (216) with a magnification level modified as a function of the received zoom order; mapping the workspace of the haptic device (204) with the space representing the new virtual scene (216). System according to claim 11, characterized in that the zoom command is an instruction to punctually modify the magnification level of the initial virtual scene (212), wherein the graphic processing unit (206) is configured to enlarge or reduce, depending on the zoom order, the initial virtual scene (212) with a predefined magnification level. System according to claim 11, characterized in that the zoom command is an instruction to progressively modify the magnification level of the initial virtual scene (212), where the graphic processing unit (206) is configured to progressively enlarge or reduce, as a function of the zoom order, the initial virtual scene (212) according to a determined refresh rate, and where the graphic processing unit (206) is configured to calculate the zoom level applied to a scene ei from the zoom level applied to a previous scene ei_1 according to the following formula: ax zoom ( ei_1) zoom ( ei ) = zoom ( ei_1) ----------------------- where zoom ( e) is the zoom level of the scene e, zoom ( e¡-i) is the zoom level of the previous scene ei, a is a constant, fps is the frame rate. System according to any one of claims 11 to 13, characterized in that the haptic device (204) is configured to detect (104) a zoom command of an initial virtual scene (212) and send (106) the zoom command to the graphics processing unit (206). System according to claim 14, characterized in that the haptic device (204) is configured to detect (104) the zoom command by detecting the pressing of a button (205) of the haptic device (204). 16. System according to any of claims 11 to 13, characterized in that the graphics processing unit (206) is configured to detect (105) the zoom command by detecting a voice command captured by a microphone. 17. Systems according to any of claims 11 to 16, characterized in that the graphic processing unit (206) is configured to: zooming in on the initial virtual scene centered on a virtual avatar (214), and repositioning the haptic device (204) toward the center of the workspace. System according to any one of claims 11 to 16, characterized in that the graphic processing unit (206) is configured to zoom in on the initial virtual scene, keeping a virtual avatar (214) at the same distance in percentage of the edges of the virtual scene. 19. System according to any of claims 11 to 18, characterized in that the display device is a virtual reality viewer (208). 20. System according to any of claims 11 to 18, characterized in that the display device is a screen.