Keywords

1 Background

With the fast development of the machine technology, augmented reality (AR) is a rapidly evolving field in human-computer interaction (HCI), since it helps humans to communicate interactively and also allows humans to visually interact with computing devices in various ways, particularly using gesture-based interaction. Recently, there are some surveys of gesture interaction and gesture recognition. For instance, in 2017, Asadi-Aghbolaghi et al. [1] reviewed systematically methods for gesture recognition and action in various image sequences using deep learning approaches. The fusion strategies, architectures, main datasets, and competition for gesture recognition and action were explained. For the architectures, as displayed in Fig. 1, they categorize the methods of convolutional neural network (CNN) based on how each method handles the temporal dimension of videos into three different groups: 3D convolutions, motion-based approaches, and sequential models. For fusion strategies, they categorize each strategy based on timing into three major groups: slow fusion, early fusion, and late fusion. Also in 2016, Kerdvibulvech [2] surveyed some important methods for gesture interaction for helping people with disability. A pose tracker and a combination of superpixels and support vector machine used in gesture interaction applications for supporting disabled people, such as hearing impairment, were explained. In this case, a pose tracker used six-degrees-of-freedom. More recently in 2018, Al-Shamayleh et al. [3] gave a good literature review on computer vision-based recognition techniques, specifically focusing on hand gesture recognition for sign language. Nevertheless, to the best of our knowledge, most of recent literature reviews have not focused on gesture-based interaction for using augmented reality in human-computer interaction applications. For this reason, we have a different goal from the aforementioned review works.

Fig. 1.
figure 1

Three different groups of the methods using deep learning based on how each method handles and deals with the temporal dimension of videos are categorized by Asadi-Aghbolaghi et al. [1].

Full size image

This paper discusses and studies various state-of-the-art methodologies that researchers, including the author, have recently attempted to achieve gesture-based interaction for using in augmented reality-based human-computer interaction applications. This paper is divided into two main parts: recognitions of hand gestures using augmented reality and utilizing augmented reality for gesture-based interaction. The first main part is, because hand gesture is usually very important and essential for an interaction medium in augmented reality-based human-computer interaction applications, that we explore the recognitions of hand gestures using augmented reality technology. More generally, the second main part is that we explore the possibilities of utilizing augmented reality for human-computer interaction applications of gesture-based interaction. The remainder of this paper is organized for easy to read and understand as follows. Section 2 presents and discusses the recognitions of hand gestures using augmented reality technology in this field of gesture-based interaction research. Next, Sect. 3 reviews the literature on utilizing augmented reality technology for human-computer interaction applications of gesture-based interaction. Finally, Sect. 4 summarizes and concludes the state-of-the-art methods on augmented reality-based human-computer interaction applications of gesture-based interaction.

2 Recognitions of Hand Gestures Using Augmented Reality

Gestures basically convey information through physical movements of some human body parts, such as face, body, hands, legs, and feet. Generally, the hand is utilized for recognitions of gestures compared with other body parts, so that hand gesture is usually very important for an interaction medium in augmented reality-based human-computer interaction applications. The first main part is that we explore the recognitions of hand gestures using augmented reality. For example, because hand gestures can be applied for navigating and manipulating big data, the interactivity of touching the cube with the markerless hand poses utilizing an augmented reality interface [4] is introduced. Their augmented reality cube-like framework is called Augmented Reality for Public Engagement (PEAR).

Furthermore, due to the rapid development of gesture-based interaction using hand gestures, a user-study of hand gestures to design an intelligent in-vehicle interface [5] is researched. Also, Hürst and Wezel [6] built augmented reality-based interaction metaphors on smartphones using finger recognition in front of a smartphone’s camera. In their system, people can see the live image of the smartphone’s camera, and then computer generated-contents are augmented in the scene that they look at. However, their limitation is about the markerless finger tracking and the user’s confusion when using the system. In other words, some users feel confused and face a cognitive overload. In addition, since human-computer interaction in many augmented reality applications requires a fast method for accurately hand tracking rapid, a hand motion analysis in real-time using adaptive probabilistic models is studied in [7]. Similarly, a hand tracking in real-time using convolutional neural networks is also proposed by Mueller et al. [8], as represented in Fig. 2(a), but they aim to handle occlusion problem from a sensor. Moreover, a hand tracking approach, as illustrated in Fig. 2(b), by using hand model and distance transform [9] in real-time for applying in augmented reality application in arts [10] is achieved in term of human-computer interaction principles. In this hand tracking approach, it can run in real-time, so that it is convenient to utilize in augmented reality-based human-computer interaction application. Alternatively, hand gestures can be interactively detected by the Leap Motion sensor, as suggested by Kim and Lee in [11] that uses for interacting with three-dimensional augmented reality objects for three-dimensional transformation to support usability of related-devices. In addition, Frikha et al. [12] presented a method for natural gesture-based interaction with virtual content and gestures from users (i.e., hands and fingers) in an augmented reality heart visualization interface. They used a computer vision-based technique to track gestures and then change the shapes of gestures on object commands.

Fig. 2.
figure 2

(a) Kinematic skeleton using convolutional neural networks for localizing the hand and regressing 3D joint locations is developed in [7]. (b) A model-based hand tracking approach for recovering of three-dimensional hand gesture using extended distance transform is implemented in [8].

Full size image

3 Utilizing Augmented Reality for Gesture-Based Interaction

The second main part is that we explore the possibilities of utilizing augmented reality for human-computer interaction applications of gesture-based interaction. As gesture-based interaction can be utilized in designing mass volumes of buildings, we explore a gesture-based interaction study on an enhanced mobile augmented reality environment. In 2018, Gül [13] presents a gesture-based interaction work for understanding the concept of the co-design cognition and interaction behavior of the building designers using augmented reality, as shown in Fig. 3. This gesture-based interaction study is expected to further help the development of the augmented reality-based human-computer interaction works for supporting the design activity using multi-touch user interfaces. In fact, the pilot work of Gül et al. for understanding the effect of the employment through smartphone-based augmented reality is introduced in [14] based on a co-design situation and co-modelling situation.

Fig. 3.
figure 3

A gesture-based interaction study augmented onto the sessions and the table for designing mass volumes of buildings is developed by Gül [13].

Full size image

In addition, due to a rapidly aging population’s possibility, we explore the augmented reality work for interacting gesture-based interaction. In [15], Sorgalla et al. builds gesture-based interaction using augmented reality for interacting with virtual world to manipulate a variable smart environment using Eclipse SmartHome for supporting senior citizens. Besides, due to the popularity of an untethered mixed reality headset called Hololens from Microsoft, a mixed-scale gesture design using a Hololens augmented reality display with wearable sensors is discussed in [16] by Ens et al. Their aim is to develop a system that can interleave interactively microgestures with larger gestures for human-computer interaction. Figure 4 depicts the demonstration of Ens et al.’s work using wrist-worn sensor mounted under the wrist. For example, a finger’s motion can manipulate the speed of a running/walking animation in the virtual world. In their similar study, led by Simmons et al. [17], they explore a comparative evaluation of two hand gesture recognition sensors for micro-gestures. In other words, they evaluate the capabilities of the devices to detect small movement using three distinct gestures.

Fig. 4.
figure 4

A wrist-worn sensor, built and designed by Ens et al. [16], is used for controlling animations in the virtual world and designing mixed-scale gesture using a Hololens augmented reality display.

Full size image

Furthermore, the multimodal augmented reality work is studied in [18] by Lugtenberg et al. for estimating the thickness of the objects, so it is able to allow people to adjust the perception of thickness (hollow or solid) of its material by changing some stimuli factors, including augmenting auditory, in the environment. In other words, their research goal is to find a different perception of thickness of any random material by just changing auditory feedback. However, their limitation is that audio alone is not always successful in every case. In some specific circumstances, it gives an incorrect answer by changing perception from solid to hollow. Therefore, by relying solely on audio alone, haptics and other modalities are believed to possibly help to address this problem. Figure 4 depicts the demonstration of this multimodal augmented reality work. In fact, the initial work of Lugtenberg et al. using multimodal augmented reality is also introduced in [19] for modulating the sound emitted by a material (either a hollow or solid cube) when touched based on the multimodal feedback psychophysically. From their experimental results, the Leap sensor can give more accurate results and lower difficulty than the Soli sensor with their test gesture set using three distinct gestures: a movement of the thumb and forefinger depicting a slider, a movement of moving a finger up and down, and a gesture of moving the hand up and down.

Moreover in 2018, Xue et al. [20] built an interactive hand rehabilitation supplementary system using augmented reality and gesture recognition. Their research purpose is to focus on helping the treatment activity of hand rehabilitation in any circumstance. In this way, their input device is Leap Motion device. Unity3D in this system is used as the development engine in three different parts: conventional training, augmented reality game training, and auxiliary functions. Therefore, because of different levels of challenges in each part using augmented reality, it can increase the attraction and difficulty of the rehabilitation process for users. Next, an optical see-through augmented reality system was explained in [21] by Zhen et al. for gesture-based interaction. In their system, Single-Point Active Alignment Method (SPAAM) based on RGB-D camera is used for virtual scene rendering, optical see-through calibration, and object tracking. Hence, they can achieve the mixture of virtual and real scenes for gesture-based interaction in real-time. Furthermore in early 2019, Aliprantis et al. [22] created a prototype system using gesture-based interaction approaches naturally for several interaction categories (rotation, scaling, and translation) in a Leap Motion device-augmented reality context. Figure 5 shows two different gestures used in their prototype system (high and low level of naturalism each). Their research focuses on gesture-based interaction methods and naturalism levels on the design of 3D user interfaces, called natural user interfaces (NUIs), in augmented reality framework. The Leap Motion device is integrated in a head-mounted display for building an augmented reality natural interface (Fig. 6).

Fig. 5.
figure 5

Augmented sound experiment is designed by Lugtenberg et al. [18] for receiving a perception of thickness of an object differently when tapping it using two cubes (solid and hollow).

Full size image
Fig. 6.
figure 6

Two different gestures used in [22] and presented by Aliprantis et al. for the three basic interaction categories.

Full size image

In fact, the developments of gesture-based interaction methods are not limited in using augmented reality. There is also the possibilities of utilizing virtual reality for human-computer interaction applications of gesture-based interaction. Li et al. [23] gave a good overview of gesture interaction in virtual reality. Generally, virtual reality gesture interaction devices are classified into three main types: touch screen-based interaction devices, wearable interaction devices, and computer-vision-based interaction devices. In recent years, a non-contact and non-expensive device, called Leap Motion, is usually used for building gesture-based interaction applications for virtual reality. For instance, Khan et al. [24] discussed and evaluated the effects of adding hand and finger gesture interaction in a virtual world for 360◦ panoramic movie watching experience. They used Leap Motion, and a SoftKinetic RGB-D camera for tracking the hand and finger movements and capturing the texture of the hands and arms virtually. In their evaluation, they tested four different cases: showing either a rigged virtual hand or a point-cloud of the physical hand, with and without interaction. Hence from their experimental results, when people can have interact with virtual embedded content in a 360◦ panoramic movie, they can feel strong embodiment and ownership from gesture-based interaction. More recently in December 2018, Céspedes-Hernández et al. [25] presented a system of gesture-based interaction for allowing people to navigate environments in a virtual world naturally. They focused on body gestures through user-defined commands within virtual reality environments. Therefore, people can give the commands using navigation tasks and the Wizard of Oz method for interaction virtually.

Therefore, according to our review and discussion, we propose that future directions for gesture-based interaction would be practically reinforced the way we design, develop and implement augmented reality and virtual reality applications. Therefore, we recommend the gesture-based interaction researchers to focus on recent methods of reality-based technologies, especially computer vision-based methodologies. General speaking, the possible methods of augmented reality and virtual reality we mentioned include accurate object tracking, automatic occlusion handling, and robust camera calibration.

4 Conclusions

In summary, we bring a discussion about gesture-based interaction methods for using in augmented reality-based human-computer interaction applications. We introduce the recent works of recognitions of hand gestures using augmented reality. We then outline the possibilities of utilizing augmented reality for human-computer interaction applications of gesture-based interaction. Although discussing about the advantages of each method, we note some limitations involved with using their systems in some specific situations. Therefore, we can understand how the interactions that humans would have with modern innovations do. We firmly believe that future directions for gesture-based interaction would be practically linked to the way we design augmented reality and virtual reality applications. Future work will continue the developments of gesture-based interaction methods for using in human-computer interaction applications of other extended reality (XR) technologies, including augmented virtuality (AV), in more hybrid approaches, such as the hybrid concept of model of human hand motion [26], the hybrid concept of model-based reactive control for robotic system in high dimensions [27], and the hybrid concept of virtual reality and spatial augmented reality [28], in an integrated cross-disciplinary area for the future of human-computer interaction.