Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design

Synonyms

3D object manipulation; Augmented reality; Gesture recognition; User interaction

Definition

Augmented reality (AR) has the ability to blend the virtual and real objects. It enables users to interact with virtual objects or images with a real object as a medium of interaction. Hence, to provide natural and intuitive user interaction in AR environment, real hands are widely explored as an input. This entry explains the natural user interaction using real hand gesture in AR and explains the ARDeco application development process. The fundamental guide to developing AR application for interior design using real hand gestures with Leap Motion controller to interact with the UI and manipulate the objects with hand gestures, including color swatch and texture change panel for UIs and translation and rotation, scaling, create object, delete object, and clone object.

Introduction

Augmented reality (AR) is a technology that allows computer-generated or digital information that includes text, videos, and two-dimensional (2D) or three-dimensional (3D) virtual images to be overlaid onto the real-world environment in real-time setting (Azuma 1997; Mann et al. 2018). AR technology can be a potential solution to create an intuitive way of interaction with the digital content, as it diminishes the boundary between the physical and virtual world by merging them in a same space inside the real-world environment. Even though many research has been carried out in creating various ways in enhancing the experience in using AR, interactions that are included in AR interfaces often constrained the users to interact freely and manipulate the AR content as well as prevent them from interacting with it more naturally and intuitively (Billinghurst 2001).

The use of conventional interaction in computers was common and has been used for most people until the current times. However, conventional interaction method that depends on mouse-keyboard-screen interfaces is divided between a space to act and perceive where the input and output devices are separated between virtual and physical spaces (Billinghurst et al. 2015). Hand gestures are one of the possible inputs that can be used in interacting and manipulating the AR content in a natural manner.

User Interaction in Augmented Reality

Researches in human-computer interaction (HCI) explore the ways for intuitive user interaction with an interface. Hand gesture is one of the most influential methods applied in HCI and one of the natural, ubiquitous, and important parts of spoken language (Lin and Schulze 2016). However, for mixed reality (MR), the Hololens device has used user’s real hand gestures to catch user’s input, while Magic Leap still uses controllers to capture user’s input. Hololens and Magic Leap both run independently without the need of computer processing power because it already has its processing power installed within them making them wireless and mobile. Hololens also focused on the development of the virtual user interface (UI). According to Evans et al. (2017), traditional work on AR instructions provided by HMD is limited due to technological defects, but extensive works more into 3D UIs and user interactions can provide insight into potential solutions. However, 3D UIs need a change from 2D WIMP to 3D menus and interactions without a usual mouse. The biggest problem is that all user interactions must be intuitive and accessible through hand gestures rather than a wall, controller, or other physical instruments for optimal functionality. As presented in Table 1, holographic devices such as Microsoft Hololens (2016) and Magic Leap (2018) use a see-through lens in their headsets that enabled the user to see their physical environment while digital information is projected at their respective lenses. Holographic devices still allow the user to see their physical environment where immersive devices, such as Oculus Rift and HTC Vive, purposely obstruct the user’s view from the physical environment with their opaque headset and substitute the user’s view with a fabricated digital environment. Table 1 shows the comparison between the holographic and immersive devices.

Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design, Table 1 Devices and experience

We examine one of the primary interaction tasks for the AR-based interior design application which is virtual object manipulation. Therefore, there is a need to enhance user’s experiences in the augmented world by providing natural user interaction technique for natural and intuitive interaction (Ismail et al. 2015). Hence, there is a need for a better interaction device that uses natural hands and avoids the restriction of hardware between humans and machines to interact with virtual contents (Billinghurst et al. 2015) and promote six degrees of freedom (DOF) to make the experience more engaging in AR-based applications (Ismail and Sunar 2015). In addition, since AR technology is dependent on marker tracking and registration, the problem would be whether the marker is from interacting by hands or other means of physical objects (Azuma 2017); thus, it is important that real hand gesture interaction should be occlusion-free, that is, the hand should not block the marker for seamless tracking (Ismail et al. 2015). This entry aims to describe the implementation of gesture-based object manipulation technique for 3D interior design using real hand gestures in AR.

Virtual Object Manipulation

This section explains the related works and existing projects that have been done by others. Some of the notable works in Tangible AR interface are VOMAR (Kato et al. 2000), Move the Couch Where (Irawati et al. 2006), Studierstube (Szalavári et al. 1998), and FingARTips (Buchmann et al. 2004). As explained in VOMAR (Kato et al. 2000), it has produced a scene assembly prototype of an interior design application for AR. The application consists of a book that holds the virtual models of furniture, a large piece of paper as the workspace, and a marked physical paddle as the interaction metaphor to interact with the models. The virtual object can be manipulated with simple paddle gestures applied to it. Also, an extended and modified version of VOMAR (Irawati et al. 2006) integrates the existing VOMAR prototype. The prototype’s functionality is still retained, but it integrates with a speech input system Adriane for specifying commands in object interaction and manipulation. Studierstube project (Szalavári et al. 1998) was a pioneer collaborative AR application that was meant to demonstrate the interface performance using 3D interactive media simultaneously in a collaborative environment. The users wear an HMD, allowing them to view the virtual models in collaboration as it superimposed in real physical world. FingARTips (Buchmann et al. 2004) is an urban planning-based workspace in AR environment that uses a vision-based interaction method in capturing the glove with fiducial markers applied on the wearable glove. The captured part of the hands (thumb and index fingers) can be used to pick up and place the virtual buildings around the workspace. The AR-based mobile manipulation application (Hürst and Van Wezel 2013) was developed to investigate the potential of finger tracking for the use of hand gestures in manipulating objects in handheld devices. This includes evaluation of canonical operations in basic manipulation techniques, such as translating, scaling, and rotating objects.

AR-based technology has provided users various interaction means with the virtual objects (Lin and Schulze 2016), and the progressive works in improving the current interaction issues are still ongoing until today (Kato et al. 2000). There is a need to have a better user interaction for AR-based application to make the user experience more intuitive and natural. By integrating virtual object manipulation with real hand gesture in AR environment for 3D interior design application, the user can interact with the virtual object more naturally and intuitively as it diminishes the borders of virtual and real-world environment.

As suggested by (Ismail et al. 2015), for the extended version of Ancient Malacca project (Ismail and Sunar 2015), they have developed an advanced method to improve the user interaction. They have implemented the multimodal interaction using gesture and speech input, as presented in Fig. 1. The project has implemented the sensor-based hand tracking; the motion data is presented with virtual hand skeleton that is created every time the Leap Motion controller (Ismail and Sunar 2015) senses a real hand in the motion space. The hand skeleton is being drawn in the AR camera view consistent with the position of the image target. The key to high-precision natural hand gesture interaction is high accuracy and high degree of freedom (DOF) hand pose estimation. Their method has proven that it was able to provide 6 DOF natural hand tracking; tracking of the wrist, 15 joints, and five fingertips for both hands at 30 frames per seconds (fps).

Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design, Fig. 1

Multimodal interaction technique (Ismail et al. 2015)

Real Hand Gesture Tracking

The following sections describe the gesture recognition process that enables 3D object manipulation for natural gesture interaction using ARDeco (Syafiqa and Ismail 2018) as an example application.

Stage 1: Gesture Recognition Process

For gesture recognition process, Leap Motion is being used to obtain a depth data. Four stages for the sensor-based recognition process need to be accomplished, as illustrated in Fig. 2. Firstly, the depth data is reproduced by Leap Motion in real time once the device captured the human hands. Next, the tracking process continued to produce pose estimation containing position and orientation. Then, the camera viewpoint pose estimation searches for the hand gesture skeleton to call the gesture shape and 3D representation for both hands. Leap Motion uses raw processing data to perform tracking. The motion data are processed with fingers, and 3D representation of what Leap Motion sees is reconstructed. After tracking the information, the fingers are extracted from the tracking layer. Position and orientation are used to map the 3D representation of the fingers and later used to invoke gestures. Gesture manipulations, including pinch, grab, and release, are implemented in the application. 3D hand skeleton-based interaction using a Leap Motion captures the hand skeleton and identifies 3D finger positions and orientations; thus, we can support a more natural hand gesture-based interaction in an AR scene.

Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design, Fig. 2

Hand gesture tracking method

Stage 2: Performing AR Tracking System

The application acts as a platform to integrate the AR tracking with the gesture interaction for 3D object manipulation. For AR to work, the application uses natural feature tracking for more robust tracking. Figure 3 illustrates the AR tracking devices. A personal computer or laptop is used to execute the application; Leap Motion controller is attached to the computer for gesture recognition, while a web camera is used to capture the printed marker. The marker acts as an anchor for the AR environment.

Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design, Fig. 3

ARDeco application setup

In Fig. 4, AR interface focuses on discovering the manipulation techniques for manipulating virtual objects with hand gesture, including how the user interacts in the AR environment with gestures and basic three-dimensional (3D) spatial manipulation that can be done. ARDeco has come out with a technique for virtual object manipulation to effectively provide an intuitive and natural way of interaction in AR environment. More about ARDeco will be explained in the next section.

Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design, Fig. 4

ARDeco application with user interaction

Stage 3: Acquiring 3D Gesture Inputs

The integration of virtual object manipulation using real hand gesture with AR application is described in this stage. The AR application is to use real hand gestures to interact with the virtual elements, including color swatch and a panel to change the virtual object texture. The 3D object manipulation such as translation and rotation, scaling, object creation, object deletion, and object cloning. These manipulation methods are implemented by ARDeco to run a robust virtual object manipulation technique.

The hand gestures for object manipulation tasks such as select or deselect object, move and rotate object, and scaling object are implemented with some modifications. Other manipulation tasks such as create, delete, and clone object do not have specific hand gesture, but the new gesture for these tasks follows the guideline that has been recommended by (Nielsen et al. 2003; Piumsomboon et al. 2014). Common interaction approaches for 3D manipulation are translation, rotation, and scaling. On top of that, users are allowed to pick, drop, change, or remove the selected virtual content from the workspace. The user’s real-time finger orientation and movement are obtained via Leap Motion device. Table 2 illustrates the gesture inputs that are used for manipulation task in ARDeco.

Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design, Table 2 Manipulation task description

ARDeco: Augmented Reality Interior Design

AR interior design, namely, ARDeco (Syafiqa and Ismail 2018), has a virtual panel that consists of four buttons that can be triggered by gesture in a small range of an AR scene called a virtual room. The room consists of several elements, which are four walls and a floor and a ground. There is also a furniture in the scene that is tagged with certain properties, and this furniture is manipulatable. The color and the texture of this furniture, as well as the walls and the floor, can be changed using gesture input by picking the desired colour inside the color swatch, and the texture can be changed by using the panel assigned to change the object’s texture. The color swatch button consists of a color swatch image that can change the object’s properties and the color of the furniture object’s material. To initiate this function, the user needs to hit the corresponding button using gesture input where it can change the color in real time. After that, the user can use his hand in pointing pose by extending the index finger to select the desired color available on the color swatch panel. The ARDeco application has only five types of furniture, which are a sofa, coffee table, chair, sideboard, and lamp. The furniture would be spawned in the AR scene when a user hits the creation panel. To perform object deletion, the delete button needs to be hitted, and it will only work if an object or furniture is selected. Object selection can be performed by simply touching the desired furniture in the scene.

There are three basic manipulation techniques applied in ARDeco application, which are translation, scaling, and rotation. The user interface of ARDeco lets the user interact with the virtual buttons to perform the manipulation tasks based on gesture inputs. The translation action provides the relocation task, and rotation action gives the furniture’s orientation value increase and decrease using the pinch method. The pinch method will be executed when the distance between the index finger and thumb reaches a certain threshold. The rotation action offers the ARDeco the ability to rotate the selected furniture based on the axes in AR interior scene. The scaling provides the furniture with the ability to resize. The task can alternatively be executed with keyboard commands. All of the methods mentioned above will only work if there is or are furniture selected.

Figure 5a shows the overall look of the ARDeco environment with the four buttons that can be seen on the right side of the figure. The virtual room consists of several furniture such as sofa, lamp, coffee table, and cupboard as well as the walls and the floor. The first button, which is the color swatch button, is for changing the selected furniture’s color (as in Fig. 5b). The second button is for changing the texture of the selected object, and the user is given 12 types of textures to choose from (as in Fig. 5c). The third button which is the object creation panel is for adding new furniture into the AR scene, and there is a selection of five furniture types to choose from (as in Fig. 5d). The fourth button, which is the object deletion panel, is responsible for removing or deleting the selected furniture from the AR scene (as in Fig. 5e).

Object Manipulation Using Real Hand Gesture for Augmented Reality Interior Design, Fig. 5

ARDeco with real hand gesture interaction