Synonyms

Assisted healthcare; Realism; Virtual human character

Definition

Modern healthcare is constantly discovering and exploring the contributions that virtual reality can make to healthcare. The adoption of virtual human characters has had a success growth, in the past decades, in education, movies, games, and embodied conversational agents (ECAs). However, the growing interest of virtual reality in healthcare might stimulate a renewed interest in virtual human character. Therefore, this paper introduces the potential application areas of virtual human character (VHC) in healthcare from the point of view of realism and draws attention to the technology behind VHC as it pertains to realism.

Introduction

Modern healthcare is constantly discovering and exploring the contributions that virtual reality can make to healthcare. The usefulness of virtual reality in healthcare was discovered as far back as the 1990s during a surgery planning when a team of medical practitioners needed to visualize some complex data for a surgery (Chinnock 1994). Whenever the term “virtual reality” is used, attention is usually directed toward a three-dimensional (3D) imaginary world that allows for integration of human senses and interaction within a computer-generated environment using devices, such as head-mounted display, DataGloveTM, or a joystick (Satava 1995). However, there is the other aspect of virtual reality that is minimally discussed in literature, the virtual human character. Virtual human character (VHC) is a computer-generated human look-alike that could act as replacements for human in virtual environments. The VHC, this paper is particularly concerned with, is the one to do with the human face. The adoption of virtual human characters has had a success growth in other aspects of life, such as education, movies, games, and embodied conversational agents (ECAs).

Virtual human character as human look-alike should be able to express facial expression, facial emotions, and exhibit interaction characteristics, and these are fundamental components in achieving realism. Currently, there is a growing interest of virtual human in healthcare. Though the “uncanny valley” has in some ways slowed the focus on VHC in other application areas (Kokkinara and McDonnell 2015), in healthcare, it should never be an object of concern. The reason is this: imagine being a medical student and the virtual agent instructor lack human characteristics, will you have a committed interest and engagement during the training? Realism for virtual humans is key for such a scenario. Therefore, VHC should be developed in such a way that the “uncanny valley” effect is minimized/negligible as much as possible. Simply put, uncanny valley is human look-alike characters that lack real human emotions.

A great deal of healthcare systems requiring virtual humans needs realism to sustain the goals of virtual human adoption. A study in Seyema and Nagayama (2007) show that higher levels of animation of virtual faces are better perceived as being appealing. The question is: does this apply to all applications? This paper serves to assemble at one point the application areas of virtual humans in healthcare with emphasis on the dependency of the VHC on realism based on the application areas. Also, highlight will be thrown on current state of VHC research and technology relevant to virtual human assisted healthcare.

The rest of the paper is organized as follows. In section of “Virtual Human Application in Healthcare”, the application areas in healthcare where virtual human requires realism is presented. The technology, through literatures, behind virtual human as it pertains realism are discussed in section of “Virtual Human Character Technology”. In section of “Virtual Human Character Challenges” is a highlight on observable challenges of the virtual characters in realism. The conclusion is presented in the last section.

Virtual Human Application in Healthcare

Virtual human in healthcare has several applications, some of which are identified and discussed independently as listed in Magnenat-Thalmann and Thalmann (2004).

  • Virtual patients for surgery and plastic surgery

  • Virtual humans for treatment of social phobia and virtual psychotherapies

  • Virtual teachers for distance learning, interactive assistance, and personalized instructions

  • Virtual human for simulation-based learning and training

Virtual Patients for Surgery and Plastic Surgery

During surgery planning, surgeons usually simulate how a procedure will play out on a given patients face by using a simulated image of a patient. The simulation offers an avenue for the patient to build confidence in the doctor and can also help doctors understand and practice the procedure that best fits a given patient. Virtual humans are used in surgery to practice aesthetics and conduct maxillofacial surgery, oral surgeries, etc. Since the interest is more on the face of virtual humans’ character, the discussions here are limited to literatures with emphasis on the face of the virtual character. The research work by Patel et al. show that high face fidelity of virtual humans helps surgery patients interact easily with VHC (Patel et al. 2013). The use of virtual humans in patient surgery is one of the oldest areas of virtual human application in healthcare (Raibert et al. 1998; Altobelli et al. 1993; Rosen et al. 1996). The study in Karaliotas (2011) used virtual patients to provide normal and pathological conditions in the simulation of human anatomy and with emphasis on realism. In Smith et al. (2005) a virtual reality model is created for simulating aesthetic surgery procedures. The adoption of virtual humans in surgery help patients know and experience what can be expected from a surgery (Khor et al. 2016). Thus, the patient and doctor can have a good understanding of successes and implications of certain procedures.

Virtual Humans for Treatment of Social Phobia and Virtual Psychotherapies

In Sjolie (2014) virtual characters are proposed as a tool for helping dementia patients and persons who are experiencing cognitive decline. However, the extent to which realism is to be attained for this use case is not yet clear. What is clear is to be able to help fight phobia among social phobic individuals requires the scene of commonly experienced social woes to be constructed to mimic real-world scenario (Didehbani et al. 2016). The scene alone is not enough; rather the manipulation of the virtual human verbal and behavioral responses is an avenue to assert control of the level of therapy measures invoked with the use of VHCs (Brinkman et al. 2008).

Anxiety is one of the medical disorders that can be medically treated through psychotherapy. The treatment process involves psychological counseling which comprises of cognitive-behavioral therapy (CBT), psychotherapy, or a combination of therapies. Pan et al. (2012) through their study showed that presenting socially anxious participants to a VHC help in dealing with situations that might trigger anxiety. The VHC does not necessarily have to be human look-alike nevertheless should be able to exhibit human facial gestures and expressions, give and maintain eye contact, and talk and interact with the human participant (Pan et al. 2012; Grillon et al. 2006).

Virtual Teachers for Distance Learning, Interactive Assistance, and Personalized Instructions

The teacher’s role in a learning environment is void of learning without interaction, eye contact, exhibiting emotions, and expressing such emotions. A virtual agent should easily assume these behaviors and characteristics within a learning environment to be able to meet the expectations and acceptance of its human users.

Healthcare, particularly medical and nursing education and training, often involves the acquisition of knowledge through use of voluminous study materials which are constantly being updated, thus presenting a challenge to learning. Virtual reality has stimulated the increasing excitement on the integration of virtual humans in medical training process. According to a study on support for technology in healthcare education in 2010 Kron et al. (2010), 98% of over 200 medical students supported the inclusion of virtual teachers in healthcare training. Another study in Gromala et al. (2015) suggest that training of mindfulness meditation, a healthcare intervention program which has been used over decades to reduce addictive behaviors, is more effective with the use of virtual humans than with humans themselves.

Virtual Human for Simulation-Based Learning and Training

Realism has been our focus so far, but it is more heightened here because simulation-based learning and training for medical practitioners require high level of realism. Why is this the case? The simulation should mimic the real thing. For instance, imagine that virtual human suffering from internal pains (depicting a real patient; see Fig. 1) is presented as practical bed for junior doctors, the virtual human should be able to express pains when touched at various locations of the body pain is experienced. A VHC that lacks the ability to show emotions and express feelings will send across a different skill set and knowledge other than the required to the team of the junior doctors being trained. Therefore, VHC in simulation-based learning for healthcare practitioners should be able to simulate the real thing. The application of VHC for simulation-based learning and training in healthcare are endless. The study (Bertrand et al. 2010) used virtual humans to simulate hand hygiene practices for the full medical hygiene cycle: doctor, nurse, and patient. According to Zielke et al. (2010), a huge advantage of virtual humans in simulation of reality for medical education is that the learning environment and scenario of design can be adapted to simulate various levels of medical conditions; high-risk and low-incidence cases. The trainees can as well be able to learn the various factors such as patient regional, social, economic, behavioral, and cultural factors that can influence diagnosis and treatment sessions with a patient.

Virtual Human for Assisted Healthcare: Application and Technology, Fig. 1
figure 224figure 224

Medical simulation-based training. (Source wiki)

Full size image

Virtual Human Character Technology

In the preceding session, various literatures point to the significance of realism of VHC as it pertains to healthcare. In this section, we seek the technology as identified in literatures for achieving realism for virtual humans. Therefore, the reviews of literature are based on the underlying methodologies that are currently used for VHC realism and are discussed under key topics in the succeeding subsections.

Facial Action Coding and Expression

The facial action coding system (FACS) is adversely applied for understanding virtual character model (Kopp et al. 2006). In Shapiro (2011) FACS was used to achieve virtual character simulation, control, and realism. (Čereković and Pandži 2011) employed neural networks together with a manual synthesis of face expressions, speech, and gestures with facial animation model. This work was improved by Wang et al. (2010). Their work made use of the action units (AUs) which defines 46 points on the face. These units are assumed to house the regions of the face facial muscles or group of muscles contract and reveal the face emotions. The Fig. 2 shows the simpler AUs for a typical human face. The arrows show the directions of movement of the AU, while the endpoint red arrows are the center of mass for emotion. To achieve facial expressions alongside continued human dialogue emotions, latent semantic analysis concept is adopted for embedded conversation for virtual characters (Zhang et al. 2013).

Virtual Human for Assisted Healthcare: Application and Technology, Fig. 2
figure 225figure 225

Action units and main codes on the face (Morten Havmøller Laursen 2012)

Full size image

Character Deformation and the Physics Model

Facial deformation can be achieved through manipulating on element-wise basis, mesh units by some deformation functions. Multi-facial expressions can be created by combining muscles, though not the structure, and representing spline surfaces as hierarchical B-splines (Thiebaux et al. 2008). A muscle model was created by Platt and Badler in 1981 using the human facial structure and to generate multi-facial expressions; they applied the muscle arcs to flexible meshes. This approach was also adopted in (Ali et al. 2018), and the efficiency of synchronizing speech with the virtual character was evaluated using a context-dependent viseme. The observed that the number and distribution of fiduciary points effect on the quality of viseme presentation. The study in Leigh and Zee (2006) on physics muscle resulted in Greta: a virtual agent developed by Pasquariello and Pelachaud (2002) to synchronize gestures with speech to coordinate.

Performance-Driven Virtual Character

The performance-driven VHC is useful in circumstances where difficulties in controlling facial animation result from inaccuracies of movement tracking. This is a significant issue in real-time healthcare scenarios, especially for simulation-based training. In simulation-based training, interactivity of characters must be ensured to naturally combine motion and expressions. Deng and Neumann (2008) proposed to synthesize an expressive speech animation system through control at the phoneme level. In Hofer et al. (2008) an automatic lip-motion trajectory synthesis of the ECAs was presented. The work was extended by Zoric et al. (2011). They used animation trajectories achieving desired lip-speech synchronization with the ECA.

MPEG-4

Several literatures adopt the MPEG-4 technique in facial animation (Parke and Waters 2008). The MPEG-4 is used for generating animation based on three face data; face definition parameters (FDP), the face animation parameters (FAP), and the FAP Interpolation Table (FIT), which define the FAP interpolation rules. The FDP is used create 3D face geometry. The FAPs is designed to encrypt animation facial emotions, expressions, and speech pronunciation. After identifying 68 face parameters (see Fig. 3), they are animated through the face animation parameter units (FAPU). To achieve neutral facial emotion, the FAPUs are used to calculate distance between face units. Balci et al. (2007) created the Xface tool which is for the ECA using the MPEG-4 and achieved key-frame driving using the SMIL-agent scripting language. This method is visually depicted in Fig. 4. The purpose of their method is to be able to mimic human expression and emotion in the absence of gesture.

Virtual Human for Assisted Healthcare: Application and Technology, Fig. 3
figure 226figure 226

Some facial features points for the face as the MPEG-4 standard (Ali et al. 2018)

Full size image
Virtual Human for Assisted Healthcare: Application and Technology, Fig. 4
figure 227figure 227

The SMIL-Agent script processing (Balcı et al. 2007)

Full size image

Queiroz et al. (2009) used the MPEG-4 attain parameterization of face with a high-level facial actions and behaviors of virtual characters that can interaction human user. In Zoric et al. (2011) synchronized facial gestures and speech to virtual character using MPEG-4, it lacked emotions. Their work was improved in Kessous et al. (2010). In Kessous et al. (2010) a Bayesian classifier was used to identify the different types of emotions for different gestures, which they adopted to achieve human-like character but lacked face movements. Arsov et al. (2010) developed an approach that makes virtual character achieve interaction and learn cued speech.

Visual Speech Virtual Character

Facial expression alongside recorded speech is difficult to model. The reason is this: The human language comprises of vocabulary, phonemes, and speech coarticulation, which need to be integrated to achieve realism (Arsov et al. 2010). In Skantze and Al Moubayed (2012) the IrisTK, a toolkit designed for multiparty face-to-face interaction, while the Situation, Agent, Intention, Behavior, and Animation (SAIBA) is developed to compute ECA communication and human behavior realization (Bevacqua 2011), Berger et al. (2011) incorporated hissing and buzzing to speech synchronization with virtual character.

Virtual Human Character Challenges

The exhibition of facial expression, emotions, and interaction for VHC are not yet as good as that of a real human. However, just to avoid “uncanny valley,” it is widely acceptable if the VHC can communicate all the important expressions of emotions of the human, make eye contact, and engage in interactive activities.

To simulate the emotional state of the virtual character, it is important to determine facial expressions and behavioral characteristic that the model is to mimic prior to modeling, for example, lip-syncing synthesis animation. Lip-synching has existed for decades, but the real challenge originates in blending emotions with lip-synching to produce fuller and richer character animation. For model-based talking head systems, it is more elegant to work on the facial animation parameters to bring about coarticulation effects. Giving information to a user in the form of a talking head increases acceptance level of the VHC by the humans. Therefore, the integration of speech to an animated character may also make the use experience acceptable to users, especially for social phobic patients.

During synchronization of the VHC with human emotions identified during FAP, conversation should be simulated to obtain the varying emotional states of the human model, and care should be taken to ensure that the lexicon obtained contains enough emotions variables before encoding. This is to guarantee that different personalities and dissimilar emotions required in a dialogue are captured. More also, eye movements are very important factor for achieving realism for VHC. The believability of the virtual human is not necessarily of being human look-alike but in its ability to maintain facial behaviors that mimics human behavior, such as eye contact, blinking, and squinting.

Conclusion

The future holds a great potential for VHC in healthcare. The benefits of VHC in healthcare are endless and will serve to enhance the medical practices, therapies, and education. The successes of VHC in other fields of application, such as military, games, movies, and sports gave us a glimpse of its possibilities which will continue to drive the interest of VHC developers, researchers, and the medical practitioners, who are the healthcare end users. Therefore, there lies the challenge of achieving valuable experiences, limiting “uncanny valley” effect while attaining realism, between the virtual human and various application usage scenarios in healthcare. Finally, the design features that enables the VHC to assume human behavior and character which would make virtual human in healthcare enjoyable and readily adopted is still lacking. Though, the sole purpose should not only focus on human behavior but richly capture material content as well, especially for simulating an educational instructor or surgery.