Understanding the effects of hand design on embodiment in virtual reality

Abstract

Understanding user perceptions of interacting with the virtual world is one of the research focuses in recent years, given the rapid proliferation of virtual reality (VR) and driven to establish the metaverse. Users can generate a familiar connection between their bodies and the virtual world by being embodied in virtual hands, and hand representations can induce users’ embodiment in VR. The sense of embodiment represents the cognitive awareness of one's manifestation and includes three subcomponents: the sense of body ownership, agency and self-location. There is insufficient evidence in the literature about the effects of hand designs on the embodiment, especially based on studying its three subcomponents. This study investigates how virtual hand designs with five realism levels influence the three subcomponents of embodiment in VR. This research employs a self-report questionnaire commonly used in the literature to assess embodiment and evaluates agency and self-location by introducing implicit methods (intentional binding and proprioceptive measurement) derived from psychology. Besides, the objective data of eye tracking is used to explore the connection between embodiment and hand designs, and classifying participants’ eye tracking data to help analyze the link between embodiment and user attention. Overall, this research makes a major contribution through a systematic exploration of users’ embodied experience in VR and offers important evidence of the effects of virtual hand designs on body ownership, agency, and self-location, respectively. In addition, this study provides a valuable reference for further investigation of embodiment through implicit and objective methods, and practical design recommendations for virtual hand design in VR applications.

Introduction

The demands for virtual reality (VR) applications are increasing rapidly in various fields nowadays, such as entertainment, education, healthcare, and engineering, and there is a significant need to understand how people interact with the designed virtual world (Argelaguet et al., 2016; Yu et al., 2021; Wang et al., 2022). Different avatars and tasks can be created in VR by designers for specific virtual scenes. The avatars, or their parts (e.g., virtual hands) that are designed to represent users’ physical selves, play a central role in the self-presentation in virtual environments (VEs) (Argelaguet et al., 2016; Freeman and Maloney, 2021; Wang et al., 2022). Besides, it is worth noting that the characteristics of virtual avatars can affect the users’ attitudes and behaviors when they experience the activities in VEs (Freeman and Maloney, 2021).

The virtual world brings immersive experiences for users to gain different perceptions and feeling toward various avatars (Gonzalez-Franco and Peck, 2018; Yu et al., 2021). The sense of embodiment as one of the important user perceptions in VR is regarded as the feeling of owning, controlling, and being inside a virtual body (Kilteni et al., 2012), which corresponds to its three subcomponents respectively: the sense of body ownership (namely the feeling of owning the virtual body), agency (namely the ability to control the virtual body's actions), and self-location (namely the perceived virtual body's location relative to the physical self). Studies have found that the VE can induce the sense of embodiment with various avatar designs, including virtual bodies or hands, which have drawn the attention of researchers (Argelaguet et al., 2016; Zhang et al., 2020, 2022a, 2022b; Krogmeier and Mousas, 2021). However, few works explored the effect of virtual hands on the sense of embodiment by studying its three subcomponents, and only subjective measures were used to evaluate the embodiment in these studies (Ogawa et al., 2019; Heinrich et al., 2020).

To further explore how virtual hand designs affect users’ sense of embodiment, it can be beneficial to integrate subjective and objective data to achieve more accurate and reliable conclusions. Prior studies mainly used self-reports (questionnaires) to assess the embodiment subjectively. In this study, it is found that implicit measures originating from the field of psychology can be adopted for the embodiment research, such as the intentional binding for the agency (Nataraj et al., 2020a, 2020b) and proprioceptive measurement for self-location (Ingram et al., 2019). Besides, compared with the traditional research method, biometric measures are valuable methods to assist researchers in understanding the user experience (Borgianni and Maccioni, 2020; Krogmeier and Mousas, 2021). Recent studies suggested adding physiological detection to experiments for objective measurements, and eye tracking is a promising objective method for investigating user experiences in VR (Lu et al., 2019; Krogmeier and Mousas, 2020, 2021; Zhang et al., 2022a), such as eye tracking for exploring the sense of embodiment (Krogmeier and Mousas, 2020, 2021; Zhang et al., 2022a). In addition, the degrees of user attention can also be analyzed by classifying eye tracking data, which could help us understand the relationship between the embodiment and user attention. The integration of subject self-report, implicit measure, and physiological detection offers new possibilities for exploring the influences of virtual hand designs on users’ embodiment in VEs.

This paper aims to investigate the sense of embodiment based on its three subcomponents: body ownership, agency, and self-location with various virtual hand designs. In view of insufficient evidence about the impact of hand design on embodiment in the literature, this study combines subjective ratings, implicit measures derived from psychology, and an objective method of eye tracking for systematic investigation. This study brings new insights into users’ embodied experience in VR and offers important evidence about embodiment including its three subcomponents and the link with user attention under different hand designs. Furthermore, this study provides practical design recommendations for virtual hand design in future VR applications.

Related work

Virtual hand design

The VR environment can deceive the user's brain, and the sense of embodiment can make users believe that the virtual bodies are their own bodies (Wolf et al., 2021). Virtual avatars or hands can be used to create a familiar connection between users’ physical bodies and the virtual world. Participants can interact with the virtual world by controlling the virtual hands (e.g., grabbing or moving) in first-person perspective. The users’ upper limbs interact more with the virtual world through motion tracking techniques (e.g., Leap Motion hand tracking or handheld controllers) (Argelaguet et al., 2016; Ogawa et al., 2019). In addition, the appearance of virtual hands can be created with various realism levels by the shape of the hand and the rendering style. It has been established that participants would react differently according to different hand designs, and the representation of virtual hands could vary from highly abstract to highly realistic (Ogawa et al., 2019; Zhang et al., 2020, 2022a). Meanwhile, it is worth noting that there is a phenomenon defined as the Uncanny Valley, referring to people's attitudes changing from positive to negative when they watch a robot or a virtual avatar or hand that looks like a natural person (Hepperle et al., 2021).

Evidence from studies revealed that various virtual hand designs could influence users’ embodiment, and limited research evaluated the embodiment by the three subcomponents (body ownership, agency, and self-location) simultaneously (Ogawa et al., 2019; Heinrich et al., 2020). A previous article indicated that hand realism (realistic, iconic, and abstract hands) affected perceived object sizes as the size of the virtual hand changed, and the result was the more realistic hand with a stronger sense of embodiment (including three subcomponents) (Ogawa et al., 2019). A recent study investigated that four types of realism of non-mirrored and mirrored hand representations (personalized, realistic, robot, and wood hands) altered perceived embodiment (Heinrich et al., 2020). In this experimental task, a higher sense of agency was found with the realistic hand, and the author argued the lack of importance of self-location toward embodiment. These inconsistent conclusions of three subcomponents were assessed by subjective rates; thus, it is necessary to utilize multiple measurement methods to help understand the influence of hand designs on the embodiment in this study.

Sense of embodiment

Embodiment as a critical perception mechanism in VEs refers to the feeling that participants’ own virtual bodies (e.g., virtual avatars) or body parts (e.g., virtual hands or feet) as their own body (Kilteni et al., 2012; Toet et al., 2020). When a virtual body or hand generate congruent information with the real one (including visual, tactile, motion, and proprioceptive information), participants can feel the illusory perception of the embodiment (Ehrsson, 2012). The sense of embodiment has been defined as the sense of owning, controlling, and being inside a virtual body, and these three statuses correspond to the sense of body ownership, agency, and self-location, respectively (Kilteni et al., 2012).

The sense of body ownership is referred to as users’ self-attribution of a virtual body. The famous rubber hand illusion experiments demonstrate that people can create the illusion that – “the fake rubber hand is their own hand” through multi-sensory interaction (Botvinick and Cohen, 1998). Researchers have utilized advanced VR equipment to replicate and explore the users’ responses to rubber hand experiments, known as the virtual hand illusion experiment (Perez-Marcos et al., 2009). It is worth noting that the coherent and synchronized visual feedback of the virtual arm movement is sufficient to create the ownership illusion of the arm (Kokkinara and Slater, 2014). The illusion experience of body ownership can be evaluated by subjective and objective methods. The most common subjective method is self-report (questionnaire). Moreover, eye tracking as one objective physiological method can be used to explore the sense of body ownership in the literature (Krogmeier and Mousas, 2020, 2021). It has been shown that more body ownership could be reported from participants when they looked at a virtual avatar sooner.

Agency refers to the virtual body's experience of action and control in active movements. When the users’ motion is mapped to the virtual body in real-time or near real-time, the sense of agency can be generated. Research has shown that the synchronicity of vision-related movements affects the development of the agency (Wang et al., 2021). There are subjective and objective methods for measuring the sense of agency. Self-report is a subjective way to evaluate the sense of agency, which is the only method adopted in most works. Beyond self-report, intentional binding as one implicit method of the agency has been used in several interaction tasks recently (Suzuki et al., 2019; Nataraj et al., 2020a; Van den Bussche et al., 2020; Nataraj and Sanford, 2021). The standard basis of the intentional binding paradigm is the estimation of the time interval between an action (e.g., a button press) and a sensory result (e.g., sound feedback) (Suzuki et al., 2019; Nataraj et al., 2020a, 2020b; Van den Bussche et al., 2020; Nataraj and Sanford, 2021). It is an implicit way to evaluate the agency by measuring the time perception of the action effect, and participants are asked to estimate the time interval between action and consequence. In addition, based on VR-based eye tracking technology, one study has indicated the potential link between self-reported agency and eye tracking data under three virtual hands (Zhang et al., 2022a).

Self-location refers to the subjective feeling of “where I am in space”, and a detailed explanation is a spatial experience of being inside a virtual body (Kilteni et al., 2012; Toet et al., 2020). Visual perspective is essential for participants to experience the self-location inside a virtual avatar, especially in the first-person perspective (Gorisse et al., 2017; Huang et al., 2017). The self-report is a commonly subjective method for assessing self-location, but the objective method seems limited based on the literature. One objective solution was derived from the field of psychology, and it was proposed that proprioceptive measurement addressed hand position in the case of non-VR (Ingram et al., 2019). The location value that indicated how far the proprioceptive self-location was perceived toward visual self-location from the actual hand position. Participants were required to use their left hand to identify the landmarks of their right hand on a board with position references, and the difference between perceived and actual hand positions was recorded to assess perceived self-location (Ingram et al., 2019). It is an inspiration to evaluate the self-location by this proprioceptive position method. In addition, a recent work reported a weak negative correlation between the data of eye tracking and self-location, and it found less self-location when participants watched their virtual avatar for more time (Krogmeier and Mousas, 2020).

It is worth understanding user attention in the virtual world in the current study, which is a valuable way to help us explain users’ embodied experience (including the sense of embodiment) more comprehensively. For example, exploring whether users pay more attention to the virtual avatar representations or others during the interaction with the virtual world. Limited articles have speculated about the relationship between the sense of embodiment and user attention in VR, but it still lacks supportive evidence in the literature (Tieri et al., 2017; Cui and Mousas, 2021). Users’ attention can be analyzed through the recorded eye behaviors data of users in VEs through eye tracking (Batliner et al., 2020; Li et al., 2021; Mousas et al., 2021). Large amounts of participants’ eye tracking data will be collected in this study and then classify them using the K-means clustering that group information in an unsupervised way. K-means algorithm is a widely used source of data mining, and its classification effectively reduces large amounts of data into smaller sections in artificial intelligence (Huang et al., 2018). The previous work captured user attention by fixation spans, and clustering fixation spans in groups by the K-means algorithm reflected different degrees of user attention (Dohan and Mu, 2019). However, when it comes to studying the sense of embodiment by user attention, very little is known. Clustering these eye tracking data provides new ideas for this article to help understand and explore the sense of embodiment by analyzing user attention in VR.

Methodology

An experiment was designed to systematically investigate the sense of embodiment with virtual hand designs through self-report, implicit, and objective measurements. The experiment involves five virtual hand designs (with realism levels from low to high) to induce the participants’ sense of embodiment when performing interactive tasks in VEs. There was a self-report for assessing the embodiment and implicit methods for evaluating agency and self-location. Additionally, the eye tracking data was applied to explore the sense of embodiment.

Apparatus

The physical experiment equipment includes the VR system (HTC Vive Pro Eye), a Windows-based computer (Intel(R) Core (TM) i7-9800X CPU), and the hand tracking module (Leap Motion). The virtual experimental environment was designed and run by a game platform named Unity 3D (2019.4.16). Applying Leap Motion instead of traditional VR controllers changed how participants interacted with VEs. The front of the VR head-mounted display (HMD) installed the hand tracking module that can capture the participants’ hands and fingers (see Fig. 2). When participants wore the HMD, they could see a virtual lab room, including a wooden table and several buttons on the table. If they started the experiment, their right hand was tracked, and the virtual scene presented one of the virtual hand models.

Participants

Sixty-two participants (34 males and 28 females; 22.74 ± 3.90 (SD) years old) as volunteers were involved in this study, and they had normal or corrected-to-normal visions. Most participants were young students from university, and only three were above 30 years old. Moreover, only two participants were left-handed, and they reported that completing the tasks using their right hand did not affect their experimental experience. Forty-eight participants have experienced VR before, and 22 participants have used Leap motion.

The university ethics committee approved this study, and all participants signed the informed consent form before the experiment. In addition, considering the accuracy data of capturing eye tracking, they were asked to complete the eye calibration first when they wore the HMD. All 62 participants performed the Whack-a-Mole task. In order to avoid possible participants’ fatigue caused by the long experiment time, they were randomly assigned to one of the following parts (intentional binding or proprioceptive measurement) in the second step after completing the first step. In the intentional binding part, there were 30 participants (20.25 ± 2.70 (SD) years old), including 13 males and 17 females, and 32 participants (21 males and 11 females; 25.47 ± 3.96 (SD)) involved in the proprioceptive measurement part.

Virtual hand designs

Based on the virtual hands applied in the literature and VR applications, the entire region of virtual hand realism was divided into five categories, from low to high realism. One typical case was selected from each category as the research object in this study. Five types of hand designs (block hand, minimal hand, capsule hand, low-poly hand, and human hand) that vary in five levels of realism were designed (see Fig. 1). According to controlling the experiment's unique variable, all participants would use the same five virtual hand models with the same size to perform the whole experiment.

• Block hand (Low realism): The shape of this virtual hand looks like a simplification of the palm part of the actual human hand. The palm model is represented by a rectangle with a light-yellow surface, but without five fingers and wrists.

• Minimal hand (Low to medium realism): The minimal hand with the light-yellow surface is composed of a circle and some small ellipses blocks, representing palm and knuckles, respectively. These knuckles are separate and look like the hand skeleton.

• Capsule hand (Medium realism): The capsule hand is composed of a combination of several spheres and cylinders. These red spheres connect the framework of the virtual hand, and these white cylinders represent the hand's knuckles.

• Low-poly hand (Medium to high realism): The number of fingers and hand joints of the low-poly hand is similar to the real hand, but the low-poly hand has an angular appearance. The surface of the hand is rendered with a smooth light-yellow material.

• Human hand (High realism): This hand model, including five fingers and fingernails, is highly similar to real people's hands, and the skin texture of this virtual hand is neutral. There are fine hairs and blood vessels on the hand skin.

Fig. 1. Five hand designs in five types of realism levels (1: block hand; 2: minimal hand; 3: capsule hand; 4: low-poly hand; 5: human hand).

Procedures

In order to collect objective and subjective data on the embodiment, the experiment was set to allow participants to interact actively with their virtual hands in the virtual scene. Considering all types of virtual hands could perform interactive experiments smoothly to induce user perceptions during the continuous time. The experiment in this study consists of two sessions: a pre-training session and an experimental session (see Fig. 2).

Fig. 2. Experimental procedures (Top: flow diagram of the experiment; Bottom: experimental scene of real and virtual environments).

The essential operation of experiment tasks requires participants to perform the Whack-a-Mole task that the button with a 10-cm diameter represents the mole hole. One of these buttons changed color randomly to guide the participant to press the color-changing button. The following random button started to change color after they successfully pressed it, and the duration of the entire process was 30 s as one task. According to experimental requirements, participants needed to press virtual buttons with a virtual hand during the specified 30 s for each condition, and each of them was asked to use these five virtual hands randomly for five tasks.

Pre-training session

Pre-training: This session aims to allow participants to get familiar with the virtual scene and interaction in the task with an oblate hand, which can take approximately 2 min. To distinguish the oblate hand (in the pre-training session) from these five virtual hands (in the following experiment), it is designed as a simple gray round pie without a rendering surface (see Fig. 2a). Only the three buttons in the middle row can be used in this session. At the beginning of this session, participants were required to locate their virtual hand in the initial position (the central button with blue color). Participants pressed the central button first to initiate the task; then another button would turn red to guide the participant to press it. The participant pressed the three buttons in the specified order (first the central button, then the left button, and last the right button), and they repeated this operation twice to familiarize the virtual scene and interaction in this session.

Experimental session

Experimental task: Participants were asked to perform the experimental task using five virtual hands randomly, and five test blocks correspond to different hand designs. In the first step, when participants were ready, each virtual hand was used to complete the Whack-a-Mole task based on accuracy. Then, participants were asked to accomplish the self-report to evaluate users’ embodiment. The second step was the survey of the sense of agency and self-location by implicit methods (intentional binding and proprioceptive measurement, respectively).

First step: Whack-a-Mole task and self-reported embodiment

When participants started to test in random order, they first observed the virtual hand and moved hand fingers to familiarize themselves with it, which took approximately 20 s. Then, as in the pre-training session, this task required participants to locate their virtual hand in the initial position (the central button with blue color) at the beginning of the task. After participants first pressed the central button, one of the six surrounding buttons would randomly change its color to red. Within the specified 30 s of one task, participants needed to press the red button and follow the color stimulus to press the corresponding button one after another (see Fig. 2b). All experimental virtual scenarios of each task were identical, except for using different hand models. The eye tracking data from participants can be collected during the 30 s task, including the total duration of fixations and the number of fixations for virtual hands. The order of test blocks was randomly set with 2–5 min break between every two blocks. The first step of the experimental task session (Whack-a-Mole task and self-report part) with five virtual hands took approximately 30 min to complete. In addition, participants were reminded to use their palms rather than fingers to perform the task, because the research paid more attention to participants’ experience of hand movement than hand speed in this study.

Eye tracking data recording in Whack-a-Mole task: In this experiment, eye tracking data in VR is collected using HTC Vive Pro Eye that combines VR and eye tracking technology. This device provides an opportunity for researchers to perform eye gaze behavior analysis and gaze position prediction under a natural virtual viewing condition with experimental control. Through the eye tracking software algorithm (SRanipal SDK provided by HTC Corporation) implemented in the virtual scene, the eye movement raw data from participants can be collected. The data of participants’ eye tracking during the process as objective information was collected for later analysis of user perceptions. For instance, when participants started the tasks, the number of fixations about what they viewed as virtual hands or other places during the specified time was recorded in each task. These eye tracking data include eye behaviors under five tasks, such as fixation count, fixation length, and time to fixation.

Questionnaire data recording in the self-reported embodiment: This part took place after each Whack-a-Mole task with five hands, and the aim was to collect the participants’ perception data through the embodiment questionnaire. Participants needed to assess the sense of embodiment by the standardized questionnaire (Kilteni et al., 2012; Gonzalez-Franco and Peck, 2018). There are ten questions about the sense of embodiment (Q1–Q3: body ownership; Q4–Q7: agency; Q8–Q10: self-location), which can be seen in Table 1. They were required to tell the experimenter the results of each question orally. The application of the virtual questionnaire in this study allows participants to answer questions directly in VEs, which can avoid the influence of frequent behavior about putting on a VR headset on experimental data (Schwind et al., 2017). The virtual questionnaire with a 7-point Likert scale would pop up in front of participants, and they needed to finish these 10 questions about the embodiment (see Table 1).

Table 1. Questionnaires results

BH, block hand; MH, minimal hand; CH, capsule hand; LH, low-poly hand; HH, human hand.

Second step: intentional binding and proprioceptive measurement

The purposes of the intentional binding and proprioceptive measurement parts are to assess the sense of agency and self-location (two subcomponents of the embodiment) with different hand designs by implicit methods from psychology, respectively.

Intentional binding: The implicit method of measuring the sense of agency under five virtual hands was shown in this part. The participants’ perceptions of the time interval between the press and the audio feedback with five virtual hands were collected. In order to reduce participants’ habitual actions on the first step and prevent participants from confusing the sound and feedback in one trial, the number of virtual buttons in this virtual scene has been reduced to three (see Fig. 2b₁).

One trial in this part started with participants pressing the middle blue button (the initial position), and then pressing the left button when its color changed to red. The audio feedback was generated with a specific time interval after the left button pressing, and they needed to verbally tell the experimenter the estimated time interval to end one trial. Then, the subsequent trial followed the same procedure, but participants were required to press the right button. There were four pre-training trials and six training trials for each hand design, and participants would be given 2 min break after finishing one test block to reduce fatigue. There were five test blocks in total in this part (which took approximately 15 min) corresponding to the five hand designs, and the order of hands was randomly set.

The procedure of this part was composed of two steps: four pre-training trials and six training trials. Pre-training trials: The first step was to allow participants to get familiar with 0.1 s time interval under four trials (pressing two left and two right buttons), which served as a reference for the interval estimation in the next step. Training trials: After the first steps, the second was to ask participants to estimate the randomly implemented interval between button pressings and beep feedback. Participants were informed in advance that the interval was randomly set, ranging from 0.1 to 1 s in each trial in this part. However, the actual interval was set to random 0.1, 0.3, 0.5, 0.7, or 0.9 s.

Proprioceptive measurement: This part implicitly measured the sense of self-location with five virtual hands. In order to reduce the visual impact of other redundant buttons, the virtual scene reserved one button with a prompt pattern to hint hand position on the table in VR (see Fig. 2b₂). Instead of pressing virtual buttons, participants were asked to place their hands in the fixed position in this part. A transparent acrylic board with coordinates was placed in front of participants to assist in recording the position of their right hand (see Fig. 3). Based on the thickness of each participant's palm, the height of the board was fine-tuned to roughly 1 cm higher from their right hand.

Fig. 3. Participants pointed the hand positions of the middle fingertip (a) and wrist (b) on the transparent acrylic board with coordinates.

Participants placed the right hand on the fixed hand pattern in VEs, and they needed to use their left hand to identify the perceptive location of the right virtual hand (by pointing the positions of the middle fingertip and wrist) on the acrylic board (see Fig. 3a,b respectively), which process was repeated twice under one virtual hand. The experimenter recorded the actual and perceived positions of the virtual hand through the coordinates on the board, which would be used later to implicitly analyze the users’ sense of self-location. The five virtual hands appeared randomly, and the whole part took about 15 min.

Measurement and analysis

Self-report

The questionnaires, including body ownership, agency, and self-location, were used for evaluating the embodiment through its three subcomponents, and the detailed questions can be seen in Table 1. The subjects were asked to rate the answers between −3 (totally disagree) to 3 (totally agree) using a 7-Likert scale. The method (adding and subtracting the data from the Likert scale) used in this study is an accepted and common way to calculate the self-reported embodiment (including its three subcomponents and total embodiment) in the literature (Gonzalez-Franco and Peck, 2018; Roth and Latoschik, 2020; Peck and Gonzalez-Franco, 2021). Following the arithmetic addition and subtraction of question scores, the values of three subcomponents (body ownership, agency, and self-location) and total embodiment were analyzed by this classical method in this study.

• • Body ownership = Q1−Q2−Q3

• • Agency = Q4 + Q5−Q6−Q7

• • Self-location = Q8−Q9−Q10

• • Total Embodiment = ((Body ownership/3) + (Agency/4) + (Self-location/3))/3

Intentional binding

The implicit agency method was to compare the differences between the actual and estimated time interval (Δt) between action and consequence via intentional binding, which referred that the sense of agency was defined as Δt (actual) − Δt (estimated). Participants judged the time interval between the action (pressing the virtual buttons) and the consequence (delayed sound beep between 100 and 1000 ms randomly). The values of Δt (actual) − Δt (estimated) with different virtual hand designs were analyzed. The estimated time interval that was relatively compressed indicated a positive measurement of the agency. The more positive value is interpreted as greater agency due to underestimating actual time intervals.

Proprioceptive measurement

The participants’ actual and perceived hand positions were recorded in the proprioceptive measurement part, which was an implicit measurement of self-location. The hand position is defined by the coordinates of two points (the participant's middle fingertip and wrist illustrated in Fig. 3). The position data of the actual hand and perceived hand from one participant, as an example, are shown in the left image of Figure 4, with A representing the point of middle fingertip and B representing the point of the wrist. Participants were asked to identify each hand position of the middle fingertip and wrist twice, and the average positions as final coordinates were calculated in this study. Based on the final coordinates of actual and perceived hand positions, the right image of Figure 4 shows an example of proprioceptive measurement data of block hand from all participants.

Fig. 4. Left: Explanation about the positions and actual hand (orange color) and perceived hand (blue color); Right: An example of proprioceptive measurement data of block hand from all participants.

The related formulas for calculating the hand position location are shown below:

• • Coordinate of actual hand's middle fingertip: A (a ₁, a ₂)

• • Coordinate of actual hand's wrist: B (b ₁, b ₂)

• • Coordinate of perceived hand's middle fingertip: A′ (a′ ₁, a′ ₂)

• • Coordinate of perceived hand's wrist: B′ (b′ ₁, b′ ₂)

• • Length of actual hand: $\vert {{AB}} \vert = \sqrt {{( {{a}_1-{b}_1} ) }^2 + {( {{a}_2-{b}_2} ) }^2}$

• • Length of perceived hand: $\vert {{{A}^{\prime}{B}^{\prime}}} \vert = \sqrt {{( {{{a}^{\prime}}_1-{{b}^{\prime}}_1} ) }^2 + {( {{{a}^{\prime}}_2-{{b}^{\prime}}_2} ) }^2}$

• • Error between actual and perceived hand position (middle fingertip): $\vert {{A{A}^{\prime}}} \vert = \sqrt {{( {{a}_1-{{a}^{\prime}}_1} ) }^2 + {( {{a}_2-{{a}^{\prime}}_2} ) }^2}$

• • Error between actual and perceived hand position (wrist):$\vert {B{B}^{\prime}} \vert = \sqrt {{( {{b}_1-{{b}^{\prime}}_1} ) }^2 + {( {{b}_2-{{b}^{\prime}}_2} ) }^2}$

• • Error between actual and perceived hand length: |AB| − |A′B′|

Eye tracking

Based on previous studies (Dohan and Mu, 2019; Krogmeier and Mousas, 2020, 2021), the data on eye gaze behaviors could be collected and analyzed by fixation count, fixation length, and time to fixation in this study. The fixation count shows the entire number of eye fixations recorded on these virtual hands, the fixation length refers to the amount of time each fixation lasted, and the period it takes to first fixation at these virtual hands is explained as time to fixation.

In addition, according to this basic eye tracking data (such as fixation count, fixation length, and time to fixation) of virtual hands collected in this study, the users’ attention on these five virtual hands can be analyzed by the clustering algorithm (Dohan and Mu, 2019). The clustering algorithm named the K-means, as one of the more commonly used classification techniques, was used in this study, and it is a valuable technique for finding conceptually meaningful groups with a similar state. The algorithm has the number of clusters as research parameters that creates center reference points in an unsupervised way.

Results

All results obtained from the study were summarized in this section, and the analyses were performed using Excel and IBM SPSS statistics analysis software. The normality assumption of the subjective ratings was evaluated using the Kolmogorov–Smirnov test. The Kruskal–Wallis test, as a non-parametric test, compared samples from more groups of independent observations. Bonferroni's post-hoc test was conducted to check the significance for multiple comparisons, and the significance level α is at 0.05.

Self-report

The questionnaires including body ownership (Q1–Q3), agency (Q4–Q7), and self-location (Q8–Q10) were used in the study for assessing the embodiment through its three subcomponents (see Table 1). According to the results of the Kolmogorov–Smirnov test, the non-parametric test for the remaining analysis was used. The Kruskal–Wallis test has statistically evaluated the data from each question. The results of each subcomponent of the embodiment are shown in Table 1, with M standing for mean and SD standing for standard deviation, and M and SD are used to assist in understanding the final results after statistical calculations of significant differences.

Body ownership

The significant effect with P < 0.05 for body ownership-related questions (Q1–Q3) was found in this study (H = 24.534, P < 0.001). The minimal (M = 4.45; SD = 3.99) and capsule hands (M = 4.98; SD = 3.56) showed higher mean values than other virtual hands. However, the human hand (M = 3.65; SD = 4.40) reported a lower mean value of body ownership. By applying post-hoc tests, significant differences between the block hand and other hands (P < 0.05) were found, and the block hand (M = 2.00; SD = 3.55) induced the lowest sense of body ownership.

Agency

Based on the results of agency questions (Q4–Q7), this work found a significant effect of hand designs on the sense of agency (H = 33.181, P < 0.001). These four virtual hands proved similar scores denoting a higher mean value of agency than the block hand. In addition, the minimal (M = 8.20; SD = 2.79) and capsule hands (M = 8.08; SD = 2.99) showed the highest mean value of agency. The block hand (M = 4.62; SD = 4.14) was rated significantly lower than the minimal, capsule, low-poly, and human hand by pairwise comparisons.

Self-location

There was a significant result (H = 12.268, P = 0.015) from self-location questions (Q8–Q10). The minimal hand (M = 3.17; SD = 2.69) and capsule hand (M = 3.10; SD = 2.78) revealed a higher mean value about self-location. In addition, the human hand (M = 2.75; SD = 2.87) showed a lower mean value over the minimal hand and capsule hand. Pairwise comparisons proved significant differences between the block and the other two (minimal and capsule) hands. The block hand (M = 1.40; SD = 3.07) was rated significantly lowest.

Total embodiment

The total embodiment showed a significant result (H = 28.998, P < 0.001) in this research. Except the block hand (M = 0.76; SD = 0.90), minimal hand (M = 1.53; SD = 0.81), capsule hand (M = 1.57; SD = 0.80), low-poly hand (M = 1.43; SD = 0.84), and human hand (M = 1.33; SD = 0.96) revealed similar mean values. Pairwise comparisons of embodiment showed significant differences (P < 0.05), and the block hand indicated the lowest embodiment scores among these five hand designs.

Intentional binding

This data between the actual and estimated time interval was not following a normal distribution by the Kolmogorov–Smirnov test. However, for the results of the estimated time interval, no significant differences (H = 1.392, P = 0.846) were found under these five virtual hands by the Kruskal–Wallis test. The block hand (M = 65.00; SD = 245.29) revealed the lowest mean value among the five hands. The mean values of other hands were reported in the minimal hand (M = 86.11; SD = 193.40), capsule hand (M = 70.56; SD = 227.62), low-poly hand (M = 88.33; SD = 209.09), and human hand (M = 95.00; SD = 223.05).

Proprioceptive measurement

The Kolmogorov–Smirnov test (P < 0.05) determined that all position data were not normally distributed, then the non-parametric statistical test was required. The errors between the actual and perceived hand positions (including the middle fingertip and wrist) were obtained, and the hand lengths of participants’ actual and perceived were calculated. The Kruskal–Wallis test also analyzed differences in participants’ proprioceptive perceptions among five different virtual hands. The final coordinates for these two positions (middle fingertip and wrist) with five hands were recorded in Figure 4.

1. 1) Error between the actual and perceived hand positions (middle fingertip and wrist): According to calculating the relative value between the coordinates of the actual and perceived hand positions (middle fingertip and wrist), the error between them could be obtained.

   Middle fingertip: The average errors between the actual and perceived hand middle fingertip were shown in the block hand (4.55 cm), minimal hand (4.08 cm), capsule hand (4.26 cm), low-poly hand (4.00 cm), and human hand (4.16 cm). The Kruskal–Wallis test showed no significant difference in the position of the middle fingertip (H = 2.398, P = 0.663) under five virtual hands.

   Wrist: The average errors of the wrist between the actual and perceived hand were listed below: block hand (3.39 cm), minimal hand (3.74 cm), capsule hand (3.64 cm), low-poly hand (4.36 cm), and human hand (4.08 cm). There was no significant difference in the wrist position among these five hands by the Kruskal–Wallis test (H = 5.304, P = 0.257).

2. 2) Error between the actual and perceived hand length: There are relative errors between the actual and perceived hand length among these five virtual hands, such as the block hand (3.03 cm), minimal hand (2.48 cm), capsule hand (–2.83 cm), low-poly hand (–1.88 cm), and human hand (–2.57 cm). There is a significant difference in relative error between the actual and perceived hand length (H = 75.541, P < 0.001) with different hands observed by the Kruskal–Wallis test. Based on the relative values of real and perceived hand lengths, it was noticed that participants underestimated the actual length of their hands when they perceived capsule hand, low-poly hand, and human hand.

Eye tracking

To determine whether various virtual hand designs affected eye behaviors (fixation count, fixation length, and time to fixation) in VR, the Kruskal–Wallis test was used to explore eye tracking data toward these five hands, and the post-hoc test was conducted to check the significance for pairwise comparisons.

1. 1) Eye behaviors data: There were significant differences in fixation count (H = 56.866, P < 0.001), and fixation length (H = 88.317, P < 0.001) with five virtual hands shown in Figure 5, while no significant difference in time to fixation (H = 1.104, P = 0.894). Post-hoc test using a Bonferroni correction was used to compare all pairs of groups.

   Fixation count: Pairwise comparisons about fixation count showed that participants who experienced the Whack-a-Mole task looked significantly less times at minimal hand (M = 33.04; SD = 14.07) than other hands (all P < 0.05). The human hand (M = 51.23; SD = 12.63; P < 0.001) and capsule hand (M = 50.64; SD = 11.67; P < 0.001) with similar and higher mean values were found the significant difference between the low-poly hand (M = 42.73; SD = 12.44). Meanwhile, the block hand (M = 43.13; SD = 12.75) was rated significantly lower than the human hand and capsule hand.

   In addition, the number of times participants looked at buttons and virtual environments were also recorded in this study. However, no significant differences in participants’ viewing of virtual buttons (H = 3.779, P = 0.437) and environments (H = 2.342, P = 0.673) were found among these five virtual hands.

   Fixation length: Based on the post-hoc comparisons, the result of fixation length indicated that participants looked significantly less at the minimal hand (M = 2.85; SD = 1.71) than at other hands. The highest mean value of the human hand (M = 7.69; SD = 3.48) showed a significant difference between the block hand (M = 4.80; SD = 2.23; P < 0.001) and low-poly hand (M = 5.39; SD = 2.51; P = 0.014). Moreover, the capsule (M = 6.93; SD = 3.00) hand was rated significantly higher than the block hand and low-poly hand.

   A significant difference was found in the amount of time participants spent watching the buttons (H = 6.859, P < 0.001), while not the virtual environment (H = 4.249, P = 0.373). Participants looked significantly more at the minimal hand (M = 15.29; SD = 4.75) than other hands by post-hoc comparisons.

   Time to fixation: There were no significant effects in the result of time to fixation, such as virtual hands, buttons, and environments. The low-poly hand (M = 15.01; SD = 4.10) revealed the lowest mean value and standard deviation than other four hands. The block hand (M = 15.70; SD = 5.53) and human hand (M = 15.37; SD = 5.29) have similar standard deviations. The capsule hand (M = 15.72; SD = 6.79) and minimal hand (M = 16.18; SD = 7.42) with similar mean values have higher standard deviations among these hands.

2. 2) Correlations between eye behaviors data and self-reported embodiment: Due to the non-parametric data, the Spearman test was used to evaluate the relationship between self-report and eye tracking data for each condition with five hands (see Table 2). The Spearman correlation of different values was considered negative at –1 to 0 and positive at 0 to 1, and the coefficient was defined as: weak (0≤|r _s|<0.3), moderate (0.3≤|r _s|<0.6), strong (0.6≤|r _s|<1), and perfect 1. However, there was no relationship between the eye fixation data of virtual hands and the self-reported body ownership, agency, and self-location (all P > 0.05).

3. 3) Clustering eye behaviors data: According to the significant results about fixation count and fixation length, participants’ gaze data (fixation count and length) were separated into three degrees based on the distribution of the numerical values, such as low attention, middle attention, and high attention. The levels of gaze in three clusters of participants’ attention with five virtual hands are plotted in Figure 6, and the figure indicates different levels of attention across five hands tasks from all participants.

Fig. 5. The fixation count (left) and fixation length (right) for five virtual hand designs.

Fig. 6. Clustering results using K-means algorithm under five virtual hands.

Table 2. Correlations between eye fixations and self-report data

After removing outliers, 56 participants’ fixation count and fixation length data were collected. For each hand, all participants were grouped into three clusters including low attention (red cluster), middle attention (yellow cluster), and high attention (blue cluster), which can be seen in Table 3 and Figure 6. There were three center reference points of these three clusters in each hand condition, and the center reference points of participants’ eye tracking data across five hands were different. According to different locations of reference points, the distribution of three degrees of attention was also diverse in these five conditions.

Table 3. Distribution of participants with three degrees of attention

The result revealed that close to 30 participants used the block hand with middle attention, and about 10 participants with low attention. More than 40 participants showed low (24 participants) and middle attention (21 participants) to the minimal hand. Only 6 participants paid low attention to the capsule hand, and over 20 participants paid middle and high attention, respectively. For similar attention results of low-poly hand and human hand, just more than 20 participants owned middle attention. Most participants paid middle and high attention to these three hands (such as the block hand, low-poly hand, and human hand), but most participants used the minimal hand with low and middle attention. Less participants with low attention and most participants with middle attention were shown using the capsule hand.

Discussion

Theoretical implication

Self-reported embodiment

The questionnaire results of this study have proved that all these five virtual hand designs (block, minimal, capsule, low-poly, and human hand) in five realism levels could elicit the sense of embodiment (body ownership, agency, and self-location) in users. The findings also indicated that the low-realism hand design represented by a primitive geometry without the shape of fingers, the block hand in this study, induced the lowest sense of embodiment compared with other hand designs with higher realism, which corroborated the previous findings in the literature (Ogawa et al., 2019; Heinrich et al., 2020).

Previous studies discussed that strong body ownership over a virtual avatar in VR played a crucial role when attributing its actions to oneself (Freeman and Maloney, 2021; Wolf et al., 2021). In this study, participants’ ratings from questionnaire results revealed significantly lower body ownership with the block hand. Earlier articles that explored the influence of different appearances on body ownership also reported similar conclusions (Argelaguet et al., 2016; Zhang et al., 2020). The low-realism hand rated less body ownership than hands with other realism levels. A possible explanation is that it is difficult for users to believe that less realistic representations can replace their actual hands. For example, the block hand lacking fingers did not change significantly when participants’ fingers moved in VEs. There was no significant difference between the human hand and the other three (minimal, capsule, and low-poly) hands, which was contrary to the high realism levels of hand expected in previous research to induce a stronger sense of body ownership (Schwind et al., 2017; Ogawa et al., 2019). Due to the Uncanny Valley effect, the previous article explained that several people could feel scared when they used the realistic hand (Schwind et al., 2017; Hepperle et al., 2021). Based on participants’ comments after the experiment, some reported higher expectations of the high-realism hand at first, but they finally felt more disappointed with it because of the unnatural experience during the experiment.

Interestingly, there were controversial results in the literature about whether the more realistic hand brought a higher sense of agency (Ogawa et al., 2019; Heinrich et al., 2020; Zhang et al., 2020). The reasons for these contradictory results may be related to the different interaction methods in VR, such as using traditional VR controllers (Zhang et al., 2020) or hand tracking models (Ogawa et al., 2019; Heinrich et al., 2020). This study showed significant differences in the agency from self-report results with Leap motion (hand tracking model). The minimal, capsule, low-poly, and human hands were significantly rated higher than the block hand. The result was in keeping with a previous observational study, which found that embodiment scores in the low-realism hand were significantly lower than in other realism conditions (Ogawa et al., 2019). This result may be explained by the fact that virtual hand designs that own fingers can enhance peoples’ controlling experience by providing real-time visual feedback of fine movements of the hand when the hand tracking device is applied. It is concluded that the agency can be generated when participants’ hands move synchronized with virtual hands, and these hand designs can influence the sense of agency.

One focus of this research is exploring the self-location of embodiment, because there are not clear how virtual hand representations affect self-location in VEs (Ogawa et al., 2019; Heinrich et al., 2020). A work reported that hand appearances affect self-body perception in the embodiment questionnaire: low-realism hand with a lower embodiment score including self-location (Ogawa et al., 2019). In addition, an author argued that the sense of self-location contributed minimally toward embodiment than the other two subcomponents (Heinrich et al., 2020). In this study, the self-location value indicated a significant effect. Consistent with one literature (Ogawa et al., 2019), this research found that the block hand was rated significantly lowest self-location over the minimal and capsule hand from questionnaire results. The possible explanation for this is that the low-realism hand without fingers could affect the feeling of hand position, and several participants indicated that they judged their hand position by the position of the fingertip of the middle finger instead of the wrist when they felt the position of the hand.

Implicit embodiment

This study did not significantly differ in intentional binding under five virtual hand designs, even though there was a significant difference in self-reported results. Participants felt the difference in time interval when they completed the experiment, but the difference in estimating the time interval with different virtual hands was not noticeable. In fact, the relationship between the self-reported and implicit (intentional binding) method of the agency was debated in the literature (Dewey and Knoblich, 2014; Bergström et al., 2022). Research has shown that there are dissociations between self-reported and implicit agency judgments (Dewey and Knoblich, 2014), while the correlation may be moderated by the performance of a recent study (Bergström et al., 2022). The finding formed the study also accords with the earlier observation (Dewey and Knoblich, 2014), which showed no relationship between the correlation of these two methods. It is possible that the information sources of the process when participants answer the questionnaire and perceive time intervals are different. A study reported that the self-attributions of self-reported and intentional binding relied on overlapping cues (Dewey and Knoblich, 2014). As a self-reported measure, participants needed to report whether they had produced the sense of agency or not after the experimental task. They needed to rely on their own recollections to make judgments, while implicit measures focused on participants’ current estimates of the time interval.

The implicit result of self-location showed no significant effect in error between the positions of actual and perceived hands under five virtual hands, although self-reported results had. Based on a significant result about relative errors in real and perceived hand lengths, participants underestimated the actual length of their hands with capsule hand, low-poly hand, and human hand. Several participants reported after the proprioceptive measurement that when they saw the block hand lacking fingers, they relied more on their perceptual information than virtual vision information to estimate hand positions, especially in the middle fingertip. In addition, according to the data of proprioceptive measurement, it had a deviation between the perception of actual and virtual hand position from participants, even though there was no significant difference among these five hands. Participants perceived the position of the middle fingertip to the right side as closer to their body than the actual position, which was a similar result in a non-VR study (Ingram et al., 2019). It is possible that the sense of self-location relies on a combination of additional cues, including informative vision and the perceived position of the virtual hand.

Eye tracking and embodiment

As mentioned in previous articles, eye tracking data is a new method to help explore the effect of virtual hands on the sense of embodiment and understand how participants pay attention to different virtual hands in VR. It has been noted that the utility and versatility of using biometric tools and measures in design fields, such as eye tracking data. There is no need to collect subjective data alone, and these biometric measurements can objectively investigate more content and increase meaning (Borgianni and Maccioni, 2020).

Previous literature has reported significant correlations between eye tracking data and three components of embodiment: body ownership and self-location (Krogmeier and Mousas, 2020, 2021), and agency (Zhang et al., 2022a). However, this work did not find a connection between self-reported embodiment and eye tracking data. A previous article suggested the potential importance of time to fixation in eliciting the sense of body ownership: participants would report more body ownership when they sooner looked at a virtual avatar during the virtual experience. As mentioned in the literature review, the potential relationship between the body ownership and eye tracking data: a weak negative relationship between body ownership and time to fixation (Krogmeier and Mousas, 2020). For example, it has been demonstrated that participants watched the virtual zombie avatar significantly sooner than the mannequin, and more body ownership was reported with the zombie (Krogmeier and Mousas, 2020). A previous study has investigated a weak negative correlation between self-reported self-location and self-fixation count (Krogmeier and Mousas, 2020). For instance, participants reported looking at their self-avatar more times with less self-location (Krogmeier and Mousas, 2020). The potential link between the agency and eye tracking data under three virtual hands was found in an article, and the less times participants watched one hand implied the less sense of agency (Zhang et al., 2022a). Nevertheless, this study's result was different to previous studies, and no significant effect of the relationship between self-reported embodiment (body ownership, agency, and self-location) and eye tracking data (fixation count, fixation length, and time to fixation) was found under these hand designs.

It has been reported that users’ attention might affect participants’ perception of virtual avatars, such as the sense of embodiment, although there is not enough evidence yet (Cui and Mousas, 2021). Its results revealed that on-site participants paid more attention to hand appearance than remote participants, which may be related to their high degrees of attention. In order to continue to explore the relationship between the embodiment and participants’ attention, eye tracking data was recorded, and cluster analysis by K-means was conducted to classify user attention under different virtual hands.

According to the results about eye behaviors, there were significant effects in fixation count and fixation length with five hand designs, while no time to fixation. Participants looked significantly less times and length at the minimal hand than other hands and looked more at virtual buttons and environments. It may be interpreted that the skeleton of the minimal hand is smaller than other hands. When participants could see virtual buttons under the virtual hand more clearly, they may pay more attention to the task or environment than the virtual hand itself. A study also reported that participants would focus on the task rather than the tool in VEs when they embodied a tool (Alzayat et al., 2019). Interestingly, although the questionnaire's conclusion of body ownership had a lower value for the human hand, most participants with the human hand showed middle and high attention in the clustering results. The possible explanation is that participants may pay attention to the virtual human hand when its interaction is not perfectly matched with their own hand. As mentioned in the previous article, participants may focus on performing the project rather than different hand designs if they interact more smoothly in VEs. Moreover, complex hand designs or users’ familiarity may affect participants’ attention to some extent, which could be considered.

Summarized embodiment

Significant differences in the sense of embodiment with various hand designs were found in questionnaire results, even not in implicit measures. It has a reasonable explanation that these two measurements rely on different informative sources from participants. In addition, several participants commented that the questionnaire seemed to have similar questions when they answered them, especially in the sense of body ownership and agency. Previous works also proposed that the inter-relations between the three subcomponents probably influenced the sense of embodiment (Kilteni et al., 2012; Fribourg et al., 2020), even though there is no definitive conclusion yet.

The questionnaire result on body ownership showed a significant difference, and there was no consistent result with previous literature about the correlation between body ownership and eye tracking data (Krogmeier and Mousas, 2020). It is possible that few participants with VR experience bring in the knowledge that the virtual hands are not their real hands in advance, which influence their perceptions of embodiment, especially in the sense of body ownership. Although this study did not compare the perceptions of participants with different prior VR experiences, it could be considered in future work. In addition, several participants had prior experience with the hand tracking model (Leap motion) and paid more attention to virtual hand movements with their own synchronizing rather than hand representations. Concerning eye tracking data, a previous study found that the sense of body ownership was related to how quickly participants viewed the virtual avatar. However, the speed at which they viewed the different virtual hands was similar in this study.

There was a significant effect in the self-reported agency, and this study had not found different binding for different time intervals with five hand designs. The general conclusions about self-reported agency in the literature were contradictory, but a significant result was obtained in this study. Compared with a previous study (Zhang et al., 2020), different interaction modes may be a factor affecting the agency. The implicit method may be a new way to assess agency, but it is also controversial in the research community (Bergström et al., 2022). The self-report and other measures of agency (implicit and objective) were not significantly correlated with each other in this study, and the finding still did not fully support the link between self-reported agency and the data of intentional binding and eye tracking.

It has been shown that a significant effect from questionnaire results on self-location, but there was no correlation between the proprioceptive measurement and the self-reported result. Previous literature reported that the contribution of the sense of self-location was relatively small (Heinrich et al., 2020), but there were deviations with all virtual hands between the perceived and actual positions by the implicit measurement in this study. Although there is no significant effect with the five hands, the application of the implicit method in this study could be a reference for further research about exploring the sense of self-location in VR. In this study, there was no relationship between self-reported self-location and eye tracking data, while the correlation between self-location and time to fixation was reported from a previous finding (Krogmeier and Mousas, 2020). Therefore, the current conclusion about the relationship between self-location and eye tracking is unclear.

A surprising result about perceiving hand length was found in this study. Participants underestimated their real hand length when using virtual hands. They perceived their hands in VEs to be shorter than their actual length, which was an interesting conclusion in this study. Similarly, an article with a psychological background reported underestimating finger length, even though no reasonable explanation was put forward (Ingram et al., 2019). In our study, users’ eye tracking data were detected and analyzed to help explore their embodied experience in VEs. Physiological data of users are becoming promising methods of user experience evaluation for researchers and designers, such as eye tracking (Zhang et al., 2022a) and electroencephalography (Alchalabi et al., 2019), along with the development of artificial intelligence for physiological data processing (Batliner et al., 2020; Wan et al., 2021; Yu et al., 2022). Using eye tracking and other physiological data to infer user reactions and perceptions toward difference virtual avatar designs could be an exciting direction for future work. In the current study, only a neutral virtual hand representation was used for the realistic hand design without reflecting any gender or personalization. Distinguishing different genders of virtual hands or designing personalized virtual hands for VR applications may lead to a deeper investigation of users’ embodied cognition in future. Furthermore, it is suggested that implementing cognitive artificial intelligence in VR systems can make them capable of analyzing and making decisions like human users (Zhao et al., 2022), which can help import the cognition capability to systems for better interaction between people and VR.

Design consideration

The current study found that participants produced the feeling of owning, controlling, and being inside virtual hands with different hand representations, and they experienced these virtual hands as their own hands to some extent by pressing the virtual buttons. Participants rated lower embodiment with the block hand. There are no significant findings on the other four hands, but the virtual human hand with the highest expectation owned a lower response about body ownership. These findings about the impact of hand representation on the sense of embodiment in VR might contribute to virtual avatar design in future VR applications in the design industry.

Design virtual hands that can clearly show finger movements to increase users’ embodiment. When participants used hand (including finger) tracking technology to interact with the virtual world, they reported the lowest embodiment on block hand in this study. Participants were expected to apply virtual hands with fingers, which could present the detailed movement of each finger. They could feel bored when completing virtual tasks with no-finger hand. Therefore, it is recommended to design virtual hands that can clearly show the finger movements to increase the embodiment in VEs, when users are expected to interact through finger tracking.

Virtual hand design with hollowed representations can be a good solution when the virtual scene beneath the hand is important during the interaction. Several participants who used Leap motion before reported that they felt more comfortable using the capsule hand than other hands for interaction in VR, which may affect subjective conclusions to some extent. In addition, participants commented after the experiment that they could watch more virtual space with minimal and capsule hands, due to these hollowed representations. These two hands have the unique advantage due to their hollowed appearance, which allows users to understand better what is happening beneath the virtual hand and eventually leads to a better feeling of control during the interaction. When the virtual scene beneath the virtual hand is important for manipulation in VR applications, the minimal and capsule hands could be good choices for designers and developers, resulting in a better controlling experience for users.

The realistic hand with high realism level should be used with particular attention to avoid negative influence on user experience. The human hand (with a high realism level) that was similar to users’ own hands came with the highest expectations, and participants were more likely to carefully compare the difference between the virtual hand and their own hand. A sense of loss could be generated if they felt that the human hand did not match their own hands perfectly. Meanwhile, the fear toward a realistic hand could be generated due to the Uncanny Valley effect. Additionally, it was noticed that most females had a negative attitude toward the human hand with dissimilar hand sizes and appearances. The findings from this study suggest that the realistic hand need to be applied in VR applications with caution. Because it looks like the real hand and leads to high expectations, users will feel a sense of loss if it does not meet the prospects after comparing it with their real hands. The application of the human hand in VR applications needs to consider many aspects, such as size, gender and personalization, and avoid the negative effect of Uncanny Valley.

Conclusion

This research investigated the impact of five virtual hand designs on users’ embodiment in VR by systematically integrating subjective and objective measurements. This study has offered new insights into the subcomponents of embodiment, that is, the sense of body ownership, agency, and self-location, respectively. Besides self-reporting measures, this study evaluated the agency and self-location through implicit methods derived from psychology. In addition, eye tracking data were used as the objective method to help explore the connection between the virtual hand designs and the sense of embodiment. Our findings indicated that the low realism level of hand (block hand) led to the lowest scores of three subcomponents of embodiment from self-reports, while implicit methods showed no significant differences. We also found that the results of using the human hand showed middle and high user attention. The theoretical implication of the results of the embodiment and its three subcomponents were discussed, and the practical implication was provided on the design of virtual hands. This study makes an essential contribution to further understanding users’ embodiment with various virtual hand designs in VR and the practical design considerations for virtual hand design in VR applications.