Using augmented reality to assess spatial neglect: The Free Exploration Test (FET)

Abstract

Background:

To capture the distortion of exploratory activity typical of patients with spatial neglect, traditional diagnostic methods and new virtual reality applications use confined workspaces that limit patients’ exploration behavior to a predefined area. Our aim was to overcome these limitations and enable the recording of patients’ biased activity in real, unconfined space.

Methods:

We developed the Free Exploration Test (FET) based on augmented reality technology. Using a live stream via the back camera on a tablet, patients search for a (non-existent) virtual target in their environment, while their exploration movements are recorded for 30 s. We tested 20 neglect patients and 20 healthy participants and compared the performance of the FET with traditional neglect tests.

Results:

In contrast to controls, neglect patients exhibited a significant rightward bias in exploratory movements. The FET had a high discriminative power (area under the curve = 0.89) and correlated positively with traditional tests of spatial neglect (Letter Cancellation, Bells Test, Copying Task, Line Bisection). An optimal cut-off point of the averaged bias of exploratory activity was at 9.0° on the right; it distinguished neglect patients from controls with 85% sensitivity.

Discussion:

FET offers time-efficient (execution time: ∼3 min), easy-to-apply, and gamified assessment of free exploratory activity. It supplements traditional neglect tests, providing unrestricted recording of exploration in the real, unconfined space surrounding the patient.

Introduction

Spatial neglect is the dominant cognitive disorder subsequent to mainly right hemispheric brain injuries (Becker & Karnath, 2007; Buxbaum et al., 2004). Typically, spatial neglect is induced by strokes in the territory of the right middle cerebral artery that damage the right perisylvian network, consisting of the superior/middle temporal, parietal, and ventrolateral frontal cortices (Karnath & Rorden, 2012). When exploring the surroundings, the patient’s visual and tactile exploratory activity shows a notable bias toward the side of the brain lesion, that is, mostly to the right side (Karnath et al., 1998; Karnath & Perenin, 1998). This shift is paralleled by a consistent bias of eyes and head to this side (Becker & Karnath, 2010; Coelho-Marques et al., 2022; Fruhmann Berger et al., 2006). Accordingly, objects or people on the left side are not found and ignored.

A large part of the diagnostic effort in identifying spatial neglect therefore consists of documenting the patient’s exploratory activities. Although it would be desirable to record the distorted activity in the real, unconfined space, this is not an easy undertaking and would also be costly when considering the use of gaze and/or limb motion capture systems. Therefore, a number of simple, clinical methods have been developed over time to estimate the bias of exploratory activity as best as possible within a manageable framework: the Letter Cancellation Test (Weintraub & Mesulam, 1987), Bells Test (Gauthier et al., 1989), Ota Test (Ota et al., 2001), or, for example, the Apples Test (Bickerton et al., 2011). The idea behind these tasks is comparable; the patient is presented with a sheet of paper on which target stimuli are to be crossed out among distractors. Depending on the severity of the disorder, patients disregard a greater or lesser proportion of the characters on the contralesional side of the search fields. A sensitive and robust measure of the severity of the spatial neglect in such tasks is the center of calculation measure (Center of Cancellation [CoC]; Rorden & Karnath, 2010). In recent times, presentation of such search and cancelation tasks has been extended to digital platforms (e.g., tablets). This appears to be beneficial due to the possibility of collecting additional variables, the reduction of error-proneness, and time savings. It also seems possible because neglect patients ignore a comparable ratio of contralesional target stimuli, regardless of modalities (paper and pencil vs. digital) and regardless of display sizes (Rosenzopf et al., 2023). With higher task complexity (Knoppe et al., 2022), it seems possible to detect milder forms of neglect when the diagnostic tasks are presented on larger display sizes (Villarreal et al., 2022).

Nevertheless, the exploration behavior of patients in these tasks still is restricted to a working space defined by the sheet of paper or by the screen of the digital device, respectively. The search field to be processed has been significantly expanded through the use of immersive virtual reality (VR) technology, using a head-mounted display (HMD). In these tasks, patients are asked to respond to single target stimuli (Dvorkin et al., 2012, Ogourtsova et al., 2018; Yasuda et al., 2020), respond to target stimuli under a series of distractions (Kim et al., 2021; Knobel et al., 2021; Perez-Marcos) or to freely explore the virtual environment while the patient’s eye and head movements are recorded (Belger et al., 2023; Hougaard et al., 2021). Target stimuli should be responded to either by touching them with a controller held in the dominant hand (Knobel et al., 2021), by giving verbal feedback if the target is detected (Perez-Marcos et al., 2023; Yasuda et al., 2020), or by pressing a button (Belger et al., 2023; Dvorkin et al., 2012; Kim et al., 2021). While these tasks allow to overcome the spatial limitations of a two-dimensional sheet of paper or a tablet, the exploration activity is still limited to the area of the virtual stimuli presented: the implemented arrays vary from 110° to 180° horizontally and 60°–120° vertically.

A task of “free exploratory activity,” however, would have to go beyond any spatial restriction. This has been undertaken – to our best knowledge – only once. Karnath et al. (1998) used search coils, that is, contact lenses with embedded magnetic coils, in a spatially non-restricted area that entirely surrounded the patient. The patient sat on a chair in a light bulb shaped booth; the trunk was fixed to the chair. A random array of visual stimuli was presented inside the booth, covering an area of 280° of visual angle in the horizontal plane (± 140° right and left of the body’s mid-sagittal plane) and 100° in the vertical plane (± 50° above and below the patient’s eye level) so that it clearly exceeded − in the relevant horizontal plane − the anatomical limits of maximum rotation of the eye in the orbit and of the head on the trunk. The patient was instructed to search for a particular (non-existent) visual target, while the eye in the orbit and the head on trunk movements were recorded by two search coils. All other work using eye-tracking procedures reported so far used PC monitors or overhead projectors for stimulus presentation (Behrmann et al., 1997; Cazzoli et al., 2016; Kaufmann et al., 2020; Machner et al., 2012; Ptak et al., 2009), again restricting exploratory activity to a limited area of space.

Of course, the implementation of recording free exploratory activity by search coils in an area of 280°×100° is impractical for clinical practice. Our aim thus was to develop an exploration task that (i) does not restrict search activity to a predefined, restricted area but rather allows free exploratory activity in space (up to the normal anatomical limits of effectors) and (ii) is easy to implement and administer in clinical practice.

Methods

The Free Exploration Test (FET)

We have developed an exploration task that is specifically optimized for tablet usage and operates through augmented reality (AR) technology (will become available under https://www.negami.de/en/free-exploration-test). AR allows to add virtual elements to reality that is captured via a tablet camera’s video stream. Participants, seated in a chair or wheelchair, are presented with a live video stream from the back camera of the tablet, which is displayed on the tablet screen (Figure 1). While performing the task, participants are allowed to hold the tablet in various ways, including ambidextrously from the right and left (if the force in both arms is sufficient) or centrally from the top or bottom (by using only one arm in cases of severe hemiparesis). To reduce strain, an optional Velcro hand strap can be attached to the back of the tablet. The task of the subject is to find a virtual element, an origami bird, in the real space surrounding him/her by moving the tablet. Starting from the patient’s straight-ahead position (0°) from which the task is initiated, the patient moves the tablet through space by arm movements in combination with active trunk rotation.

Figure 1. Familiarization phase of the Free Exploration Test. After some exploratory search movements (indicated by the orange track), the patient has found the bird at + 10° (i.e., 10° right of his/her straight ahead body orientation [corresponding to 0°] in the horizontal plane) and is about to place it in the circle to successfully complete the first run of the familiarization phase.

The Free Exploration Test (FET) begins with a familiarization phase. In this phase, the bird search task is performed twice. The investigator adjusts the settings to hide the bird at + 10° and then hands the tablet to the patient. When the patient holds the tablet in its straight-ahead position at 0°, the investigator presses the green button with the inscription “Start” to begin the task. The virtual origami bird is then hidden automatically with a spatial distance of + 10° on the subject’s ipsilesional, right side of surroundings space. The task is successfully solved when the participant has found the virtual origami bird and placed the bird into the circle in the middle of the screen (see Figure 1). The investigator then takes the tablet back out of the patient’s hand and sets up the second run in which the severity of the search is increased. The origami bird is now (automatically) hidden with a spatial distance of −10° on the subject’s contralesional, that is, the neglected left side (see Figure 1). If the patient should show difficulty in finding the bird in this or the first run of the familiarization phase, the investigator stops the search after 30 s respectively, to avoid frustration and demonstrates the location of the bird. For less impaired patients who would find the bird very quickly at ± 10° locations used in the familiarization phase, the investigator has the possibility to reduce the size of the bird so that it cannot be found immediately at first sight.

After completion of the familiarization phase, the actual test phase begins. Participants are instructed to find the origami bird again (the third time for the participant), but this time − without the patient being informed − no bird is hidden and exploratory tablet movements of the participant are recorded for 30 s (see Figure 2).

Figure 2. Test phase of the Free Exploration Test. Healthy subject (a) and neglect patient (b) searching for the (non-existent) target. In contrast to the familiarization phase (cf. Figure 1), no origami bird is hidden in the test phase (unknown to the subjects). The healthy subject explores left and right sides of space almost equally, while the neglect patient’s exploration is biased mainly to the right.

For the present study, we used an Apple iPad Pro 12.9 3rd generation on which we installed the FET app (will become available under https://www.negami.de/en/free-exploration-test). For a more comprehensive description of the technical implementation, further details are given in Stammler et al., (2023a). Based on the recorded exploratory movements during the test phase, the FET app calculates (i) in discrete 2° sectors along the horizontal axis the time spent (dwell time) exploring the surroundings and (ii) the average position of the patient’s horizontal exploration during the recorded 30 s.

Subjects

There were 20 patients with right-sided stroke and spatial neglect and 20 healthy elderly adults who participated in the study (for demographic details, see Table 1). Patients were recruited from three different rehabilitation facilities (Kliniken Schmieder, Stuttgart-Gerlingen; Neurologisches Rehabilitationszentrum Quellenhof, Bad Wildbad; Kreiskliniken Reutlingen, Reutlingen); 18 patients have already taken part in the study by Stammler and co-workers (2023b). All participants gave their informed consent to participate in the study, which was conducted in accordance with the ethical standards of the 1964 Declaration of Helsinki and was approved by the Ethical Committee at the Medical Faculty of Tübingen University.

Table 1. Demographic and clinical data of all right brain-damaged neglect patients and healthy elderly participants

Data are presented as mean (SD); CoC = Centre of Cancellation (Rorden & Karnath, 2010); EWB = endpoint weighting bias (McIntosh et al., 2017).

The inclusion criterion for the stroke patients was the presence of spatial neglect. In addition to clinical behavioral observations, diagnostic criteria had to be met in at least two of the following four neglect tests: the Letter Cancellation Test (Weintraub & Mesulam, 1987), the Bells Test (Gauthier et al., 1989), a Copying Task (Johannsen & Karnath, 2004), and a Line Bisection Task (McIntosh et al., 2005). All four neglect tests were performed on a Samsung S7+ tablet with screen dimensions 285×185 mm. The severity of spatial neglect in the cancellation tasks was determined by calculating the center of gravity of the target stimuli marked in the search fields, that is, the CoC (Rorden & Karnath, 2010). A CoC value ≥ 0.08 indicated left-sided spatial neglect (Rorden & Karnath, 2010). The Copying Task consisted of a complex scene consisting of four objects (fence, car, house, tree); points were assigned based on missing details or whole objects. One point was given for a missing detail and two for a whole object. The maximum number of points is therefore eight. A score higher than 1 (i.e., > 12.5% omissions) indicated neglect (Johannsen & Karnath, 2004). In the Line Bisection Task (McIntosh et al., 2005), patients were presented with 4 different line lengths 8 times each, that is, 32 lines in total. The cut-off value for spatial neglect was an endpoint weightings bias (EWB) value ≥0.07 (McIntosh et al., 2017).

Results

In Figure 3, the mean percentage of exploration activity is illustrated in discrete 2° sectors of visual angle along the horizontal plane; individual values of each patient and each healthy subject is illustrated in Figure S1 in the Supplementary. Negative gaze positions indicate positions to the left of the trunk’s mid-sagittal (straight-ahead) orientation; positive values indicate positions to its right. Both groups exhibited normally distributed exploratory activity in space (Figure 3). While the mean of exploration was close to the trunk’s mid-sagittal plane (+0.98°, SD 15.65) in the group of control subjects, exploration movements in the group of patients with spatial neglect were markedly biased toward the right. Exploratory activity of this group distributed around a mean of + 18.41° (SD 15.78) on the right. The 95% confidence interval for healthy controls was −1.6, 8.5 clearly different from that of the neglect patients (12.19, 24.42). Statistical comparison revealed a significant mean difference with large Cohen’s effects between the two groups (T [23.8] = 0.465, p < 0.001, d = 1.47). Further, we found that the span of exploratory activity along the horizontal plane was smaller with a mean of 98.79° (−28.71° left and 70.06° right) in the group of neglect patients than the span in the control group with a total of 212.64° (−103.76° left and 108.86° right). Statistical comparison revealed a significant difference with large Cohen’s effects (T [38] = −6.143, p < 0.001, d = −1.918).

Figure 3. Mean dwell time of exploratory activity recorded for the group of healthy subjects and the group of neglect patients along the horizontal plane during the 30 sec test condition in which no target object (origami bird) was hidden. Exploration is plotted in discrete 2° sectors along the horizontal axis. The dashed line shows the normal distribution curve for each group.

To compare the performance of our participants in the new FET with traditionally used neglect diagnostics, in each patient we calculated Person product-moment correlation coefficients between the FET and the CoC value measured in the Letter Cancellation and the Bells Test as well as the achieved points in the Copying Task and the EWB value of the Line Bisection Task. We observed positive correlation coefficients for all four tests: the Letter Cancellation Test (r = 0.56), Bells Test (r = 0.49), Copying Task (r = 0.41), and the Line Bisection Task (r = 0.22) (Figure 4). The correlation coefficients between all further tests as well as a mean value of all correlations are given in Table S1 of the Supplementary Material; the mean correlation values were highest for the Bells Test and FET.

Figure 4. Correlation between the new Free Exploration Test and the four traditional neglect tests: Letter Cancellation Test, Bells Test, Copying Task, and the Line Bisection Task in the patients with spatial neglect and their 95% confidence intervals (CI). CoC = Center of Cancellation; EWB = endpoint weighting bias.

To evaluate the performance of the FET, the receiver operating characteristic (ROC) value was calculated, also known as the area under the curve (AUC) (Hanley & McNeil, 1982). The calculated AUC value for the ROC analysis showed a high discriminative power of 0.89. This AUC value serves as an indication of the overall performance. To further refine the diagnostic utility of the FET, an optimal cut-off point was identified using a method described by Zweig and Campbell (1993) by using the point closest to 0-1 corner on the ROC curve. This point efficiently balances the sensitivity and specificity of the test and this procedure appears to be superior to other methods in estimating the true cut-off point (Rota & Antolini, 2014). With a sensitivity of 0.85 and a corresponding 1 specificity of 0.00, the selected optimal cut-off point was determined to be 8.99°. This is the threshold above which individuals were classified as having spatial neglect based on their performance on the FET.

In our dataset, 17 out of 20 neglect patients successfully met the designated cut-off score in the FET. The three patients (numbers 5, 16, and 18 in Table 2) who have not exceeded the cut-off score exhibited distinct characteristics: they either consistently bisected the presented lines accurately in the Line Bisection Task or displayed no rightward bias in the Cancellation Test. Thus, in contrast to the other 17 of the 20 neglect patients these three did not manifest neglect behavior in at least one of our four clinical tests. In order to measure the overall severity of spatial neglect expressed in all four clinical tests, we z-standardized the values of all patients. The 17 patients who were correctly diagnosed with the FET achieved a z-standardized mean of 0.18 (SD 0.09) and a 95% confidence interval ranging from 0.07 to 0.3. The three other patients achieved z-standardized means of −1.22 (SD 1.02), −1.18 (SD 0.38), and −0.72 (SD 0.87), respectively. They were all outside the confidence interval, indicating that their neglect was less severe than that of the other 17 patients.

Table 2. Results of all 20 neglect patients in the five tests: Free Exploration Test (FET), Letter Cancellation Test, Bells Test, Copying Task, Line Bisection Task. Black diamonds (♦) mark those results which are below the cut-off value of the respective test

CoC = Center of Cancellation; EWB = endpoint weighting bias.

Discussion

We evaluated a novel exploration task − the Free Exploration Test (FET) − utilizing AR technology to record exploratory activity in neurological patients. The FET successfully identified spatial neglect. Patients with spatial neglect exhibited a marked rightward bias in their exploratory movements, while the healthy control group displayed normally distributed exploratory activity around the trunk’s mid-sagittal plane. The FET turned out to have a high discriminative power; the optimal cut-off point for distinguishing neglect patients from healthy subjects was 9.0° on the right. Moreover, the FET correlated considerably with traditional neglect tests, that is, the Letter Cancellation Test, Bells Test, Copying Task, and the Line Bisection Task. The mean correlation values were highest for the Bells Test and the FET, indicating their validity as a diagnostic tool for spatial neglect. However, there were also three patients from our sample with neglect who were not diagnosed by the FET. These three patients were characterized by the fact that they did not meet the diagnostic criterion in at least one other traditional neglect test, which may reflect that the overall severity of their neglect was not particularly pronounced. This underlines the frequently stated insight that the diagnostic assessment of neglect should be based on several different tasks and not just one (Esposito et al., 2021; Karnath et al., 2023).

The decisive difference between the FET and traditional paper-and-pencil diagnostic tasks as well as the diagnostic exploration tasks implemented in VR environments is that no target stimuli are presented in the FET. This means that there is no predefined field to be explored, allowing to capture the patient’s “free exploratory activity” without restrictions (except for the normal anatomical limits of effectors) in the real, unconfined space. Beyond, the FET differs from traditional neglect tests in its time efficiency (execution time: ∼3 min) as well as in the gamification of the search task. The assessment can result in an enjoyable experience for the patients which is known to increase motivation (Darina et al., 2015; Popović et al., 2014). Three-dimensionality and gamification have also been implemented by the approach of assessing neglect through VR technology (Belger et al., 2023; Dvorkin et al., 2012; Hougaard et al., 2021; Kim et al., 2021; Knobel et al., 2021; Daniel Perez-Marcos et al., 2023; Yasuda et al., 2020). However, some of these virtual assessment tasks create VR that is so close to actual reality (e.g., Perez-Marcos et al., 2023: patients have to find and reach for a mug on a desk) that the question arises as to whether it is useful to generate them artificially with an HMD instead of performing them in reality with less technical effort. Instead, the use of VR methods appears to be particularly beneficial for creating and evaluating neglect patient’s behavior in an environment that cannot be tested in real life for safety reasons, such as crossing a street (Belger et al., 2023). Nevertheless, the use of any VR remains a cost-benefit consideration, and not only in financial terms due to the need for expensive hardware. VR may also cause symptoms of “cybersickness” such as nausea, blurred vision, headaches, and dizziness (Caserman et al., 2021; Simón-Vicente et al., 2022), caused by a sensory mismatch between the sensory signals received from the visual, vestibular, and proprioceptive systems. The symptoms can persist minutes (Szpak et al., 2020; Woo et al., 2023) or hours (Champney et al., 2007; Merhi et al., 2007) after virtual exposure. Whether or not improvement in technology (e.g., the improvement of visual display resolution) may reduce these symptoms is controversial (Rebenitsch & Owen, 2016). AR technology – as the FET in the present study – on the other hand, triggers cybersickness much less frequently compared to VR or is not known at all for tablet-based AR (Lawson & Stanney, 2021; Stammler et al., 2023). In addition, AR can be used on widely available existing devices such as tablets and smartphones; thus no additional costs are incurred by using existing devices.

Another advantage of FET is that it can be used as an independent test instrument for follow-up diagnostics during treatment and rehabilitation of the disorder. In neglect therapy, patients are often trained to focus their attention on specific target stimuli in the external environment in order to practice compensatory search strategies. An assessment task that captures the patient’s exploratory activity but is distinctly different from the material used in therapy is an interesting addition to the diagnostic arsenal, especially when assessing the impact of new therapy approaches.

It is important to note that our study has certain limitations that should be considered. Firstly, we did not include a third group of right hemispheric brain-damaged patients without neglect for comparison. While there is evidence that there is no significant difference in the performance of this group and healthy controls in the execution of virtual extended diagnostic tasks for neglect (e.g., Kim et al., 2021; Ogourtsova et al., 2018), it still needs to be proven in future studies. Secondly, while the FET is a time-efficient and patient-friendly diagnostic tool, future studies also should investigate its test-retest reliability and its correlations with activities of daily living. Lastly, in addition to the core aspects of spatial neglect (cf. Corbetta & Shulman, 2011; Karnath & Rorden, 2012), that is, the egocentric bias of patients to orient themselves to the right and ignore contralesionally located people or objects, further aspects exist in neglect patients that are not covered by the FET. For example, patients may exhibit allocentric deficits (Demeyere & Gillebert, 2019), disturbed auditory perception (Gokhale et al., 2013), and/or personal neglect of the own body (Committeri et al., 2006) (for an overview see Buxbaum et al., 2004; Williams et al., 2021). Additional tests must be used to record these aspects in neglect patients.

In conclusion, the FET appears to be a valuable addition to the arsenal of tools for diagnosing spatial neglect. Its ability to capture free exploratory activity and its correlation with traditional neglect tests makes it a promising tool for assessing the core aspects of this cognitive disorder. As a time-efficient, patient-friendly, and ecologically option, especially because existing devices such as tablets can be utilized, the FET can be easily integrated into clinical practice. Future research may further explore its potential for tracking changes in neglect severity over time and in response to rehabilitation interventions.