Feasibility and Acceptance of Direct-to-Home Tele-neuropsychology Services during the COVID-19 Pandemic

Abstract

Objective:

Neuropsychological assessment via video conferencing has been proposed during the COVID-19 pandemic. Existing literature has demonstrated feasibility and acceptance of neuropsychological measures administered by videoconference, although few studies have examined feasibility and patient acceptance of TNP visits directly to patients’ homes (DTH-TNP).

Methods:

We modified a previously published patient satisfaction survey for DTH-TNP and developed a clinician feasibility survey to examine experiences during DTH-TNP.

Results:

Seventy-two patients (age range: preschool-geriatric) evaluated by DTH-TNP for cognitive problems at an academic medical center responded to voluntary surveys between April 20, 2020, and August 19, 2020, and 100% indicated satisfaction. Fifty-nine percent of patients reported limitations (e.g., technological concern) during the appointment. 134 clinician surveys were collected and indicated that clinicians achieved the goal of their appointment in 90% of encounters.

Conclusions:

These qualitative data suggest that patients and clinicians found DTH-TNP to be satisfactory during the COVID-19 pandemic, while also recognizing limitations of the practice. These results are limited in that voluntary surveys are subject to bias. They support the growing body of literature suggesting that DTH-TNP provides a valuable service, though additional research to establish reliability and validity is needed.

INTRODUCTION

The novel coronavirus (COVID-19), declared a global pandemic on March 11, 2020 (Cucinotta & Vanelli, 2020), placed unprecedented demand on healthcare systems, resulting in delays of nonurgent appointments and widespread closures. Many neuropsychology clinics were abruptly closed, and neuropsychologists were faced with a choice between ceasing to care for patients or finding alternative methods for delivering services.

The remote diagnosis and treatment of patients by means of telecommunications technology, “telemedicine”, originated in the 1960s (Wittson, Affleck, & Johnson, 1961). Telemedicine use expanded dramatically during the first month of COVID-19 closures, leading to a 683% increase in the use of telehealth (Mann, Chen, Chunara, Testa, & Nov, 2020). The application of telemedicine practices to neuropsychology and teleneuropsychology (TNP) has proven to be feasible in multiple studies (Harrell, Wilkins, Connor, & Chodosh, 2014; Hodge et al., 2019; Parikh et al., 2013; Sutherland, Trembath, Hodge, Rose, & Roberts, 2019; Turner, Horner, Vankirk, Myrick, & Tuerk, 2012; Wadsworth et al., 2016), and preliminary reliability studies have been promising (Brearly et al., 2017; Cullum, Hynan, Grosch, Parikh, & Weiner, 2014). Parikh et al (2013) used patient surveys to evaluate the feasibility and acceptance of neuropsychological evaluations via remote video connection, and 98% of patients reported positive experiences with the evaluation process.

Prior to the initial submission of this manuscript, most studies evaluated assisted-TNP, in which a neuropsychologist interacts with patients in a clinical space at a remote location, ensuring adequate internet connectivity, a distraction-free environment, and in some studies, providing a proctor to manage test materials. The assisted-TNP model has extended the reach of neuropsychology to remote areas (Harrell et al., 2014; Turner et al., 2012), but did not reduce risk of COVID-19 exposure to the same extent as could be accomplished by providing direct-to-home-teleneuropsychology (DTH-TNP). The Inter-Organizational Practice Committee (a collaboration of neuropsychology professional groups) published guidance for DTH-TNP (https://iopc.online/teleneuropsychology-guidelines), and recent update has provided a careful consideration of risks and benefits of neuropsychological evaluation in different settings during the COVID-19 pandemic (Postal et al., 2021). Over the past several months, the feasibility and acceptance of DTH-TNP has been a focus of study in children (Harder et al., 2020), as has validation of a telephone cognitive screening test in patients with Parkinson disease (Benge & Kiselica, 2021).

We report preliminary findings regarding clinician feasibility and patient acceptance of DTH-TNP in an academic medical center lifespan neuropsychology practice that serves patients from a broad range of referral sources, the Massachusetts General Hospital Psychology Assessment Center (MGH-PAC). Based on findings in the assisted-TNP literature, we hypothesized that patients would endorse high acceptance rates and that neuropsychologists would attest to the feasibility of DTH-TNP.

Methods

Participants and Process

The MGH-PAC was closed on March 13, 2020. Over the ensuing weeks, a service development process took place (detailed in a separate manuscript under review) to develop DTH-TNP capabilities. Patients underwent an informed consent process reviewing the novel nature of this service and were given the option to be seen in person at a later date. Patients completed DTH-TNP visits, involving interview, formal testing, and/or feedback. This project was undertaken as a Quality Improvement Initiative of the MGH-PAC and followed the policies laid out by the Institutional Review Board.

Patient Surveys

A previously published patient survey (Parikh et al., 2013) was modified for DTH-TNP and implemented in the Research Electronic Data Capture (REDCap) system. After completion of a DTH-TNP appointment, patients or their caregivers were invited to complete the voluntary satisfaction survey. To reduce any possible feelings of coercion, surveys intentionally did not include patient identifying information (e.g., age ranges only), though patients were given the option to identify the clinician. Surveys, offered in both English and Spanish, examined satisfaction using a five-point Likert scale (see Supplementary Table 1a) and free-text responses.

Clinician Surveys

Following the assessment, the 10 neuropsychologists at the MGH-PAC were encouraged to complete a survey to assess feasibility (see Supplementary Table 1b). Individual clinician identifiers were not captured and surveys were combined in the data set.

Data Analysis

Responses to survey items are reported as percentages. Responses of “strongly agree” and “agree” were combined to indicate agreement, “strongly disagree” and “disagree” to indicate disagreement, and responses of “neutral” were not analyzed. Subgroup comparisons were conducted using chi-square tests for categorical data (e.g., checked vs. not checked items on the survey). Free-text responses were reviewed by three members of the study team (MWP, MG, KP) and are reported qualitatively.

Results

Patient Feedback

The patient feedback survey was launched on April 20, 2020, and 72 surveys were completed by August 19, 2020 (not all respondents completed every item, thus the below totals may not sum to 72). Respondents were primarily the patient themselves (n = 59, 82%), though some were completed by a parent/guardian (n = 7, 10%) or caregiver (n = 5, 7%). One survey was completed in Spanish. The median duration of DTH-NP appointments was 120 min (standard deviation = 80 min; range = 30–440 min). Patients across the lifespan were represented in the survey (Figure 1a). Appointments included multiple activities (interview, testing, and feedback), with 56 (78%) respondents indicating that their appointment included testing (median length = 140 min). The remaining visits were interview (n = 13, 18%) or feedback (n=3, 4%) only (median length = 60 min). All respondents indicated satisfaction with the DTH-TNP visit (Figure 1b), indicated that they felt understood by the examiner, and the vast majority would recommend DTH-TNP to others. There were no differences in the degree of satisfaction (e.g., “strongly agree” vs “agree”) between patients who did vs. did not have testing as part of their appointment, X ² (1, n = 72) = 2.77; p = .10 (ns). Technical difficulties were noted in some instances, and a small number of individuals felt that there were difficulties with communication. Despite the favorable responses, a majority of patients indicated that they would travel some distance for an in-person appointment (Figure 1c).

Fig. 1. Agreement with specific items from the patient survey expressed as percentages of the total number of respondents (n = 72). (a) Age of respondents (note, patients provided age range rather than specific age to protect anonymity of respondents), (b) response to satisfaction questions, and (c) preference for driving distance or virtual assessment. For the purposes of illustration, responses of “strongly agree” and “agree” were combined into the bar indicating agreement and responses of “strongly disagree” and “disagree” were combined into the bar indicating disagreement.

Respondents endorsed a number of favorable features of the virtual assessments compared with their expectations of in-person assessments (59 of 61 surveys included responses and endorsed saved travel time: n = 48, 79%; reduced risk of COVID exposure: n = 47, 78%; reduced anxiety: n = 13, 21%; easier to concentrate: n = 9, 15%), though some also indicated potential advantages of an in-person visit (36 of 61 surveys included responses and indicated that an in-person assessment would: allow more extensive assessment: n = 19; 53%; promote better personal connection with the examiner n = 15; 42%; allow better communication n = 8; 22%; make it easier to express concerns n = 7; 19%).

Thirty-eight respondents provided free-text comments, which were categorized as positive (n = 19, 50%), negative (n = 7, 18%), or a mixture of positive and negative input (n = 12, 32%). Positive comments related to the examiner (e.g., “She was extremely thorough… made sure I was comfortable…”; “the doctor was unbelievably competent, caring, and thoughtful…”) and the convenience of being able to complete testing from home (e.g., “My husband was able to join the session from quarantine…”; “Virtual assessments are less intrusive to my schedule.”). Themes from negative feedback included technical difficulties (n = 10; e.g., “My computer had problems and I lost connection many times”; “There was a small glitch in getting connected…”) and limitations on the testing imposed by the virtual modality (n = 3; e.g., “I didn’t like that the examiner could not see my drawing and writing task”).

Clinician Feedback

Providers completed 134 surveys (of a total possible evaluation count of 237, representing a 57% return rate) between April 22, 2020, and August 21, 2020. The median age of patients for which surveys were completed was 58 years (SD = 21.79; range = 5 to 88) and 47% (n = 64) were female. The median duration of virtual visits was 120 min (SD = 68; range = 30–480). As shown in Figure 2, 82% of cases involved testing. Providers reported that they were able to achieve the goal of the assessment in 90% of cases, and rate of goal achievement did not differ between visits that included formal testing (88%) and those that did not (94%); X ² (1, n = 134) = 1.25; p = .26 (ns). Neuropsychologists reported factors that interfered with their ability to achieve the goal of assessments included the inability to administer high-level executive or attention tasks in (n = 6; 4%), severe cognitive impairment or behavioral regulation issues on the part of the patient (n = 4; 3%), and technical or logistical issues (poor connection, provider not licensed in patient’s state; n = 4; 3%). The assessment was divided into more than one session in 51% of cases (n = 69); the remainder were completed in a single session. There was no difference in the rate of goal achievement between single session evaluations (86%) or multisession evaluations (93%); X ² (1, n = 134) = 1.56, p = .21 (ns). Testing assistance (e.g., psychometrist) was used in 30% of cases (n = 40).

Fig. 2. Agreement with specific items from the clinician survey expressed as percentages of the total number of respondents (n = 134). (a) Proportion of assessments involving specific activities, (b) methods of assessment, (c) problems endorsed by clinicians, and (d) overall feedback. Only the question regarding achievement of evaluation goals included an option for “partially”.

Clinicians indicated that no modifications to standard procedures were needed in most evaluations (n = 104, 77%), but endorsed using brief instructions (n = 12, 9%), taking frequent breaks (n = 12, 9%), or having another person or family member assist with technical issues (n = 19, 14%) as common strategies. Specific challenges that occurred during the assessments (64 surveys included responses; 47%) included interruptions, technical problems, and patient cognitive or perception problems.

The most common themes in free-text response regarding challenges during the assessment were technical issues (n = 51 comments, e.g., “Patient could not hear audio if they responded at the same time”; “Patient experienced poor internet connection”), poor engagement, distraction, or fatigue on the part of the patient (n = 37 comments, e.g., “Although I asked him to turn off his phone, the patient was clearly getting text messages that distracted him”; “Men were cleaning windows outside of where the patient was sitting”), and delays in the examination (n = 15 comments; e.g., “Patient overslept and I had to call to wake them up”). Strategies used by the clinicians included technical assistance for patients (e.g., family member in the home, 36 comments) and taking extra time to manage the evaluation (e.g., extended planning time, breaking testing into multiple sessions, 58 comments).

DISCUSSION

Patients and clinicians found DTH-TNP to be both acceptable and feasible. One hundred percent of patients who responded to this survey reported satisfaction and 93% indicated that they would recommend this type of appointment to others, despite the fact that 59% of patients noted some issues or limitations. These findings suggest that patients recognized both the benefits and limitations of the DTH-TNP evaluation. Indeed, only 26% of patients stated that they preferred virtual to in-person appointments. Providers were able to achieve the goal of their visit in 90% of the clinical encounters. To our knowledge, this is the first demonstration of acceptance and feasibility of DTH-TNP in a lifespan clinical sample.

DTH-TNP allows neuropsychologists to reach patients when traditional face-to-face approaches are not possible. In our clinical experience, results from the DTH-TNP evaluations that we conducted have been accepted as valid by consumers of neuropsychological services in the medical, educational, and legal communities. Neurologists, psychiatrists, primary care, and therapy providers are using recommendations from these DTH-TNP evaluations to inform differential diagnosis or adjust treatment course. School districts are using information about diagnosis and recommendations to provide services for students. Results from some of these DTH-TNP evaluations are being used in pursuit of legal guardianship. Thus, despite the novelty of the practice, the DTH-TNP evaluations we are conducting seem to be recognized across multiple settings as similar to in-person evaluations.

Limitations include the fact that survey responses were voluntary and were not completed by all patients or providers after every encounter, reflecting the fact that the survey was implemented as an early QI project. We also do not have data regarding individual characteristics of respondents, other than age range. While this step was taken to ensure anonymity and reduce potential bias in the responses, it may be that those patients who were not satisfied with the experience chose not to respond to the survey. We believe this is unlikely, given the relatively balanced feedback we received, which included challenges, concerns, and areas for improvement. Because the patient survey was completed by either a patient or a caregiver, the results may reflect more than one perspective, which cannot readily be disentangled. Our approach does not allow a comparison of satisfaction with DTH-TNP to in-office neuropsychological assessment, since most of these patients had no basis for comparison (though there were some comments suggesting a favorable view of the virtual visit as compared with patients’ prior neuropsychological experience: e.g., “Regarding virtual vs. non-virtual, I would say they were about equal”). The research was conducted during the initial phase of the COVID-19 pandemic, and while there have been intermittent peaks in infection rate since that time, neuropsychologists may now be more informed and equipped to provide in-office services than they were during that early stage. During the period covered by this survey, many states made provisions for service by providers licensed elsewhere. In the future, modifications to licensure laws, such as those initiated by the Association of State and Provincial Psychology Boards (https://www.asppb.net/page/PSYPACT), may be needed to provide DTH-TNP services to patients outside their home state.

Acceptance and feasibility are only the first steps in developing a clinically viable service. The reliability and validity of neuropsychological tests in the DTH-TNP setting have not been established. Although the early studies of reliability and validity using assisted-TNP are promising (Brearly et al., 2017; Cullum et al., 2014; Marra, Hamlet, Bauer, & Bowers, 2020; Sutherland et al., 2019; Wadsworth et al., 2016), the lack of control over the testing environment in DTH-TNP raises concerns for compliance with standardized procedures, additional distractions, technical limitations, and problems with test security (Bilder et al., 2020; Hewitt & Loring, 2020). Further exploration of limitations on the test battery imposed by DTH-TNP conditions, as compared with in-person evaluations, may be important in understanding the limits of this modality. The ability to accurately assess test effort and performance validity in the DTH-TNP context is an unresolved issue. Given that clinicians are likely relying more heavily on experience and clinical judgment when using TNP methods, investigating whether additional training is warranted and how psychometricians and neuropsychology trainees will respond/adapt may also be important issues to study. Perhaps of most concern, some individuals may lack access to the equipment needed for DTH-TNP (e.g., computer with camera) or are in a home that does not provide a suitable environment for testing. Methods to address disparities in access will be critical to the value of DTH-TNP.

These preliminary and qualitative results demonstrate that patients were satisfied with their DTH-TNP experience, and clinicians felt that they were able to successfully conduct evaluations using a DTH-TNP approach. These findings and those of many others (Cullum et al., 2014; Cullum, Weiner, Gehrmann, & Hynan, 2006; Harder et al., 2020; Hewitt & Loring, 2020; Marra et al., 2020; Temple, Drummond, Valiquette, & Jozsvai, 2010) suggest that DTH-TNP may be a relevant method for delivery of neuropsychology services and may have the potential to expand access to neuropsychological services over the long term. These results build on TNP research over the past two decades, but work remains to determine the long-term viability of this practice. Given the potential advantages of these methods, we feel that continued exploration of its strengths and limitations is warranted.