As part of the Research Domain Criteria (RDoC) initiative, the NIMH seeks to improve experimental measures of cognitive and positive valence systems for use in intervention research. However, many RDoC tasks have not been psychometrically evaluated as a battery of measures. Our aim was to examine the factor structure of 7 such tasks chosen for their relevance to schizophrenia and other forms of serious mental illness. These include the n-back, Sternberg, and self-ordered pointing tasks (measures of the RDoC cognitive systems working memory construct); flanker and continuous performance tasks (measures of the RDoC cognitive systems cognitive control construct); and probabilistic learning and effort expenditure for reward tasks (measures of reward learning and reward valuation constructs).

The sample comprised 286 cognitively healthy participants who completed novel versions of all 7 tasks via an online recruitment platform, Prolific, in the summer of 2022. The mean age of participants was 38.6 years (SD = 14.5, range 18-74), 52% identified as female, and stratified recruitment ensured an ethnoracially diverse sample. Excluding time for instructions and practice, each task lasted approximately 6 minutes. Task order was randomized. We estimated optimal scores from each task including signal detection d-prime measures for the n-back, Sternberg, and continuous performance task, mean accuracy for the flanker task, win-stay to win-shift ratio for the probabilistic learning task, and trials completed for the effort expenditure for reward task. We used parallel analysis and a scree plot to determine the number of latent factors measured by the 7 task scores. Exploratory factor analysis with oblimin (oblique) rotation was used to examine the factor loading matrix.

The scree plot and parallel analyses of the 7 task scores suggested three primary factors. The flanker and continuous performance task both strongly loaded onto the first factor, suggesting that these measures are strong indicators of cognitive control. The n-back, Sternberg, and self-ordered pointing tasks strongly loaded onto the second factor, suggesting that these measures are strong indicators of working memory. The probabilistic learning task solely loaded onto the third factor, suggesting that it is an independent indicator of reinforcement learning. Finally, the effort expenditure for reward task modestly loaded onto the second but not the first and third factors, suggesting that effort is most strongly related to working memory.

Our aim was to examine the factor structure of 7 RDoC tasks. Results support the RDoC suggestion of independent cognitive control, working memory, and reinforcement learning. However, effort is a factorially complex construct that is not uniquely or even most strongly related to positive valance. Thus, there is reason to believe that the use of at least 6 of these tasks are appropriate measures of constructs such as working memory, reinforcement learning and cognitive control.

Baseline assessment of cognitive performance is common practice under many concussion management protocols and is required for collegiate athletes by the NCAA. The purpose of baseline cognitive assessment is to understand an athlete’s individual uninjured cognitive performance, as opposed to using population normative data. This baseline can then serve as a reference point for recovery after concussion and can inform return-to-play decisions. However, multiple factors, including lack of effort, can contribute to misrepresentation of baseline results which raises concern for reliability during return-to-play decision-making. Measuring effort across a continuum, rather than as a dichotomous variable (good versus poor effort) may provide informative insight related to cognitive performance at baseline.

Collegiate athletes (n = 231) completed the Immediate Post-Concussion Assessment and Cognitive Test (ImPACT) as part of their baseline pre-participation concussion evaluation. ImPACT creates composite scores of Verbal Memory, Visual Memory, Visual-Motor Speed, and Reaction Time. Baseline self-reported symptoms and total hours of sleep the night prior to testing are also collected through ImPACT. ImPACT has one embedded indicator within the program to assess effort, and research has identified an additional three embedded indicators. Athletes were also administered one stand-alone performance validity test, either the Medical Symptom Validity Test (n = 130) or the Rey Dot Counting Test (n = 101), to independently measure effort. Effort was estimated across a continuum (zero, one, two, or three or more failed effort indicators) with both stand-alone and embedded effort indicators. We evaluated the relationship between effort, symptoms, self-reported sleep, Reaction Time composite score and Visual-Motor Speed composite score using a linear regression model.

We found that 121 athletes passed all effort indicators, while 39 athletes failed only one effort indicator, 40 athletes failed two effort indicators, and 31 athletes failed three or four (three+) effort indicators. Self-reported symptoms and total hours of sleep were not related to effort, but Reaction Time and VisualMotor Speed composites were. Specifically, performance on the Visual-Motor Speed composite was significantly worse for athletes who failed two or three+ effort indicators compared to athletes who did not fail any, and performance on the Reaction Time composite was significantly worse only for athletes who failed three+ effort indicators. Additionally, athletes who failed one or more effort indicators and reported less sleep performed worse on both the Visual-Motor Speed and Reaction Time composites, compared to those who reported less sleep and did not fail any effort indicators.

Athletes who failed one effort indicator did not perform significantly worse on Reaction Time and Visual-Motor Speed composites compared to those who passed all effort indicators. However, 31% of athletes failed two or more effort indicators and these athletes performed worse on cognitive tests, likely due to factors impacting their ability to put forth good effort. These results suggest that effort is more complex than a previously used dichotomous variable and highlights the importance of using several indicators of effort throughout baseline assessments. In addition, the importance of sleep should be emphasized during baseline assessments, especially when effort is questionable.

We examined the use of pupillometry as an early risk marker of Alzheimer's disease (AD). Pupil dilation during a cognitive task has been shown to be an index of cognitive effort and may provide a marker of early change in cognition even before performance begins to decline. Individuals who require more effort to successfully perform a task may be closer to decline. We previously found greater compensatory effort to perform the digit span task in individuals with amnestic mild cognitive impairment (aMCI) who may be at greater risk for AD than individuals with non-amnestic MCI (naMCI). Task evoked pupil dilation is linked to increased norepinephrine output from the locus coeruleus (LC), a structure affected early in the AD pathological process. In this study, we measured pupil dilation during verbal fluency tasks in participants with aMCI or naMCI, and cognitively normal (CN) individuals. Based on our findings using the digit span task, we hypothesized that participants with aMCI would show greater compensatory cognitive effort than the other two groups.

This study included 101 older adults without dementia recruited from the UC San Diego Shiley-Marcos Alzheimer's Disease Research Center and San Diego community (mean [SD] age = 74.7 [5.8]; education = 16.6 [2.5]; N=58 female; N=92 White); 62 CN, 20 aMCI and 19 naMCI participants. Pupillary responses (change relative to baseline at the start of each trial) were recorded at 30 Hz using a Tobii X2-30 (Tobii, Stockholm, Sweden) during semantic (animals, fruits, vegetables) and phonemic (letters F, A, S) fluency tasks. Participants generated as many words as possible in a category (semantic) or starting with a given letter (phonemic) in 60 seconds.

Repeated measures ANOVA (3 groups X 2 fluency conditions) with age, education and sex as covariates showed a significant main effect of group (F(2,95)=3.64, p=.03), but no group X condition interaction (F<1). Pairwise comparisons showed significantly greater fluency task-evoked dilation for aMCI relative to CN (p=.015) and naMCI (p=.019) participants. When controlling for performance (total letter or category words produced), pupil dilation (cognitive effort) remained significantly greater in aMCI relative to the other two groups in both fluency conditions, suggesting pupil dilation informs risk beyond information provided by task performance.

In a previous sample of community-dwelling men who were an average of 13 years younger than the present sample, we found significantly greater pupil dilation during a digit span task in aMCI relative to naMCI and CN groups. In the present study, we replicated those findings in an older sample using a different cognitive task. Significantly greater pupil dilation was found in individuals with aMCI on verbal fluency tasks, indicating greater compensatory cognitive effort to maintain performance. Pupillometry provides a promising biomarker that might be used as an inexpensive and noninvasive additional screening tool for risk of AD.

To examine whether suboptimal performance as determined by formal validity testing would predict neurocognitive scores in a sample of 83 pre-surgical, non-litigating epilepsy patients.

Participants were 83 patients who underwent comprehensive outpatient neuropsychological testing as part as their evaluation as epilepsy surgery candidates. The sample consisted of 41 females and 42 males, with 72 patients identifying as White, 5 as Black, 2 as Hispanic, 1 as Asian, and 2 as other. Mean age was 36 (SD=12.4) mean FSIQ was 87 (SD=12.7), mean years of education of 12.9 (SD=2.1). Each patient’s assessment included a stand-alone performance validity test (PVTs)— Word Memory Test (WMT), the Test of Memory Malingering (TOMM), or the Medical Symptom Validity Test (MSVT)—as well as two embedded measures of validity—the California Verbal Learning Test Forced Choice (CVLT FC) and WAIS-IV Reliable Digit Span (RDS). Pass/fail rates were analyzed, with valid performance being defined as pass score on at least two of the completed PVTs (N=73 Pass Effort group 86.9%; N=10 Failed Effort group 11.9%). Point-biserial Pearson correlations were conducted to determine the relationship between validity pass/fail status and WAIS-IV FSIQ, VCI, and PRI scores, CVLT-II Trials 1-5 Total T scores, CVLT-II Long Delay Free Recall z scores, WMS-III Logical Memory II T scores, BVMT Total Recall T scores, BVMT Delayed Recall T scores, and Trail Making Test (TMT) B T scores.

Significant relationships were found between Failed Effort group and all neurocognitive scores except BVMT Total Recall. On average, the Failed Effort group obtained significantly lower FSIQ (M=76.57, SD=10.94), VCI (M=80.89, SD=16.03), PRI (M=81.00, SD=14.91), CVLT-II Trials 1-5 Total (M =34, SD=6.89), CVLT-II Long Delay Free Recall (M =-2.44, SD=1.43), WMS-IV Logical Memory II (M =4.83, SD=2.79), BVMT Delayed Recall (M=26.38, SD=6.41), and TMT B (M=29.70, SD=11.46) standard scores compared to the Pass Effort group (FSIQ M=88.09, SD=12.52; VCI M=92.13, SD=13.61; PRI M=91.14, SD=12.06; CVLT-II Trials 1-5 Total M=47.86, SD=12.02; CVLT- II Long Delay Free Recall M=-.44, SD=1.11; WMS-III Logical Memory II M=8.41, SD=3.17; BVMT Delayed Recall M=39.19, SD=12.66; TMT B M=39.34, SD=13.18). Correlation coefficients were r=-.266* (FSIQ), r=-.255* (VCI), r=-.271* (PRI), r=.361**(CVLT-II Total), r=-.474** (CVLT-II LDFR), r=-.298** (WMS-IV LM II), r=-.308** (BVMT DR), and r=-.240* (TMTB). All coefficients were significant at the .05 (*) or .01 (**) level.

Results suggest that pass/fail status on formal validity testing predicts depressed performance on a variety of neurocognitive measures. Therefore, predicting surgical outcome of resection/ablation (e.g., compensation of contralateral hemisphere) should not be based upon neuropsychological memory performance alone when there are failures on tests of engagement as memory scores have strong correlations to pass/fail status on formal validity testing. Overall, this emphasizes the importance of routinely integrating PVTs as part of pre-epilepsy surgery neuropsychological evaluations.

This retrospective study compared base rates of failure on a series of standalone and embedded performance validity tests (PVTs) in a sample of Spanish-speaking forensic litigants and explored the impact of demographic factors on PVT performance.

62 Spanish-speaking participants involved in litigation (primarily for work-related mTBI) underwent outpatient neuropsychological evaluation. Country of origin spanned South American (56.5%), Caribbean (22.5%), Central American (16.1%), North American (3.2%), and Spanish (1.6%) regions. Of this sample, 56 completed the Test of Memory Malingering (TOMM), 45 completed the Rey Fifteen Item Test (RFIT), and 49 completed the Dot Counting Test (DCT). Embedded validity measures, Reliable Digit Span (RDS) and the WHO-Auditory Verbal Learning Test (WHO-AVLT) were completed by 32 and 48 participants, respectively.

Effects of age (M=42.4, SD=11.72) and region of origin did not significantly impact overall performance on any measure. Mean scores across all standalone PVTs were below cutoffs, which have been previously suggested for use with Spanish-speaking populations (TOMM <40, RFIT total <21, DCT e-score >18). Overall base rates of failure were as follows: 52.5% TOMM (T1 M= 37.5, SD=10.7; T2 M=35.1, SD=10.6), 64.9% RFIT (M=17.8, SD=7.8) 57.6% DCT (M=18.3, SD=8.8), 51.1% RDS (M=6.1, SD=1.6), 29.4% WHO-AVLT (M=10.7, SD=3.9. Years of education (M=9.98, 3.96) was significantly correlated with RFIT total score (r(43) = .48, p<.01) and DCT e-score (r(47) = -.34, p<.05. When stratified by level of education (0-6, 7-11, and 12+), a large discrepancy in base rate of failure was observed on the RFIT, with failures in 92% of participants with less than six years of education, as compared to 52% and 59% failure in those with 7-11 and 12+ years, respectively. Variability in base rates of DCT failure across levels of education, although less extreme than on the RFIT, again demonstrated higher rates of failure in participants with less than six years of education (0-6: 71%, 7-11: 54%, 12+: 52%).

These findings add to the existing literature surrounding measurement of suboptimal effort in Spanish-speaking populations. Base rates of PVT failure on both standalone and embedded measures were generally much higher than those reported in prior studies of forensic or compensation-seeking groups, including some with Spanish speaking participants. These high rates of failure are likely attributable, at least in part, to sample characteristics, due to the high proportion of individuals engaged in litigation associated with workplace injuries on construction sites at the study location. Such findings illustrate the importance of a thorough effort assessment for this population. Finally, results demonstrating reduced specificity of the RFIT in Spanish-speaking participants with less than six years of education, suggesting caution is warranted for its use in neuropsychological evaluations with such individuals.

Neuropsychological evaluations are used to examine a person’s current cognitive functioning. Performance validity tests (PVT) are included in neuropsychological test batteries to ensure that examinees are performing to the best of their abilities and identify non-credible performance. There are two types of PVTs: freestanding and embedded. A freestanding PVT is a cognitive test created to evaluate performance validity and do not measure any type of cognition directly. Meanwhile, an embedded PVT is a task design to evaluate some sort of cognition (e.g., memory) by using traditional neuropsychological tests (e.g., Trail Making Test) and performance validity. Research suggests that undergraduate college students are not always performing to the best of their abilities when completing a comprehensive neuropsychological battery. In fact, in one study where an undergraduate college sample was given three PVTs, it was reported that 56% of the participants failed at least one PVT in their first session and 31% in their second session. Research has also shown that speaking multiple languages can influence cognition. The purpose of this study was to identify in three credible language groups of college students what PVTs does bilingualism influence higher failure rates. It was predicted that bilingual college students would significantly demonstrate higher PVTs failure rates compared to monolingual college students.

The sample consisted of 70 English first language monolinguals (EFLM), 33 English first language bilinguals (EFLB), and 68 English second language bilinguals (ESLB) that were psychologically and neurologically healthy. All participants completed a comprehensive neuropsychological battery in English. The Rey-Osterrith complex figure copy test, Comalli Stroop part A, B, and C, Trail Making Test part A and B, Symbol Digit Modalities Test written and oral parts, Controlled Oral Word Association Test (COWAT) letter fluency, and Finger Tapping Test were the tasks used as embedded PVTs to evaluate failure rates in our sample. Moreover, all participants were credible (i.e., they did not fail two or more PVTs). PVT cutoff scores were selected for each embedded PVT from previous literature. Chi-square analysis were used to evaluate failure rates between language groups on each PVT.

We found no significant failure rate differences between language groups on any of the PVTs. However, while no significant group differences were found, on the COWAT letter fluency results revealed higher failure rates between the three language groups (i.e., 13% EFLM, 24% EFLB, and 22% ESLB) compared to other PVTs.

Our data suggested no significant failure rate differences between language groups. It has been suggested in previous studies that linguistic factors impact PVT performance and test interpretation. On the COWAT letter fluency task, it is possible that language is driving higher failure rates between bilingual speakers, even though we found no significant failure rates or performance differences between the three language groups. Future studies should examine language groups and other cultural variables (e.g., time perspective) to determine what may be driving high failure rates on the COWAT letter fluency task in credible participants.

Workload is a useful construct in human factors and neuroergonomics research that describes “the perceived relationship between the amount of mental [and physical] processing capability or resources and the amount required by the task”. We apply this concept to neuropsychology and assess several dimensions of workload as it relates to performance on the Trail Making Test.

Twenty college students completed the Trail Making Test (TMT). After completion of each Part A and B, workload was assessed with the NASA-Task Load Index (NASA-TLX), a popular self-report measure of workload including subscales: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration, with an overall average total score as well.

Completion time differed of course between Parts A and B (p < .001). Of more interest, overall workload differed between TMT A (M = 20.33, SD = 13.32) and TMT B (M = 35.79, SD = 17.37) (p < .001, h2 = .68). The greatest subscale differences were with Mental Demand (p < .001, h2 = .68) and Effort (p < .001, h2 = .59), but Physical Demand also showed a difference (p < .007, h2 = .33). Temporal Demand showed the smallest and nonsignificant difference (p = .081, h2 = .152).

Based on previous research in our lab, most results were expected and understandable. As we know with the TMT, Part B is more cognitively demanding (in various ways) than Part A. The greater Physical Demand with Part B is a somewhat more complex finding, needing a solid explanation. Finally, the NASA-TLX appears to be a valid instrument of workload with a standard neuropsychologist test. We argue it can provide useful interesting information in the assessment of cognitive status in clinical populations.

The Test of Memory Malingering (TOMM) is a performance validity test (PVT) that aims to assess whether participants are giving adequate effort to perform well on tasks of memory performance (Tombaugh, 1996). Other PVTs, specifically the Forced Choice Recognition Trial in the California Verbal Learning Test, have shown that even single errors may indicate invalid performance (Erdodi et al., 2018). Finally, youth are often understudied in the PVT literature, and athletes are at increased risk of invalid performance on baseline testing due to many wanting to return to play following concussion (Erdal, 2012). Therefore, the objective of the current study is to examine whether single errors on TOMM Trial 1 are indicative of lower, and possibly invalid, cognitive performance in a youth sample, given that cognitive performance declines with even small decreases in effort (Green, 2007).

Healthy youth athletes (n=174) aged 8-16 years (M=12.07) completed the TOMM as well as other neuropsychological measures during baseline neuropsychological evaluation in a clinical research program for sports concussion. Independent samples t-tests compared youth athletes who scored 49 points on the TOMM (n=28) to youth athletes who scored a perfect 50 (n=50) across several groupings of neuropsychological measures. Participants who scored less than 49 or who didn’t complete the TOMM were excluded from the analyses.

Participants scoring 50/50 on TOMM Trial 1 scored significantly higher on Stroop Color Naming task (p=0.036), Verbal Learning Delayed task from the second edition of the Wide Range Assessment of Learning and Memory (WRAML-2, p=0.018), and Letter Number Sequencing task from the Weschler Intelligence Scale for Children (WISC-IV, p=0.025), relative to participants scoring 49/50. Though not statistically significant, results also showed a trend toward participants scoring 50/50 scoring higher on nearly every test in the battery.

Participants with a single error on TOMM Trial 1, as compared to participants with a perfect score, performed significantly worse on a processing speed task, a verbal learning task, and a working memory task as part of a comprehensive neuropsychological battery. The single-error group also trended toward scoring lower on nearly all of the remaining attention, processing speed, perceptual ability, memory, and executive functioning tasks in the battery. The results could lead to a more liberal interpretation of TOMM scores, given that the trend towards lower performance may be due to putting forth significantly less effort. These results point to the need for a similar comparison of the TOMM in a larger sample size, as greater power may reveal even more significant differences in performance. Findings also emphasize the importance of viewing performance validity on a continuum rather than as a dichotomous pass/fail. Understanding the TOMM and how single errors may be indicative of poorer performance in a youth sample could help to reframe the way PVT results are interpreted in clinical and forensic settings.

As neuropsychologists aim to collect valid data, maximize the utility of assessments, make effective use of time, and best serve patient populations, measurement of performance validity is considered a critical issue for the field. As effort may vary across an evaluation, including performance validity tests (PVTs) throughout the assessment is important. Incorporating embedded PVTs in addition to free standing PVTs can be particularly useful in this regard. COWAT and animal naming are commonly administered verbal fluency measures. While there have been past investigations into their potential for detecting invalid performance, they are limited, and more research is needed. Perhaps most promising, Sugarman and Axelrod (2015) described a logistic regression derived formula utilizing the combined raw scores of COWAT and animal naming. The current study aimed to investigate the use of embedded PVTs within COWAT and animal naming to provide further support for the use of embedded PVTs in these measures.

All subjects were from a mixed clinical sample comprising military veterans from two VA Medical Centers in the northeast U.S., who were referred for neuropsychological evaluation. Subjects deemed credible had zero PVT failures. Subjects were considered non-credible performers if they failed at least two out of a possible eight PVTs administered. Subjects who failed one PVT were excluded from the study (n = 53). The final sample consisted of 116 individuals with credible performance (Mean Age = 35.5, SD = 8.8; Mean Edu = 13.6, SD = 2; Mean Est. IQ = 106, SD = 7.9) and 94 individuals with psychometrically determined non-credible performance (Mean Age = 38.5, SD = 9.4; Mean Edu = 113, SD = 2.1; Mean Est. IQ = 101, SD = 8.7). Performance of COWAT and animals in detecting non-credible performances was evaluated through calculation of classification accuracy statistics and use of the logistic regression formulas reported in Sugarman and Axelrod (2015).

For COWAT, the optimal cutoff was a raw score of <27 (specificity = 89%; sensitivity = 31%), and a T-score of <35 (specificity = 92%; sensitivity = 31%). For animal naming, optimal cutoffs were <16 for raw score (specificity = 92%, sensitivity = 38%) and <37 for T-score (specificity = 91%; sensitivity = 33%). The logistic regression formula based on raw scores for both COWAT and animal naming was inadequately sensitive at the recommended cutoff in this sample, but a coefficient of > .28 was revealed to be optimal (91% specificity; 42% sensitivity). When the formula for T-scores was used, a coefficient of > .38 was optimal (91% specificity; 28% sensitivity).

Results of the current research suggest that PVTs embedded within the commonly administered COWAT and animal naming verbal fluency tests can effectively detect low effort, in concordance with generally accepted standards. A logistic regression formula using raw scores in particular appears to be most effective, consistent with findings reported by Sugarman and Axelrod (2015).

Previous literature has studied the cognitive processes that contribute to performance on the Stroop interference condition in adults and found that the Stroop task performance (i.e., color-word interference) is comprised of multiple cognitive skills, including speed of visual search, working memory, and conflict monitoring (Perianez et al. 2020). However, the relationship of these cognitive processes to Stroop interference in youth remains understudied. Moreover, no studies have examined the contribution of effort measurement to the interference condition in healthy youth.

Golden Stroop Test interference performance was examined in healthy youth athletes (n=174) aged 8-16 years (mean age=12.07) who participated in a baseline neuropsychological evaluation as part of a clinical research program on sports concussion. Predictor variables included speed of visual search, working memory, processing speed, verbal fluency effort (i.e., validity tests), visuospatial abilities, visual processing, and executive functioning skills such as cognitive flexibility and reasoning.

Speed of visual search as measured by Trail Making Test visual scanning time (p<0.00), and effort as measured by Reliable Digit Span and Trail Making Test ratio (p=0.03; p<0.00, respectively) significantly contributed to Stroop interference performance in healthy youth. We provided three validity measures; however, only those requiring higher-order cognitive processes predicted Stroop performance: Reliable Digit Span (p=0.03) and the Trail Making Test ratio (p<0.00). The standalone validity measure (TOMM) was not a significant predictor of Stroop performance (p>0.05).

In contrast to adults, working memory and processing speed did not significantly predict Stroop performance, while visual search speed did predict Stroop interference. Furthermore, two embedded validity indicator (EVI) measures predicted Stroop interference, in contrast to a standalone validity measure requiring lower cognitive processes, which did not predict Stroop performance. Therefore, EVI’s that include an executive functioning component may not accurately represent effort in youth, perhaps due to their less developed executive functioning relative to adults (Lezak et al., 2012; Shanmugan & Satterthwaite, 2017). Overall, understanding the cognitive processes contributing to Stroop performance in healthy youth will allow clinicians to better detect deficits in those cognitive processes and understand how they may impact Stroop performance. This would lead to a better understanding of executive functioning and the accurate measurement of effort in healthy youth.

Performance validity tests (PVTs) are included in neuropsychological testing to ensure examinees are performing to the best of their abilities. There are two types of PVTs: embedded and free standing. Embedded PVTs are tests that are derived from standard neuropsychological tests of various cognitive domains. Freestanding PVTs are tests that are designed with the intention of being a PVT. Research studies show that undergraduate samples do not always performed to the best of their abilities. The purpose of this study was to cross-validate previous research on the topic of performance validity in a college sample. It was predicted that the non-credible group would demonstrate higher failure rates on embedded PVTs compared to the credible group.

The sample consisted of 198 neurologically and psychologically healthy undergraduate students with a mean age of 19.69 (SD = 2.11). Participants were broken into two groups: non-credible (i.e., participants that failed two or more PVTs) and credible (i.e., participants that did not failed two or more PVTs). The Rey-Osterrith copy test, Comalli Stroop part A (CSA), B (CSB), and C (CSC), Trail Making Test part A and B, Symbol Digit Modalities Test written (SDMT-W) and oral (SDMT-O) parts, Controlled Oral Word Association Test (COWAT) letter fluency, and Finger Tapping Test were used to evaluate failure rates in our sample. PVT cutoff scores were use from previously validated in the literature. Chi-square analysis was used to evaluate failure rates between the groups.

Chi-square analysis revealed significant failure rate differences between groups on several PVTs. Results revealed that 15% of the non-credible group failed the CSA compared to 1% of the credible group, X2=14.77, p=.000. Meanwhile, 26% of the non-credible group failed the CSB compared to 2% of the credible group, X2=24.72, p=.000. Furthermore, results showed that 11% of the non-credible group failed the CSC compared to 1% of the credible group, X2=13.05, p=.000.Next, 48% of the non-credible group failed the Trail Making Test part A compared to 8% of the credible group, X2=31.61, p=.000. We also found that 15% of the non-credible group failed the SDMT-W part compared to 1% of the credible group,X2=19.18, p=.000. Meanwhile, on the SDMT-O part 19% of the non-credible group failed compared to 1% of the credible group, X2=25.52, p =.000. On the COWAT letter fluency task 74% of the non-credible group failed compared to 19% of the credible group, X2=36.90, p=.000. Finally, results revealed on the Finger Tapping Test 19% of the non-credible group failed compared to 3% of the credible group, X2=10.01, p=.002.

As expected, the non-credible participants demonstrated significantly higher PVT failure rates compared to credible participants. A possible explanation driving higher failure rates in our sample can be due to cultural variables (e.g., bilingualism). It was suggested by researchers that linguistic factors may be impacting higher PVT failure rates and developing a false-positive error. Future research using undergraduate samples need to identify which PVT’s are being impacted by linguist factors.

Few to no studies have directly compared the relative classification accuracies of the memory-based (Brief Visuospatial Memory Test-Revised Recognition Discrimination [BVMT-R RD] and Rey Auditory Verbal Learning Test Forced Choice [RAVLT FC]) and non-memory based (Reliable Digit Span [RDS] and Stroop Color and Word Test Word Reading trial [SCWT WR]) embedded performance validity tests (PVTs). This study’s main objective was to evaluate their relative classification accuracies head-to-head, as well as examine how their psychometric properties may vary among subgroups with and without genuine memory impairment.

This cross-sectional study included 293 adult patients who were administered the BVMT-R, WAIS-IV Digit Span, RAVLT and SCWT during outpatient neuropsychological evaluation at a Midwestern academic medical center. The overall sample was 58.0% female, 36.2% non-Hispanic White, 41.3% non-Hispanic Black, 15.7% Hispanic, 4.8% Asian/Pacific Islander, and 2.0% other, with a mean age of 45.7 (SD=15.8) and a mean education of 13.9 years (SD=2.8). Three patients had missing data, resulting in a final sample size of 290. Two hundred thirty-three patients (80%) were classified as having valid neurocognitive performance and 57 (20%) as having invalid neurocognitive performance based on performance across four independent, criterion PVTs (i.e., Test of Malingering Memory Trial 1, Word Choice Test, Dot Counting Test, Medical Symptom Validity Test). Of those with valid neurocognitive performance, 76 (48%) patients were considered as having genuine memory impairment through a memory composite band score (T<37 for (RAVLT Delayed Recall T-score + BVMT-R Delay Recall T-score/2).

The average memory composite band score for valid neurocognitive scores was T = 49.63 as compared to T = 27.57 for genuine memory impairment individuals. Receiver operating characteristic [ROC] curve analyses yielded significant areas under the curve (AUCs=.79-.87) for all four validity indices (p’s < .001). When maintaining acceptable specificity (91%-95%), all validity indices demonstrated acceptable yet varied sensitivities (35%-65%). Among the subgroup with genuine memory impairment, ROC curve analyses yielded significantly lower AUCs (.64-.69) for three validity indices (p’s < .001), except RDS (AUC=.644). At acceptable specificity (88%-93%), they yielded significantly lower sensitivities across indices (19%-39%). In the current sample, RAVLT FC and BVMT-R RD had the largest changes in sensitivities, with 19% and 26% sensitivity/90%-92% specificity at optimal cut-scores of <10 and <2, respectively, for individuals with memory impairment, compared to 65% and 61% sensitivity/94% specificity at optimal cut-scores of <13 and <4, respectively, for those without memory impairment.

Of the four validity scales, memory-based embedded PVTs yielded higher sensitivities while maintaining acceptable specificity compared to non-memory based embedded PVTs. However, they were also susceptible to the greatest declines in sensitivity among the subgroup with genuine memory impairment. As a result, careful consideration should be given to using memory-based embedded PVTs among individuals with clinically significant memory impairment based on other sources of information (e.g., clinical history, behavioral observation).

This study aimed to explore the possibility of using the Figure Weights subtest of the Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V) as an embedded validity test (EVT).

We conducted a retrospective cohort study of patients seen in the Johns Hopkins All Children’s Hospital (JHACH) Neuropsychology program between 2015 and 2019. Patient age ranged from 6-15 years (median age 11 years). All patients were administered the WISC-V as a portion of their neuropsychological evaluation. The sample included 75 patients who were generally male (63%), White (77%), English dominant (97%), and right-handed (81%).

Effort determination based on RDS identified more patients as having invalid effort. Clinicians identified only 7% of patients with invalid effort; whereas, 16% of patients with invalid effort were identified using the WISC-V RDS. Although patients having valid effort generally performed better on all WISC-V subtests, no significant differences between groups were found. Over 90% of patients were able to get items 1-10 correct on the WISC-V Figure Weights subtest regardless of their effort determination. WISC-V Figure Weights item analysis showed participants in the invalid group sometimes answered more difficult questions correctly while failing easier items which supports inconsistent effort. Further statistical analyses, including discriminant validity tests, were unable to be completed given the study was underpowered due to significant disparity between effort groups.

This study shows support for WISC-V Figure Weights subtest items 1-10 as an embedded EVT given these items were sufficiently easy to pass regardless of whether participant gave valid or invalid effort. As this is an exploratory study, results will need to be replicated in other pediatric samples. Additionally, the discriminant ability of the WISC-V Figure Weights subtest EVT will need to be further investigated.