Letter to the Editor: Importance of psychiatric confounding in non-randomized studies of heavy ecstasy users

We noted with interest the recent publication of two articles on cognitive functioning in heavy users of ecstasy (an illicit preparation represented as containing MDMA; ±3,4-methylenedioxymethamphetamine) and other drugs (Bedi & Redman, 2008; Schilt et al. 2008). We are grateful to Dr Bedi and Dr Redman for discussing the numerous methodological limitations and inconsistencies with research on cognition in ecstasy users. Cautionary notes to the readers are still warranted. Both studies were non-randomized, retrospective, cross-sectional studies, with subjects recruited from different locations, increasing the risk for important baseline differences due to selection bias. The mean lifetime ecstasy use in the two studies was 170 tablets (range 13.5–2407) and 327 tablets (range 15–2000), respectively. In contrast, only 20–30% of ecstasy users have lifetime consumption over 25 tablets (De Win et al. 2005).

Bedi & Redman found no group differences on verbal memory ability. Their groups of ecstasy-using and non-ecstasy-using polydrug users (total n=133) appeared relatively well-matched on a range of potential confounders, including education, depression, and anxiety in the subjects and substance abuse or psychiatric illness in first-degree relatives. After an automated variable reduction procedure that discarded most of the confounding variables, a weak negative correlation appeared between a ‘verbal memory’ factor and lifetime ecstasy use. Importantly, the authors fail to mention that such derived statistical models must always be verified on independent data (Good & Hardin, 2006). It would be useful to see whether lifetime ecstasy use still predicted verbal memory in the Bedi & Redman sample when all measured confounders are included in the analysis or at least examined in more detail. Overall, Bedi & Redman conclude that the ‘hypothesis that ecstasy users would display lower cognition than non-users was not supported’.

Schilt and colleagues' report is part of a large multi-part project called the ‘Netherlands XTC Toxicity (NeXT)’ study (De Win et al. 2005). Schilt and colleagues report a limited list of potential confounders, only gender, age, IQ, education level, and use of other substances. Although subjects with major psychopathology were excluded, no psychiatric family-history or lifestyle variables were reported. The limited demographic data was not reported separately for each group, and it is unclear if the groups were well-matched. Schilt and colleagues claim that in their sample ‘frequent ecstasy use is responsible for a drop of nearly two out of 15 words in a verbal delayed memory task’, an effect they call ‘quite substantial’; however, here they describe the raw difference between their non-randomized groups (total n=67), without adjusting for any confounders at all. After adjusting for age, gender, IQ, and other substance use, but not including education level or adjusting for multiple comparisons, weak associations appeared between ecstasy use and verbal delayed recall and verbal confabulations, but not verbal immediate recall or any of the other cognitive measures presented. In the same model, weak associations also appeared between alcohol use and verbal delayed recall and between gender and verbal immediate recall. Within the subjects who used ecstasy, a weak association appeared between lifetime ecstasy dose and verbal delayed recall (p=0.03, one-tailed), but not verbal confabulations. This dose–response analysis depends heavily on the unlikely assumption that the ecstasy-using subjects, with lifetime ecstasy doses ranging from 15 to 2000 tablets, are equivalent on all known and unknown potential confounders besides age, gender, IQ, and other substance use. The NeXT study design article lists three verbal memory subscores: immediate, delayed and recognition (De Win et al. 2005). However, in the current cross-sectional study the NeXT team, without explanation, replaced verbal recognition with a verbal confabulation subscore. Overall, the NeXT team overstate their findings when they conclude that their non-randomized study ‘strongly suggest a specific negative effect of ecstasy use on verbal memory’.

Non-randomized and retrospective studies are notoriously misleading on causation (Smith & Ebrahim, 2002). Childhood neglect has been associated with decreased verbal memory in adulthood (Grassi-Oliveira et al. 2008). Ecstasy users are more likely to report childhood physical abuse and neglect (Singer et al. 2004; Montgomery et al. 2008). Thus, childhood neglect is an example of one of a multitude of pre-existing factors that might both decrease verbal memory ability and influence cumulative use of ecstasy – a drug well-known to increase compassion, and closeness to self and others. Lower verbal memory has also been reported in people with no psychiatric diagnosis but with limited symptoms of schizophrenia (Hurlemann et al. 2008) or first-degree relatives diagnosed with depression (Mannie et al. 2008) or bipolar disorder (Arts et al. 2008). A previous study by the NeXT team found that depression symptoms were correlated with lifetime ecstasy use (de Win et al. 2004), a large longitudinal study in The Netherlands found that anxiety and depression in childhood were risk factors for later ecstasy use (Huizink et al. 2006), and the NeXT study design article describes psychiatric and lifestyle factors as serious potential confounders (De Win et al. 2005). Schilt and colleagues do not adjust for, or even mention, psychiatric factors in their reports.

Another methodological issue that threatens the validity of all the published studies on the NeXT non-randomized cross-sectional sample (Jager et al. 2007; de Win et al. 2008; Schilt et al. 2008) is a possible sampling bias: subjects were recruited at different locations and settings and encouraged to recruit their friends. Confounding due to lifestyle differences, for instance regular attendance at dance parties (raves), cannot be dismissed. Moreover, heavy ecstasy users may have volunteered for a study entitled the ‘Netherlands XCT Toxicity’ study in order to confirm the existence of perceived ecstasy-related problems. Ecstasy users ‘primed’ to think that ecstasy is toxic performed worse than non-primed ecstasy users specifically on a verbal memory test (Cole et al. 2006). Since all the NeXT studies have recruited subjects from different locations with different methods, such as through a webpage of the project and snowballing, a serious selection bias cannot be excluded.

To study cognitive dysfunction in socially stigmatized groups is notoriously difficult (Gould, 1996). Looking over 20 years of repeated studies looking for brain damage in ecstasy users, we see very few consistent findings and little consideration of pre-existing psychiatric factors that may influence young people to repeatedly risk criminal penalties in order to experience MDMA-mediated feelings of love and empathy. As Bedi & Redman acknowledge, cognitive functioning in ecstasy users is a highly debated topic and the data are inconclusive with no clear pattern of specific deficits.

In both articles under discussion, the authors speculate that any cognitive effects of ecstasy use could increase with age; however, there is no empirical basis for this often repeated warning. Most longitudinal studies of ecstasy users have found no change in cognitive function with continued ecstasy use, suggesting that any cognitive deficits may have been pre-existing (Gouzoulis-Mayfrank & Daumann, 2006). Cross-sectional studies in moderate ecstasy users rarely find any effects (Gouzoulis-Mayfrank & Daumann, 2006). Studies in non-randomized samples of heavy ecstasy users have little relevance for clinical studies involving infrequent doses of pharmacologically pure MDMA.

Given the accumulating evidence, it appears that ecstasy use is a comparatively minor overall problem for society compared to alcohol and many other drugs (Nutt et al. 2007). Decades with studies of cognitive ability in ecstasy users continue to reveal small and inconsistent results and should therefore be interpreted with caution.