Letter to the Editor: Bias in the measurement of bias. Letter regarding ‘Citation bias and selective focus on positive findings in the literature on the serotonin transporter gene (5-HTTLPR), life stress and depression’

de Vries et al. (2016) argue that discussion of the 5-HTTLPR-stress gene–environment interaction (G × E) (Caspi et al. 2003) is more positive than merited because authors often cast negative results as positive in abstracts, and negative papers with positive focus are differentially cited. These bold claims deserve careful scrutiny. Four methodological choices we highlight bias their primary results; the vast majority of papers disclose mixed and negative results in their abstracts (Table 1). Further, even if positive focus was prevalent, it could not bias meta-analytic results. The field can best move forward by ameliorating environmental measurement.

Table 1. Transparent sentences from abstracts of papers that de Vries et al. (2016) classified as having (partially) positive focus

5-HTTLPR, Serotonin transporter gene; RD, risk difference; CI, confidence interval; BDNF, brain-derived neurotrophic factor; OR, odds ratio; SNP, single nucleotide polymorphism; MDD, major depressive disorder; G × E, gene–environment interaction.

^a The primary focus of the paper was something other than 5-HTTLPR G × E for depression.

^b We debated whether to expect papers with a focus other than the 5-HTTLPR G × E (but which included it as an ancillary test) to report on this G × E in their abstracts. These include tests of G × G × E effects and one additive (G + G) × E test. To be conservative, we report results both ways. In each noted case, a paper tests a more complex effect but does not fully characterize the ancillary 5-HTTLPR G × E in the abstract.

Methodological concerns

de Vries et al. (2016) coded papers’ full results sections as positive or negative, then compared this with abstract conclusion sentences’ positivity. Four choices that lead to errant conclusions contrast decisions reflecting care not to bias results – selecting the smallest p value when both traditional and triallelic results were available, and when both adjusted and unadjusted results were available. Similarly, sensitivity analyses using the lowest p value should address several issues, but still provide ‘positive focus’ results that contradict disclosures we extracted from abstracts. We focus our comments on their primary approach, which informs their conclusions.

Averaging p values

When papers included multiple G × E p values, the authors averaged them in their primary analyses, an approach biased toward negative conclusions. For a hypothetical paper with three findings at the p = 0.001 level and one finding at the p = 0.300 level, the average of the four is non-significant by traditional standards, p = 0.076. But who would conclude such a paper was negative overall? Although the most inclusive 5-HTTLPR and life stress G × E meta-analysis took a similar approach (Sharpley et al. 2014), a bias for negative conclusions could be entirely appropriate for a meta-analysis that ultimately has positive conclusions. However, the negative bias favors the perspective of de Vries et al. (2016).

Dichotomizing averaged p values

The authors imposed a false negative/positive dichotomy on averaged p values. For example, Jenness et al. (2011) reported a significant interaction for 5-HTTLPR with family chronic stress (p = 0.02) but not with recent stressful life events (p = 0.88), leading to a negative classification by de Vries et al. (2016) (average p = 0.46). Despite disclosure of mixed findings in their abstract (Table 1), de Vries et al. (2016) labeled their work as having partially positive focus relative to ‘negative’ findings. An alternative if imperfect approach is to classify papers across at least three categories (positive, negative and mixed), then evaluate abstracts for fidelity to actual findings.

Unbiased or atheoretical?

The primary approach assumes each of the averaged p values are equally valid, an approach which runs roughshod over theory. Several papers specifically hypothesized that one of their tests was more valid than another – sensitivity testing that refines G × E research and ought to be highly cited – accordingly presented the results of both approaches, and found support for their hypothesis. Uher et al. (2011) found support for Brown & Harris's (2008) hypothesis that the childhood adversity G × E predicts persisting depression, p = 0.003, but not single-episode depression, p = 0.231 (a finding replicated elsewhere; Brown et al. 2013). These results transparently appear in their abstract (Table 1), yet the faulty assumption that these tests are equally valid leads de Vries et al. (2016) to classify Uher et al.’s (2011) and Brown et al.’s (2013) papers as negative with positive focus. Although sensitivity analyses selecting the lowest p value ought to allow for theory to favor a particular test, we identified abstract sentences disclosing results for more papers than these analyses suggest.

Evaluation of abstract conclusion sentences not full abstracts

To determine whether abstracts had overly positive focus, the authors rated the conclusion sentence(s), not the full abstract. Such a selective approach disregards an abstract's ‘gestalt’ without any rationale for doing so. Where is the evidence that researchers cite papers based on abstract conclusion sentences? In contrast to the authors’ assertions, we were able to identify very clear acknowledgement of mixed results in all but seven of the 22 abstracts characterized as having (partially) positive focus (Table 1).

Results of alternative rating approach

To estimate these decisions’ impact on the positive focus ratings of de Vries et al. (2016), we rated the 38 ‘negative’ papers. Two raters examined results, assigning negative, or mixed classifications, and examined the full abstract to determine whether negative or mixed results were not disclosed (ratings appear in online Supplementary Table S1). We extracted sentences demonstrating disclosure (Table 1). We deemed it unfair to papers with a primary focus other than the 5-HTTLPR G × E (e.g. focus on a G × G × E), but which included it as an ancillary test, to expect they report G × E results in their abstract; to be conservative, we present results both ways. Group discussion adjudicated non-matching ratings. Of these 38 ‘negative’ studies, we characterized them as 58% (n = 22) negative and 42% (n = 16) mixed. We assigned (partially) positive focus ratings to four to seven of the 22 articles that de Vries et al. (2016) characterized as having (partially) positive focus (depending on treatment of papers with a focus other than the 5-HTTLPR G × E). We conclude that the ratings of de Vries et al. (2016), which form the basis for evaluation of citation bias, are fundamentally flawed.

Sensitivity analyses using the lowest p value still do not square with evidence that authors disclosed results (Table 1): these indicate 12 have (partially) positive focus relative to our four to seven. Moreover, the authors suggested that sensitivity analyses did not markedly influence their findings (for citation bias), but their effect size of (partially) positive focus drops by 26% relative to their negative ratings (22/38 to 12/28) and by 45% relative to the population of 73 studies. Their procedures have a marked impact on estimating the prevalence of positive focus. We observe that this is not reported in their abstract.

Biased conclusions

A conclusion the authors draw in their own abstract is noteworthy: ‘discussion of the 5-HTTLPR–stress interaction is more positive than warranted’. How positive should the discussion be? Clearly, this is controversial. On the one hand, there have been two negative meta-analyses that included a small number of reports to use homogeneous designs (k = 5 and 14, respectively; Munafò et al. 2009; Risch et al. 2009), many G × E investigations are under-powered (Duncan & Keller, 2011), and we observed some questionable research practices as we read. On the other hand, inclusive meta-analyses from Karg et al. (2011) (k = 54) and Sharpley et al. (2014) (k = 81) both reach positive conclusions, with Sharpley et al. (2014) showing that the meta-analytic effect emerges across four separate design subtypes. Karg et al. (2011) show that differences between the negative meta-analyses and theirs are due to paper selection, not meta-analytic technique. Papers selected for their statistically homogeneous designs tend to have methodological flaws including retrospective lifetime stress and depression assessment (Moffitt & Caspi, 2014) leading to confounding (Uher & McGuffin, 2010). Moreover, Karg et al. (2011) show that reports with more robust measures of stress (interview and objective measures) possess a more robust meta-analytic effect, so much that others observe an almost 1:1 relationship between stress measurement quality and likelihood of at least partial G × E effect replication (Uher & McGuffin, 2010). Neither positive focus nor citation bias influences this evidence. There is at least a reasonable basis for concluding that this is a legitimate G × E effect. Thus, when papers characterize the results of the 5-HTTLPR G × E literature positively and cite positive studies, how is this ‘more positive than warranted?’

Where to go from here?

There is a much larger problem – and opportunity for progress – in G × E depression research. The unique environment contributes roughly 60% of risk to depression (Sullivan et al. 2000), but in G × E research we often fail to invest in environmental measurement. Many G × E researchers measure the environment with insufficiently valid measures (for discussion, see Monroe & Reid, 2008; Uher & McGuffin, 2010; Karg et al. 2011; Sharpley et al. 2014). But in addition, we must all more carefully conceptualize the ‘candidate environment’.

Recent work supports that chronic stress and major severity interpersonal stress were consistent unique predictors of depressive episode onset across two samples of emerging adults, whereas minor stressors were never unique predictors and non-interpersonal stressors were rarely so (Vrshek-Schallhorn et al. 2015). Early evidence indicates that these distinctions matter for G × E tests: Whereas no G × E effect emerged for minor events, consistent with expectations, an overall G × E effect between 5-HTTLPR and major events was accounted for exclusively by major interpersonal events and not non-interpersonal ones (Vrshek-Schallhorn et al. 2014). All forms and severities of stress are not created equal. As G × E research moves beyond 5-HTTLPR, we hope the field will work toward large-scale G × E research with valid, thoughtfully conceptualized environmental measures.

Conclusions

Although positive focus sometimes occurs in G × E research, as we expect it unfortunately does throughout science, through their methodological choices, the paper of de Vries et al. (2016) exemplifies bias. Four choices including classifying abstracts by only their conclusion sentence bias the primary results. Sensitivity tests do not overcome these issues. Ultimately, depression–genetics research enterprise aims to enhance prediction and intervention for depression. It is time we all renewed our ‘positive focus’ on that goal.