The evidence base and readability of Freeman et al on virtual reality for treating delusions

I commend Freeman et al ¹ for their innovative research using virtual reality in the treatment of persecutory delusions. The rather ‘soft’ finding – probably valid, but hardly surprising – is: ‘Cognitive therapy using virtual reality could prove highly effective in treating delusions’.

I have concerns about the evidence base of the study, that is the design, the data, the sample and the statistical methodology. Each of these concerns interrogates the validity and reproducibility of the study. ^2,3

First, the sample size is extremely small – 30 participants. The consequences of this include overestimates of effect size and low reproducibility of results.

Second, 27 of the 30 participants were unemployed. There is little point to any research if one cannot extrapolate from one's sample to some broader reference population. For this, the sample should mimic the population in important ways. However, there is no discussion about the sampling, or the reference population to which extrapolation might be extended.

Third, neither the patients nor the researchers were masked to the randomisation allocations. This, surely, is a fundamental flaw of the experiment. A double-blind experiment should be used to ensure impartiality, and avoid bias, such as, for example, the Hawthorne effect. ⁴

Fourth, the main outcomes – comparing the delusional conviction of the two groups at the beginning and end of testing, as well as their distress – were tested using ANCOVA. But in the results section the authors report: ‘For ratings of conviction in paranoia, a gradual reduction across the scenarios for the threat belief testing group can be seen, whereas the conviction scores remain stable in the exposure group’ (p. ). This suggests that the two groups diverge over time, having different slopes, rather than the assumed homogeneous slopes in the ANCOVA model.

Fifth, the term ‘repeated measures mixed model’ covers a wide range of possible models, and leads one to expect a single model incorporating the repeated measures and random effects, not ten models as are presented in the online supplement. Further, none of the models is clearly articulated in mathematical form.

Sixth, there are no graphs to display the data or statistical results. Tay et al ⁵ propose the use of graphical descriptives to enhance research rigour, especially in psychology.

It appears that the article is written on two levels. The introduction and method sections, describing participants, design and virtual reality, are clear and lucid. By contrast, the evidence base of the article, discussing the data, models, analysis and results, is almost unintelligible. Further, the small sample size, sampling bias, lack of randomisation masking, lack of model specification and lack of statistical graphics, seriously undermine the study.

The phrase ‘evidence-based research’ has become popular in psychology. Thus, it is incumbent on readers, authors and journal editors to ‘raise the bar’ and demand higher standards of the evidence base of research studies.