Dear Editor,
Adherence to RCT reporting guidelines for autism interventions: response to Chien
We welcome the recent paper ‘Effectiveness, durability, and clinical correlates of the PEERS social skills intervention in young adults with autism spectrum disorder: the first evidence outside North America’ (Chien et al., in PSM, 12th July Reference Chien, Tsai, Chen, Yang, Gau, Soong and Chiu2021). The study aimed to evaluate autism interventions, an area that is under-researched and highly important, and this report provides a helpful opportunity to raise two issues of importance to the field.
First, we were interested in the paper's lack of full adherence to Consort Guidelines – the internationally agreed consensus of the standardised guidance for reporting randomized controlled trials (Schulz, Altman, & Moher, Reference Schulz, Altman and Moher2010) – as well as the small size and questionable quality of the trial. We went through the published Consort guidelines and noted that approximately half of the Consort 2010 items had either not been reported or had only been partially addressed. We especially noted the following: there was no mention of a study protocol, no mention of trial registration, no sample size calculation, no primary outcome specified, no a priori mention of minimum clinically important effect size, no mention of the clinical importance of the raw effect size achieved, no mention of the confidence level (or significance level) used in the statistical methods section, no a priori statement concerning the analysis concept(s) (e.g. ‘as treated’, ‘ITT’, ‘per protocol’) and incomplete randomization information. Of the four Consort randomization items, three were not reported at all, and one was only partially described (stratification was mentioned, but there was no mention of block size). The sample size of the study was too small to be considered sufficiently reliable as a definitive study for purposes of generalization, and development as a pilot or feasibility study would have been more prudent, given the paucity of information relating to the intervention and the outcome, particularly in relation to the variability of the outcome measures. We note that previous trials of this intervention were even smaller in size (Chien et al., ibid). There was also no mention of any adjustment made to the familywise error rate due to multiple hypothesis testing. Given these issues, we consider that the utility of this study would have been maximised if presented as a preliminary study, whose greatest benefit would have been the generation of information to be used in the development of a larger definitive study. We consider that the large-scale omission of standard reporting procedures, together with the small size of the study and inattention to effect size interpretation, undermines the case for acceptance as a definitive study.
Second, the multiplicity of outcomes described in this study reflects a wider difficulty for autism intervention trialists and its publication is helpful therefore in drawing attention to this. Unlike in other fields of medicine, there would appear to be a lack of consensus in the autism field as to which primary outcome to focus on (Brugha, Doos, Tempier, Einfeld, & Howlin, Reference Brugha, Doos, Tempier, Einfeld and Howlin2015). Syntheses of trial findings therefore will be heterogeneous and less likely to have an impact on treatment policy. Major funders are less likely to support a proposal for an adequately powered trial that lacks a clear focus. Such trials are strikingly rare in adult autism but are now being funded (NIHR, 2021a,b). The present study on social skills has been one of the outcomes that autism intervention developers have suggested as a meaningful target. There is also a need to identify and measure reliably an outcome with similar meaning across individuals (Lee et al., Reference Lee, Shogren, Segal, Pezzimenti, Aleman-Tovar and Lounds Taylor2021). Therefore, this study and our concerns about its reporting should act as a call to autism researchers working in collaboration with people with autism to engage in a joint effort to seek greater consensus on intervention outcome prioritisation, and to major funders to make clear their interest in adequately designed and powered trials in adult autism.
Conflict of interest
Both authors are members of the Editorial Board of Psychological Medicine. The authors are corresponding here only in their capacity as independent researchers.
