Concerns regarding an evaluation of MTFC-A for adolescents in English care

We are writing to highlight concerns regarding conclusions offered by Green et al in their evaluation of Multidimensional Treatment Foster Care for Adolescents (MTFC-A) relative to usual care for at-risk youth in English foster care. ¹ We commend the authors for undertaking an independent review of MTFC-A. However, we offer some observations to help contextualise the efficacy of the evaluation with respect to the primary conclusion that MTFC-A did not result in better outcomes than usual care.

Green et al’s evaluation employed a two-arm, single-blinded (assessor) randomised controlled trial embedded within an observational quasi-experimental case-control study. An intent-to-treat (ITT) analysis was employed specific to the MTFC-A versus usual care comparison. The authors state that the study was intended to be powered at β = 0.80 to detect half a standard deviation difference between ITT and usual care (with a target n of 130), and was powered β = 0.95 to detect the same effect between ITT and usual care in the quasi-experimental study (with a target n of 90). However, the target allocation for the trial was not met. The trial randomly allocated only 34 participants (n = 20 MTFC-A and n = 14 usual care). Based on these numbers, we estimate the study was actually powered at β = 0.29 in the ITT analysis to detect half a standard deviation difference between conditions assuming equal variances, and at β = 0.28 assuming unequal variances.

Substantive conclusions therefore seem to be based on a substantially underpowered design (as far as we can tell from the detail presented in the original manuscript). Further, the quasi-experimental arm was described as a case-control design. However, it was not a matched case-control design. This is evident from multiple baseline differences between groups, some of which remained after an intensive set of propensity-score weights was applied and after elimination of cases with probability of assignment to MTFC-A above 0.95 and below 0.05. Depending on the distribution of assignment probabilities, this may have resulted in relatively limited ‘data trimming’ in order to attain desired allocation probabilities near 0.50. The observed differences included not only age but also the primary outcome scores.

Notwithstanding concerns regarding statistical power for the trial, the authors reported intervention by baseline risk interactions in the only adequately powered arm of the study (see Table 5). Given prior demonstration of MTFC-A intervention by baseline risk interactions, ² these results may have been more appropriately presented as a hypothesised replication. Statistical power is also a concern for the reported analyses of offending; β = 0.034 to detect the observed ITT odds ratio of 1.24 using an allocation of 20 and 14 cases, and β = 0.031 in the quasi-experimental arm to detect the observed ITT odds ratio of 1.07 with 93 and 92 cases. Interpretation of effects should therefore be treated with caution.

We raise one additional point of clarification regarding prior MTFC-A implementations. The authors state that the context of intervention in the UK differs significantly from that in the originating US studies, since ‘these were focused on convicted delinquent youth where the alternative [to MTFC-A] was incarceration’, thereby concluding that the ‘control condition in the US studies approximated […] to juvenile custody’. Actually, similar to the usual care condition in the Green et al study, the standard control condition in US MTFC-A studies is group care, ³ not incarceration.

We offer these points by way of lending interpretation to the efficacy of Green et al’s results and to suggest caution in accepting the conclusion that MTFC-A may not result in better outcomes than usual care among at-risk adolescents in English care.