Methodological discrepancies in the update of a meta-analysis

Leichsenring & Rabung ¹ reported that long-term psychodynamic psychotherapy (LTPP) is superior to less intensive forms of psychotherapy in complex mental disorders. Based on 10 trials, they found an overall effect size (ES) of 0.55. We found several methodological discrepancies in their study.

First, it seems surprising that the Q-test indicated no significant unexplained variance, as the between-group effect size of one of the primary studies ² (ES = 1.76) is quite outstanding in Fig. 2. To shed light on this issue, we recalculated the overall effect size using a random effects meta-analysis based on the values from Fig. 2. Our meta-analysis replicated Leichsenring & Rabung ¹ in the main. In contrast to Leichsenring & Rabung however, we found a significant unexplained variance (Q = 25.33, d.f. = 9, P = 0.003) and a larger overall confidence interval of 0.29–0.82 (in contrast to 0.41–0.67 as reported by Leichsenring & Rabung). Additionally, computing an outlier analysis, a significant outlying study effect size was found (P<0.001). Including the moderator considering the impact of this study yields an effect size of 0.44 (95% CI 0.27–0.61, P<0.001). The moderator effect, interpreted as the difference between the effect of the outlying study and the grand mean, was 1.32 (95% CI 0.57–2.07, P<0.001). After removing the outlying study, there was no significant unexplained variance (Q = 11.56, d.f. = 8, P = 0.172).

Second, we calculated the fail-safe N according to Rosenthal; ³ 16 non-published studies with an effect size of 0 had to be included in the analysis to change the results of the meta-analysis (ES = 0.44) from significant to non-significant (ES<0.16). As 16 is below 55 (5K + 10), the effect cannot be regarded as robust.

Last, to gain better insight into the interpretation of the overall effect size as small, medium or large, we calculated a Bayesian meta-analysis following Higgins et al's methodology. ⁴ The Bayesian analysis essentially replicated the findings of our random effects meta-analysis. In addition, we found the probability of the overall effect size to be small (ES<0.5) at 72.5%. Thus, in contrast to Leichsenring & Rabung, ¹ we found that the overall effect size was small rather than medium or large.

Therefore, we would greatly appreciate caution against a conclusion that the overall effectiveness of LTPP for treating complex mental disorders should now be considered as definitely proven.