Comment on ‘Effects of pistachios on anthropometric indices, inflammatory markers, endothelial function and blood pressure in adults: a systematic review and meta-analysis of randomised controlled trials’

Dear Editor,

We appreciated reading the work by Asbaghi et al. ⁽¹⁾, ‘Effects of pistachios on anthropometric indices, inflammatory markers, endothelial function and blood pressure in adults: A systematic review and meta-analysis of randomised controlled trials’. Despite the author’s efforts, this study has important methodological errors that raised our concerns.

First, the authors did not register the study protocol in the International Prospective Register of Systematic Reviews (PROSPERO). The study protocol registration is a key element of a full systematic review which is considered as a reliable approach to not only enhance and maintain clarity of a successful review but also to reduce the risk of selective reporting bias⁽²⁾.

Second, the present review might not provide representative data of the available literature. The search strategy was not sensitive enough to capture three important eligible articles on the subject^(3–5). Given that a limited number of studies were included in the current study, missing three studies might significantly affect analyses results and the question about conceptualisation of pistachio in aforementioned area might not be answered in a correct way.

Moreover, according to the statement of Cochrane handbook, entering multiple comparisons from a study with a shared intervention/control group to the overall meta-analysis must be avoided⁽⁶⁾. This act double counts the participants in the shared intervention/control group and creates a unit-of-analysis error due to the unaddressed correlation between the estimated intervention effects from several comparisons. In the present systematic review and meta-analysis, the authors included redundant effect sizes and subsequently made some statistical errors. Asbaghi et al. ⁽¹⁾ included two effect sizes from studies done by Gebauer et al. ⁽⁷⁾ and West et al. ⁽⁸⁾, which examined different dosages of pistachio with a shared control group. Additionally, after careful review, we noticed that the effect size from a study done by Nieman et al. ⁽⁹⁾, which investigated the acute phase of pistachios intake on metabolic outcomes, has been pooled with other effect sizes derived from studies which examined a longer term of pistachio intake on metabolic markers; however, we believe that this study should be excluded since the mentioned study assessed the acute phase of pistachios intake on the inflammatory markers that were assessed 45 min pre-exercise and immediately post-, 1·5-h post- and 21-h post-exercise. Actually, we think that combining the effect sizes of different natures together is not a good approach and might lead to ambiguous findings.

Lastly, the authors claimed that they performed subgroup analysis to find the potential sources of between-study heterogeneity. The authors compared subgroups separately and the between-group heterogeneity was not reported. This approach is a form of ‘differences in nominal significance error’ in meta-analyses⁽¹⁰⁾. According to the Cochrane handbook, the significant effects in mean differences or the test of heterogeneity in one subgroup, while that in the other subgroup is insignificant does not show that the subgroup analysis explains the total heterogeneity⁽¹¹⁾. Since different subgroups are likely to contain different amounts of data and subsequently have different abilities to detect effects, it might be extremely misleading to simply compare the statistical significance of the results.

Altogether, the current systematic review and meta-analysis has serious methodological errors including misidentification of relevant studies, double counting of effect sizes, incomplete subgroup analysis and errors in interpreting the results; the errors might result in misleading and biased findings. Indeed, the correction of the mentioned errors may lead to changes in findings and conclusions.