In their recent article on ethnic group differences on Raven’s well-known Progressive Matrices test, te Nijenhuis et al. (Reference te Nijenhuis, Batterjee, Van Den Hoek, Allik and Sukhanovskiy2017) refer to criticisms that I levelled against work on ethnic group differences in cognitive ability (IQ) tests. The criticism that they (indirectly) referred to concerned studies (Templer & Arikawa, Reference Templer and Arikawa2006; Kanazawa, Reference Kanazawa2008) that denied several well established facts. Specifically, several methods in these studies denied the Flynn Effect, climate change since the last ice age, global migration since the 1500s and the earth not being flat (Wicherts et al., Reference Wicherts, Borsboom and Dolan2010), to mention some of these studies’ more egregious errors. But my critique of those earlier studies on national IQ is hardly relevant for the psychometric analyses used by te Nijenhuis et al. to study why groups show varying performance on the Raven’s test. Unfortunately, their analyses too, are rather lethally flawed.
Wicherts and Johnson (Reference Wicherts and Johnson2009) already highlighted psychometric problems with Jensen’s (Reference Jensen1998) method of correlated vectors applied to dichotomous (correctly or incorrectly scored) item scores. Notably, the item statistics used in this method, like the item-total correlation, have long been known (e.g. Ferguson, Reference Ferguson1941) to be dependent on the ability of the group. Another problem is that the relations between item statistics are inherently and complexly non-linear because of the statistics being bounded by 1. After noting that te Nijenhuis and colleagues had ignored Wicherts and Johnson (Reference Wicherts and Johnson2009) in earlier works using the method of correlated vectors, I wrote a more elaborate critique (Wicherts, Reference Wicherts2017). I offered the latter manuscript to te Nijenhuis et al. when I reviewed an earlier version of their current paper for a psychometric journal, but they declined to read it as it was still under review at the time. Their manuscript is now published in the Journal of Biosocial Science but continues to ignore the psychometric problems with the method of correlated vectors.
Wicherts (Reference Wicherts2017) extensively argued that the method of correlated vectors is patently incapable of highlighting whether or not g is indeed the source of group differences on the Raven’s test. Even in the unlikely scenario that Raven’s items measure only g and do so perfectly and in the same manner in both groups (a hypothetical case in which Spearman’s hypothesis is true), the method can yield basically any correlation, even negative ones. Moreover, Wicherts (Reference Wicherts2017) showed empirically that the method is incapable of detecting instances in which Spearman’s hypothesis is not true because studied items measure different traits across groups. Crucially, in these instances, the method can yield correlations between the vectors that far exceed the correlation of 0.44 found by te Nijenhuis et al. (Reference te Nijenhuis, Batterjee, Van Den Hoek, Allik and Sukhanovskiy2017). This result casts strong doubt on their conclusion that such a correlation corroborates that the differences can be (more or less) attributed to g. The method of correlated vectors applied to item scores lacks diagnostic value in studying whether or not group difference are caused by g (Wicherts, Reference Wicherts2017) and so I suggest the authors apply established psychometric methods to their interesting data.
Open access
