In their response to my criticism of their recent article in Journal of Biosocial Science (te Nijenhuis et al., Reference te Nijenhuis, Batterjee, van den Hoek, Allik and Sukhanovskiy2017), te Nijenhuis and van den Hoek (Reference te Nijenhuis and van den Hoek2018; hereafter TeNvdH) raise four points, none of which concerns my main point that the method of correlated vectors (MCV) applied to item-level data represents a flawed method. Their response highlights their continued reluctance to confront the psychometric problems with MCV at the item level. I will discuss their four points in turn.
First, TeNvdH indicate that I used MCV at the item level in my own work with colleagues (Kan et al., Reference Kan, Wicherts, Dolan and Van Der Maas2013). But as anyone could readily glean from the abstract, Kan et al.’s (Reference Kan, Wicherts, Dolan and Van Der Maas2013) application of MCV revolved around subtests rather than dichotomous item scores. TeNvdH appear to confuse (approximately) normal data from subtests as modelled in the context of linear factor analysis with data that are dichotomous (i.e. scored 0 or 1) and modelled as appropriately non-linear in the context of item response theory. I had hoped that TeNvdH would finally discuss my article on MCV applied to the item level (Wicherts, Reference Wicherts2017), in which I certainly applied MCV to item-level data to illustrate its problems empirically. The empirical application of MCV at the item level in Wicherts (Reference Wicherts2017) was inspired by te Nijenhuis’ scepticism of data simulations, so I am disappointed that TeNvdH fail to discuss the nonsensical results I found with this method in real data comparing entirely unrelated measures of intelligence, anxiety, anger and the big five personality traits.
Second, TeNvdH refer to meta-analytic corrections developed by Hunter and Schmidt (Reference Hunter and Schmidt2004), and rightly indicated that I did not discuss these corrections. The reason again is that my main criticism (Wicherts, Reference Wicherts2017) is levelled against the use of MCV at the item level, for which the Schmidt and Hunter corrections were not developed. TeNvdH appear of the opinion that it is fine to apply a statistical method developed for one type of data (continuous) to a completely different type of data (dichotomous) without first trying to understand (or even discuss) its statistical complexities. Crucially, my psychometric argument was based on a scenario in which the items measured g perfectly, the loadings were not subject to any error, the loadings covered the entire range and there was no sampling error. In these cases, corrections for sampling error, reliabilities of the vectors, imperfect measures of g, and restriction of range in the loadings are completely futile. These corrections do not magically unbend the non-linearity implicit in MCV relations at the item level, even with perfect data.
Third, TeNvdH refer to an early criticism of MCV at the scale level by Schönemann (Reference Schönemann1997) that they consider to be ‘eerily similar’ to my criticism of MCV at the item level. Again, Schöneman’s discussion related to linear data that has little bearing on my point that MCV yields complex non-linear relations when applied to dichotomous items. Such relations might certainly also create negative relations, as I clearly indicated in my 2017 article: ‘the variation in MCV correlations is due to the complex non-linearity that is caused by the restricted range of the relevant CTT item statistics (that go from − 1 to 1) and their inherent group-specificity. That MCV correlations based on item scores can be very low (negative even) in the population even if Spearman’s hypothesis were true highlights that MCV lacks sensitivity: Even if Spearman were right and the test measures g and nothing else, MCV correlations close to 1 will only occur in rare cases’ (Wicherts, Reference Wicherts2017, p. 30). I provided an Excel file as supplementary file that allows anyone to assess this readily. The vectors can correlate negatively and positively in cases that align perfectly with g theory, and they can positively correlate highly even when the data are nonsensical in practice (Wicherts, Reference Wicherts2017). The correlations can be negative when one uses loadings that are estimated appropriately in a latent variable framework as in the example described by TeNvdH.
Fourth, TeNvdH suggest that I misused my position as reviewer by referring to an unpublished paper when I reviewed their work. They perhaps forgot that this particular paper (Wicherts, Reference Wicherts2017) spent a lot of time in the peer review pipeline at Intelligence because it required summary data that te Nijenhuis et al. had not reported in their paper. My paper was considerably delayed because it took te Nijenhuis and colleagues a full year to share the needed data to complete my analyses. In earlier correspondence, te Nijenhuis explicitly refused to read an earlier version of Wicherts (Reference Wicherts2017) because it had not yet been published in a peer-reviewed journal, and TeNvdH now repeat this argument as though this is legal case wherin certain evidence is not allowed to be considered. The paper has been published in the peer-reviewed literature in late 2016, so TeNvdH can now safely read it. Besides, they could have also considered my earlier peer-reviewed work (Wicherts & Johnson, Reference Wicherts and Johnson2009) that made a very similar case against the use of MCV at the item level.
Open access
