The purpose of this comment is to advocate for the development of a synthetic validity database along the lines described by Johnson et al. (2010), not only because of its practical utility but also because of its potential contribution to the science of industrial and organizational (I-O) psychology generally. Specifically, one of the core scientific pursuits of psychology is the understanding of individual differences (IDs). Knowledge of the context in which specific IDs matter (i.e., in terms of level of behaviors exhibited) can provide information about the nature of those IDs. For example, if the degree to which a job requires continuous learning of symbolic material is related to the predictive validity of general cognitive ability (g; i.e., learning context moderates the g–performance relationship), it is likely that g has something to do with the acquisition of new knowledge. Alternatively, or additionally, if the degree to which a job requires making decisions among large numbers of multi-attribute options is related to the validity of g as a predictor of job performance, it implies that g involves information processing capacity (and perhaps it is via this mechanism that g is related to learning). That is, evidence of the moderator effects of context (i.e., conditions on which jobs or tasks vary) on ID–performance relationships provides information about the nature of the ID construct, as well as whether the ID construct is likely to be predictive of performance in any particular context. This is another reason to develop a database that indicates what context variables predict the validities of ID constructs.

After reviewing the article by Johnson et al. (2010), I began to ask myself and other industrial and organizational (I-O) psychologists what interest groups might be impacted by a Society for Industrial and Organizational Psychology (SIOP)-sponsored synthetic validity database and its resulting mechanical behavior and performance predictions. Who are the various stakeholders and what might be the positive or negative outcomes caused by this “disruptive technology” that could cause “creative destruction” in the I-O psychology profession? Among the consumers and producers of performance prediction, who might gain and who might be creatively destroyed? From these questions and the subsequent conversations, I identified the following categories and possibilities, but I am sure there are others that I have not anticipated.

In their comprehensive apologetic treatment of synthetic validity, Johnson et al. (2010) echo Hough (2001), advocating development of a central synthetic validation database, which would serve as a repository of validity information to support future synthetic validation efforts. They offer two potential approaches for developing such a database. The first entails the conduct of “a large-scale study in which tests are administered to and performance ratings gathered on incumbents in a large number of jobs in a variety of organizations.” The authors consider this approach to be “ideal but impractical,” largely because of the scope and cost of the data collection required and the resulting investment required by any sponsoring organization. The second entails the conduct of multiple local studies to generate empirical estimates of relationships between measures of various predictor constructs and a standardized set of job components. Johnson et al. consider this approach more practical, citing Meyer, Dalal, and Bonaccio (2009) as a benchmark example.

We expected that the commentary process would provide valuable feedback to improve our ideas and identify potential obstacles, and we were not disappointed. The commentaries were generally in agreement that synthetic validity is a good idea, although we also received a fair amount of suggestions for improvements, conditional or tempered praise, and explicitly critical comments. We address the concerns that were raised and conclude that we should move forward with developing a large-scale synthetic validity database, incorporating the suggestions of some of the commentators.