Tett, Hundley, and Christiansen (2017) describe two main sources of uncertainty that are usually reported in a meta-analysis: uncertainty about the value of the underlying mean correlation (which they describe as SE(rxy)) and uncertainty about the individual values of rho that arise from the random-effects variance (the square root of which they describe as SD(rho)). They proceed to recommend descriptions of small, medium, and large values of each uncertainty that meta-analysts should report and consider for interpretation. However, there exists a simpler solution to expressing and interpreting such uncertainty.

In contrast to the view that survey key driver analysis (SKDA) is a misused and blind empirical process, we suggest it is a reasonable, hypothesis-driven approach that builds on cumulative knowledge drawn from both the literature and practice, and requires reasoned judgment about the relationships of individual items to the constructs they represent and the criteria of interest. The logic of key driver analysis in applied settings is no different than the logic of its application in fundamental research regarding employee attitudes (e.g., Dalal, Baysinger, Brummel, & LeBreton, 2012). However, there are important survey design and analysis issues with respect to how key driver analyses are best conducted. Just some of these are discussed below.

The rate of technological change is quickly outpacing today's methods for understanding how new advancements are applied within industrial-organizational (I-O) psychology. To further complicate matters, specific attempts to explain observed differences or measurement equivalence across devices are often atheoretical or fail to explain why a technology should (or should not) affect the measured construct. As a typical example, understanding how technology influences construct measurement in personnel testing and assessment is critical for explaining or predicting other practical issues such as accessibility, security, and scoring. Therefore, theory development is needed to guide research hypotheses, manage expectations, and address these issues at this intersection of technology and I-O psychology. This article is an extension of a Society for Industrial and Organizational Psychology (SIOP) 2016 panel session, which (re)introduces conceptual frameworks that can help explain how and why measurement equivalence or nonequivalence is observed in the context of selection and assessment. We outline three potential conceptual frameworks as candidates for further research, evaluation, and application, and argue for a similar conceptual approach for explaining how technology may influence other psychological phenomena.

Jones, Arena, Nittrouer, Alonso, and Lindsey (2017) make the case that discrimination is multifaceted and can be identified along several continua. They also emphasize the role that every individual may play in the propagation of discrimination. As such, they make note of several interventions from bystanders and allies to combat subtle discrimination. Although we agree that subtle discrimination causes harm and that interventions targeted at such discrimination are necessary, we propose some additional considerations for the science and practice of subtle discrimination reduction. Specifically, we discuss the limitations of focusing on subtle discrimination at the individual level, the ambiguous nature of intentionality, the view of subtle discrimination as a manifestation of a hostile environment that falls under the broader umbrella of negative interpersonal treatment, and the emphasis placed by Jones et al. on the potential for organizational level interventions by proposing several considerations for tackling a climate of negative interpersonal treatment.

Modern technology and technological advances offer a variety of benefits and challenges for assessment, data collection, communication, and other research- and practice-related endeavors. The focal article written by Morelli, Potosky, Arthur, and Tippins (2017) offers a segue into discussions about some of these issues. Although the authors offer some unique insights, we believe their view is incomplete, as it is potentially limited by their focus on testing and assessment. Below, we outline a few key points we hope will advance the conversation. Our commentary is largely grounded in the field of human–computer interaction (HCI), which is an interdisciplinary field that integrates expertise from computer science, psychology (and other behavioral sciences), and many other fields. Whereas psychology tends to place the human user at the forefront of discussions concerning technology, HCI expands beyond just the user's psychology, focusing on the design of interfaces that allow users to interact with computing technology in new ways (Card, Moran, & Newell, 1983).

The implicit philosophy for how research and practice in industrial and organizational (I-O) psychology has pursued inferences about our field's core phenomena has largely been based on a nomothetic, variable-based, and aggregate/“large-sample” ideal. As Tett, Hundley, and Christiansen (2017) expertly highlight, there are more insightful means for drawing inferences about the nature of such aggregate relationships based on meta-analytic techniques than the current practice in the organizational sciences. However, the motivating force behind our commentary has less to do with the issues raised by Tett et al. (2017) concerning the practice of using meta-analysis for purposes of validity generalization and more to do with the practice of using meta-analysis for purposes of scientific inference. Between-person philosophies in which the end-goal is to identify general conclusions that apply to the aggregate (cf. Hanges & Wang, 2012) have historically guided our scientific inferences and have supported the proliferation of meta-analytic techniques (including what Tett et al. describe as tertiary analyses based on such findings). These philosophies have led to a dearth of understanding at the within-person and social system levels—the levels at which most of our meaningful phenomena exist (e.g., Hamaker, 2012; Von Bertalanffy, 1950). Learning, performing, decision making, communicating, sense-making, feeling/expressing emotion: These are the concepts that drive the lived experiences of individuals both inside and outside of the workplace, and all are vulnerable to being misunderstood or misinterpreted by focusing only on aggregate evidence at the between-person level. Consequently, we wish to first supplement Tett et al.’s recommendations for drawing generalizability inferences in meta-analysis and suggest a “pre-emptive” question (i.e., Question 0) to the list of four they advance in their focal article.

Compared with more turbulent times in history, some might argue workplace discrimination has seen a downward trend. Others would contend that workplace discrimination has “just gone underground” and become more covert (Herring, 2002, p. 13). Either way, not-so-distant historical events such as the landmark Texaco case in 1996 and the Ford Motor case in 2000 remind us that discrimination demands our attention. Calls for research on interventions have surfaced (Becker, Zawadzki, & Shields, 2014), and proposals such as legal reforms, implicit bias training (Bartlett, 2009), and experiential learning workshops have answered (Cundiff, Zawadzki, Danube, & Shields, 2014). The focal article (Jones, Arena, Nittrouer, Alonso, & Lindsey, 2017) contributes to this discussion as it turns our attention to the construct space of discrimination and presents a framework for organizing its facets and forms. Without a doubt, a framework that lends itself to the integration of the many forms of discrimination is long overdue.

In their focal article, Cucina, Walmsley, Gast, Martin, and Curtin (2017) raise a number of concerns about survey key driver analysis (SKDA). Although many are valid, we think their critique ultimately goes too far and risks throwing the baby out with the bathwater. We have found that SKDA can be an effective way to generate insight and action when analysis is done correctly and results are used appropriately. In this response article, we first provide data that counters Cucina et al.’s findings. Then we describe the approach we take to survey key driver analytics and application. In doing so, we hope to address some of the core concerns raised in the focal article and identify ways for scientists and practitioners to conduct SKDA in a responsible way.

Over the years, many in the field of organizational psychology have claimed that meta-analytic tests for moderators provide evidence for validity generalization (Schmidt & Hunter, 1977), a term first used in the middle of the last century (Mosier, 1950). In response, Tett, Hundley, and Christiansen (2017) advise caution when it comes to our inclination toward generalizing findings across workplaces/domains and urge precision in attaching meaning to the statistic we are generalizing. Their focal article was insightful and offers important recommendations for researchers regarding certain statistical indicators of unexplained variability, such as SDρ. In this commentary, we would like to make a different point about SDρ, namely that it, and other statistics based on residual variance, will be deflated due to the lack of variance in moderators. It is this lack of between-study variance, as much as anything else, that leads to misguided conclusions about validity generalization.

In their focal article, Cucina, Walmsley, Gast, Martin, and Curtin (2017) raise an important and timely issue, but go too far in arguing that practitioners should abandon survey key driver analysis (SKDA) altogether. There are many valuable benefits to using SKDA, and rather than discontinuing the practice entirely, we suggest that a better approach is to focus on ways to make it even more valuable to organizations. It is true that this area is in need of further research, but instead of asking whether we are driving down the right road, perhaps the right question is, “Are we there yet?”

Morelli, Potosky, Arthur, and Tippins (2017) make a timely and appropriate call for authors to create conceptual models of technology in industrial-organizational (I-O) psychology. We agree with their call, but we believe that Morelli et al. overlooked the contributions of related fields that conduct research on technology in the workplace that are already consistent with their call. For this reason, we briefly detail other fields that commonly study the dynamics of technology and its influence on the workplace, followed by a discussion regarding the place of I-O psychology in the broader scheme of technology research. This discussion can aid future authors in conceptualizing appropriate contributions to the study of technology in I-O psychology as well as identifying whether these contributions benefit other fields. Perhaps more importantly, this discussion can help identify where I-O psychology fits in the broader scheme of technology research and which associated fields may be most readily available to aid in the creation of new models—two questions that currently seem unanswered.

Employee engagement has long been an instrumental component of human capital strategies and continues to dominate the conversation about how high-performing organizations attract and retain their best talent. Engagement is a construct of component parts, however, and we believe there is still much to be learned about engagement by taking an in-depth look at those components. This article examines employee motivation as a core element of engagement, including its antecedents and outcomes, the types of motivation and the dynamics between them, and the ways organizations can foster and harness motivation for improved engagement. Our research identifies a large and consistent motivation gap, such that employee intrinsic motivation is consistently higher than extrinsic motivation. This gap signals that investments in engagement can yield a higher return if strategically focused on motivation, and so we offer recommendations regarding how to close this gap via intrinsically and extrinsically motivating work structures and environments. The goal is to create a new dialogue around engagement and encourage organizations to break it down in order to understand it more fully.

Tett, Hundley, and Christiansen (2017) raise an important issue related to meta-analysis and our frequent overinterpretation of point estimates to the diminishment of variability of the estimate. We view this as analogous to the situation in which weather forecasters communicate the likely track of hurricanes. Such predictions involve point estimates of where the center of the storm is likely to be at some future time. These point estimates can be connected to identify the most likely path of the storm. In addition to these point estimates, however, forecasters caution that we should also attend to the “cone of uncertainty.” That is, we should not focus exclusively on the point estimate to the exclusion of the errors of prediction.

In industrial-organizational (I-O) psychology, much like in the organizational sciences more broadly (Hambrick, 2007), we have a bit of an addiction to theoretical models. It is commonly assumed that developing new theory is the most valuable way to solve pressing research problems and to drive our field forward (Mathieu, 2016). However, this assumption is untested, and there is growing awareness among organizational scientists that this hardline approach, which is unusual among both the natural sciences and other social sciences, may even be damaging the reputation and influence of our field (Antonakis, 2017; Ones, Kaiser, Chamorro-Premuzic, & Svensson, 2017). As Hambrick (2007) describes, the requirement for theory first “takes an array of subtle, but significant, tolls on our field” (p. 1348). As we will describe in this article, Morelli, Potosky, Arthur, and Tippins’ (2017) suggestions, if taken at face value, will likely create such tolls by encouraging the creation of new theories of dubious value. To be clear, we agree with Morelli et al. that better theory is needed for technology's impact on I-O psychology broadly and talent assessment in particular. We disagree, however, that creating new technology theories using the approaches that I-O psychology typically employs is likely to accomplish this broader goal. Rather, it will ultimately only isolate research on I-O technologies even further from both mainstream I-O research and technology research. Given that we are already quite isolated, this would be a disastrous path.

Cucina, Walmsley, Gast, Martin, and Curtin (2017) raise an important issue in evaluating whether our current approaches for key driver analysis on employee opinion survey data are indeed best practices. As has been argued elsewhere (Putka & Oswald, 2016; Scherbaum, Putka, Naidoo, & Youssefnia, 2010), there is and can be misalignment between current and best practices. We agree with Cucina et al. that our field should engage in larger discussion of these issues. That discussion is critical, as industrial and organizational (I-O) psychologists are competing with those outside our field who have either little knowledge of best practices in data analysis (but who have been empowered by technology that automates the analysis) or little knowledge of psychology (but a great deal of knowledge in big data analytical techniques). I-O psychologists are in the vanguard of survey data analysis (Ducey et al., 2015), and we have a responsibility to maintain the standards of our field as well as to wield our influence to guide other practitioners outside our field on sound theoretical and analytical approaches.

We wonder whether theory alone can solve problems and answer questions faced by practitioners working on the front lines of assessment innovation. Stated another way, to what degree can current theories influence the application of our work to new technology when it comes available? We are speaking as practitioners working in selection, the area in which technology has been studied most commonly in industrial-organizational (I-O) psychology (e.g., King, Ryan, Kantrowitz, Grelle, & Dainis, 2015). More specifically, we focus on the impact of mobile technology on our selection systems. We are excited for the focal article (Morelli, Potosky, Arthur, & Tippins, 2017) on theory development relative to technological advancement because much of the work we do in this area has not been discussed significantly in the literature. Our goal in this commentary is to review what we have learned about the implications of technology from our experience building and validating innovative prehire assessments.

In their critical review of survey key driver analyses (SKDA), Cucina, Walmsley, Gast, Martin, and Curtin (2017) contend that methodological issues limit the usefulness of SKDA and recommend that survey providers stop conducting SKDA until these issues can be overcome. I contend that many of these methodological issues are either overstated or able to be addressed through the proper application of the technique by a competent professional. In this commentary, I make recommendations for how SKDA should be applied so that methodological issues are addressed and the value of SKDA is maximized. Many of these recommendations were made in Lundby and Johnson (2006), who were cited by Cucina et al. but did not have much impact on their focal article.

Generalization in meta-analyses is not a dichotomous decision (typically encountered in papers using the Q test for homogeneity, the 75% rule, or null hypothesis tests). Inattention to effect size variability in meta-analyses may stem from a lack of guidelines for interpreting credibility intervals. In this commentary, we describe two methods for making practical interpretations and determining whether a particular SDρ represents a meaningful level of variability.