A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology

Abstract

Credibility and trustworthiness are the bedrock upon which any science is built. The strength of these foundations has been increasingly questioned across the sciences as instances of research misconduct and mounting concerns over the prevalence of detrimental research practices have been identified. Consequently, the purpose of this article is to encourage our scientific community to positively and proactively engage in efforts that foster a healthy and robust industrial and organizational (I-O) psychology. We begin by advancing six defining principles that we believe reflect the values of robust science and offer criteria for evaluating proposed efforts to change scientific practices. Recognizing that the contemporary scientific enterprise is a complex and diverse network of actors and institutions, we then conclude by identifying 12 stakeholders who play important roles in achieving a culture of robust science in I-O psychology and offer recommendations for actions we can take as members of these groups to strengthen our science.

The level of trust that has characterized science and its relationship with society has contributed to a period of unparalleled scientific productivity. But this trust will endure only if the scientific community devotes itself to exemplifying and transmitting the values associated with ethical scientific conduct. (Institute of Medicine, National Academy of Sciences, & National Academy of Engineering, 1995, p. v)

Industrial and organizational (I-O) psychology finds itself facing an uncomfortable dilemma. The field continues to witness tremendous growth in membership (e.g., SIOP Membership Committee, 2017); opportunities for I-O–related careers that improve societies, organizations, and workers’ lives in meaningful ways have never been greater (e.g., Boutelle, 2014); and the evolution of theory, research, and methodologies in our field reflects a maturing and increasingly sophisticated science (Kozlowski, Chen, & Salas, 2017). Alongside these promising and laudable achievements, however, questions and anxieties about our research practices and the broader system in which our science exists have also continued to grow (Banks, O'Boyle, et al., 2016; Banks, Rogelberg, Woznyj, Landis, & Rupp, 2016; Bedeian, Taylor, & Miller, 2010; Bosco, Aguinis, Field, Pierce, & Dalton, 2016; Byington & Felps, 2017; Franco, Malhotra, & Simonovits, 2014; Hambrick, 2007; Honig, Lampel, Siegel, & Drnevich, 2014; Kepes & McDaniel, 2013; O'Boyle, Banks, & Gonzalez-Mule, 2017; Simmons, Nelson, & Simonshon, 2011). The message expressed is a consistent and unpleasant one: If left unchecked, issues with how our scientific enterprise functions could pose significant threats to the credibility of our science and to the evidence-based practices it supports.

We believe it is important to be in front of any such gathering storm (though it could be argued that this storm has already gathered) that could undermine the credibility of our discipline and hold significant consequences for both our academic and practitioner members. In light of these issues, the purpose of this focal article is to stimulate an open dialogue within our community about how to promote and sustain a healthy and credible science in I-O psychology. We initiate this conversation by offering a vision, well-aligned with similar calls made from other sources, for what we believe defines and characterizes a robust science. It is our hope that the commentaries received from our community will help shape this vision into an agreed upon set of core beliefs that are endorsed by our professional membership and reinforced by their actions. The second goal of this article is to facilitate discussion of the positive scientific behaviors that should be taken to achieve these aspirational criteria. Recognizing that our scientific enterprise is an interconnected network of stakeholders, systems, and institutions, we approach this topic by describing key players in our scientific ecosystem and opportunities these entities have for facilitating robust science. It is our hope that input on this topic identifies responsibilities, points of leverage, and—most importantly—actionable steps that can be pursued using available resources within our profession. Striving to ensure I-O psychology operates and is perceived as a robust and credible science is important, community driven, and communicates that we are committed to the generation of trustworthy scientific knowledge and services that benefit society. Before delving into these discussions, we first provide context on the current state of affairs on scientific integrity and the impetus for engaging our community in matters related to robust science.

Research Integrity, I-O Psychology, and the Modern Scientific Landscape

In a recent comprehensive review and report on the integrity of science at large, the National Academies of Sciences, Engineering, and Medicine (2017) described three types of activity that jeopardize the credibility of scientific knowledge: research misconduct, detrimental research practices, and other misconduct. The report notes that both research misconduct (fabrication, falsification, or plagiarism during any aspect of the research process) and other forms of misconduct (e.g., gross negligence of professional activities or regulations, inappropriate use of funds, whistleblower retaliation, tampering with or sabotaging experiments/equipment) tend to be relatively rare occurrences. When such incidents are uncovered, however, they carry significant consequences for the offenders, generate a large amount of negative publicity for their specific disciplines, and contribute to the general erosion of societal trust in science (Jha, 2012; Pew Research Center, 2015). In contrast, detrimental research practices ¹ are less readily visible, though purportedly more common, and include activities such as selectively reporting hypotheses, HARKing (hypothesizing after results known), and “flexible” or misleading manipulations of statistics and data, among others (see Banks, O'Boyle, et al., 2016; Simmons et al., 2011). Although detrimental research practices and their outcomes are typically less extreme and immediately felt than are acts of misconduct, they are often difficult to observe or to recognize for what they are and thus represent a subversive and insidious threat to the products of science (National Academies of Science, Engineering, & Medicine, 2017).

Despite evidence that detrimental research practices (e.g., Banks, O'Boyle, et al., 2016; Bedein et al., 2010) and acts of misconduct (e.g., Atwater, Mumford, Schriesheim, & Yammarino, 2014) occur in I-O psychology and related sentiments that the trustworthiness of our evidentiary base may be questionable (Bosco et al., 2016; Kepes & McDaniel, 2013; McDaniel, Kepes, Hartman, & List, 2017), there is also evidence that our science is healthy on a number of fronts and that we do many things right. For example, meta-analytic research shows greater correspondence between paired lab and field study effects among I-O psychology studies than other areas of psychology (r = .89, 95% CI [.83, .93]; Mitchell, 2012). As another longstanding example, the economic benefits of using sound selection procedures have been repeatedly documented (e.g., Schmidt, Hunter, Outerbridge, & Trattner, 1986). The relatively close connection between science and practice, openness to both experimental and field methodological approaches, and the study of topics that are inherently multidisciplinary likely all contribute to promoting the health of our science. Furthermore, matters of research integrity are not unique to I-O psychology, the management sciences, the social sciences, or even the physical sciences. Virtually every discipline of science grapples with these issues and the implications they hold (e.g., Fanelli, 2009, 2011; Ioannidis, 2005a; Marcus, 2014; Rubin, 2011). Although the calls to action are seemingly louder today, robust science concerns are not unique to the modern era and have been part of the narrative for decades (e.g., National Academy of Sciences, National Academy of Engineering, & Institute of Medicine, 1992; Newcombe, 1987; Rosenthal, 1979).

Considering the above, why should I-O psychologists beat the drum about research integrity at this particular moment? At the most general level, the health of a science requires diligence and continued effort to maintain and improve; thus, we should never be content with letting this drum beat stop. Beyond such existential motivations, though, we believe there are at least two specific reasons why it is important our field engage in this issue more directly at this point in time. First, emerging trends in the environment in which research occurs have made it necessary to reevaluate existing methods for safeguarding scientific rigor. The scientific enterprise is now larger and more complex than ever before. According to the National Science Board (2016), the number of individuals enrolled in graduate science programs increased by 19% (7% increase in psychology), and the number of doctoral degrees awarded in the sciences by 43% (2% increase in psychology) between 2000 and 2013. Even more astounding, the number of scientific articles published in 2003 (~1.17 million) nearly doubled by 2013 (~2.2 million; from 19,291 to 37,194 in psychology; National Science Board, 2016). Coupled with this growth, the use of new technologies for gathering and processing data, the globalization and collaborative nature of modern research, and the increasing need for scientific knowledge to inform public policy and decisions have contributed to an industry that is dynamic, fast paced, and high impact (National Academies of Science, Engineering, & Medicine, 2017). Although these developments reflect exciting growth in the capabilities of scientists and practitioners for advancing research and applications in important and unanticipated directions, they also contribute to certain undesirable consequences. For example, more rapid knowledge creation and dissemination diminish the capacity for existing structures/practices to self-correct faulty or scientifically unsound information (e.g., debunking empirical evidence generated using detrimental research practices; Hollenbeck & Wright, 2017). Similarly, pressures to remain competitive and visible in this system contribute to patterns of reinforcement that place significant emphasis on where—as opposed to what—research is published as indicators of research quality (e.g., as evidenced by proliferation of journal ranking lists for tenure and promotion and the provision of monetary bonuses for publishing in particular journals used in some universities). Such norms engender circumstances in which researchers may feel compelled to do what is needed to “win the publication game” that make the scientific enterprise ripe for dysfunction. In sum, many of the mechanisms we have traditionally relied on for monitoring, regulating, and correcting the accumulation of scientific knowledge are beginning to show their limitations in the new scientific reality and underscore the need to reexamine key systems that protect its interests (Casadevall & Fang, 2012; Nosek, Spies, & Motyl, 2012).

Second, and more germane to our own discipline, I-O psychology is uniquely suited to be a leader in the movement toward fostering a robust and reliable science (e.g., Banks & O'Boyle, 2013). Many of our scholarly topics of interest (e.g., motivation, organizational justice, organizational change, leadership) and practice applications (e.g., performance evaluation, change management, training and development) offer insights and important tools that can be helpful for reshaping the scientific enterprise in positive ways. In fact, the relevance of the work performed by I-O psychologists for both identifying and understanding factors that contribute to research misconduct and to detrimental research practices as well as for developing solutions to these issues was cited on numerous occasions in the National Academies 2017 report on research integrity. There is thus a great opportunity not only to benefit our own discipline by pursuing proactive means that contribute to research integrity but also to demonstrate the value of our unique knowledge and expertise to the scientific community and society more broadly.

In order to meaningfully contribute to the dialogue on encouraging, facilitating, and rewarding research integrity, it is important to first identify what a healthy science looks like. We thus now turn our attention toward defining the characteristics of robust science.

Defining Principles for Robust Science

Historical reviews tracing the development of the contemporary research enterprise in the United States note that the roots of scientific industry began as an “informal, intimate, and paternalistic endeavor” in which research was considered more a hobby or a novelty than as a purpose-driven career (National Academy of Sciences, National Academy of Engineering, & Institute of Medicine, 1992, p. 67). As the economic value of science and ideals supporting the pursuit of knowledge for its own sake became more widely agreed upon throughout the 20th century, research as a profession emerged as a viable calling. These developments spurred rapid growth in a variety of institutional and societal systems to support the advancement of science, including publically supported funding agencies for research, a larger publication industry devoted to disseminating scientific work, and more explicit incentive structures to reward the production of scientific knowledge. At its core, however, the scientific profession continued—and largely still continues—to operate in ways consistent with its decentralized, autonomous, and idiosyncratic heritage. For example, there are no commonly endorsed criteria for evaluating research quality or contribution (Nosek & Bar-Anan, 2012), and formalized professional development in science education has generally been eschewed in favor of informal apprenticeships and mentor-based models. The National Academies of Sciences, Engineering, and Medicine (2017) summarize this state of affairs:

Even today, being a scientist and engaging in research does not necessarily entail a career with characteristics traditionally associated with professions such as law, medicine, architecture, some subfields of engineering, and accounting. For example, working as a researcher does not involve state certification of the practitioner's expertise as a requirement to practice, nor does it generally involve direct relationships with fee-paying clients. [. . .] One challenge facing the research enterprise is that informal, ad hoc approaches to scientific professionalism do not ensure that the core values and guiding norms of science are adequately inculcated and sustained. (p. 22)

Efforts to establish explicit and widely held standards for how the research process should be conducted or evaluated, how it should be communicated in graduate student training, and how its supporting institutions should be coordinated are thus bound by the cultural milieu of the scientific profession. To be clear, we do not wish to imply that I-O psychology, the management sciences, or science as a whole need make any radical departures from this system; we believe the intellectual freedom and diversity of thought/opinion it permits are critical drivers of scientific progress. Nor do we wish to imply that our community lacks values and norms concerning scientific integrity or that we fail to transmit and reinforce these expectations in our research, practice, and mentoring. However, recognizing the unique nature of our scientific environment provides insight into the sociocultural challenges (regardless of the logistical difficulties) faced by efforts to reinforce and foster a robust science system. Notably, it highlights the need to engage in what Luscher and Lewis (2008) describe as “working through paradox.” That is, in an effort to drive positive changes in our science, it is desirable both to retain the flexibility afforded by our research culture and to better manage our ways of doing to foster and ensure robust scientific practices.

We believe a useful point of departure for this discussion is to identify an agreed upon set of principles that reflect how our field defines robust science. The purpose of such principles should be to make explicit the values that reflect “better science” (Grote, 2016); operationalize the meaning of robust science; and provide helpful guidelines for identifying, evaluating, and allocating resources (e.g., time, effort, money) toward any proposed changes intended to improve scientific practices (cf., Nosek & Bar-Anan, 2012; Nosek et al., 2015). We do not advocate that any such principles be treated as a checklist of requirements or be used as a basis for judging any individual study or researcher. Further, we do not claim that the principles we list here are or need be novel (e.g., Antonakis, 2017). Instead, we see importance in explicating such foundational principles that serve as markers to orient behaviors and decisions in ways that contribute to robust science across the entire scientific enterprise. To this end, we offer the following defining principles for consideration.

1. Robust Science Is Relevant

There is general agreement among economists and public policy experts that investment in basic and applied research is a significant contributor to and distinguisher of socioeconomic health and success across the global landscape (cf., Fagerberg, 1994). Although difficult to directly quantify the manner by which science leads to such improvements, the fundamental premise is simple: The knowledge, understanding, and expertise generated through the pursuit of science allow scholars and practitioners to address complex problems that benefit our lives in ways big and small (Pavitt, 1996). For example, the science of I-O psychology involves generating knowledge and descriptions of the psychological experiences and social interactions that unfold within and between individuals and the organizations in which they work. The accuracy and usefulness of the knowledge generated by a discipline broadly characterizes its relevance—the degree to which science and practice improve understanding of the world, address important needs and problems of our time, and/or contribute to outcomes that benefit humanity and society (Allen, 2015). Certainly not every scientific endeavor will (nor should) result in an immediately cohesive explanation, applicable technological advancement, policy reform, or cultural improvement. However, when these are the goals that spur science, both the producers and recipients of scientific knowledge are more likely to benefit. Consistent with these sentiments, we propose that a robust science is one in which activities throughout the entire scientific enterprise are conducted with the intention of producing positively impactful and relevant knowledge.

2. Robust Science Is Rigorous

The rigor of a science is reflected in the extent to which its core concepts and their relations are operationalized with precision, and the methodologies used to collect informative observations are accurate and appropriately aligned with the analytical techniques used to infer meaning from those observations. The technologies and methodological/analytical toolboxes available to I-O psychologists for accomplishing these tasks in both research and practice are more extensive and richer than ever before. Furthermore, there is growing appreciation of the unique insights that a wider breadth of approaches can bring to understanding the phenomena that we study (i.e., triangulating on a phenomenon from multiple perspectives, compensating for weaknesses among particular methodologies; Cortina, Aguinis, & DeShon, 2017). More methodological freedom, of course, also places a greater onus on the producers, reviewers, and users of scientific knowledge to be informed of the strengths and inevitable weaknesses that choices among different philosophical (e.g., induction, deduction, abduction), methodological (e.g., self-report, behavioral, physiological), and analytical (e.g., quantitative, qualitative) perspectives hold for advancing scientific understanding. We therefore advance that a robust science is characterized by choices that favor rigor over expedience and values leveraging diverse investigative tools that align with the research questions being examined.

3. Robust Science Is Replicated

Recent discussions and headlines on the replicability/reproducibility crisis within science (Baker, 2016) and the psychological sciences in particular (e.g., Carey, 2015; Yong, 2016) have raised many concerns and anxieties. Large-scale initiatives that have attempted to gather evidence on the replicability of research findings across a variety of topics have presented many sobering statistics on the extent to which the results of published research are reproducible (Ioannidis, 2005b; Klein et al., 2014; Open Science Collaboration, 2015). These results have stimulated heated arguments about the validity of such metascience efforts and the conclusions they imply (see Anderson et al., 2016; Gilbert, King, Pettigrew, & Wilson, 2016). However, similar to the inadequacy of only using p-values and statistical significance tests to determine whether scientific knowledge is “supported” versus “not supported” (Cohen, 1990; Schmidt, 1996; Simmons et al., 2011), efforts to determine the replicability of findings can be similarly flawed when solely directed toward establishing a (false) dichotomy between scientific knowledge that “replicates” versus “fails to replicate” and debating the arbitrary criteria used to justify that distinction. ² The more critical significance of replication efforts is that they facilitate the collection of large bodies of evidence upon which scientific knowledge is based and refined. Consequently, we propose that a robust science is not one that seeks to establish the replicability of its findings per se but instead is one that values and pursues efforts to gather repeated (i.e., replicated) observations of the mechanisms and relationships among core concepts and processes of human behavior, and that these efforts are made accessible in the corpus of scientific evidence. Such practices are critical in order to establish the degree of confidence that should be placed in scientific claims and to support evidence-based practice.

4. Robust Science Is Accumulative and Cumulative

Despite the staggering number of scientific works published every year (National Science Board, 2016), many have maligned that much of this growth reflects that science is “fattening up” rather than “growing up.” That is, there is concern that the emphasis on exciting but untested theory that posits discontinuous change in current thinking is contributing to scientific bloat, whereas efforts to test and build upon existing explanations in ways that gradually refine current thinking languish in obscurity (e.g., Antonakis, 2017; Colquitt & Zapata-Phelan, 2007; Hambrick, 2007; McKinley, 2010). There is clearly a delicate and difficult balance to strike between promoting scientific pursuits in ways that foster incremental progress (e.g., reliance on widely used and tightly controlled paradigms) and encouraging “disruptive” thinking that challenges the status quo with new ideas (Nosek et al., 2012; Pfeffer, 1993). Indeed, it is difficult to envision how a field can remain impactful and credible when it favors one end of this continuum too strongly over the other. Science benefits when both incremental research and novel idea generation occur. Consequently, new ideas and ways of thinking should not be deterred, but we must also recognize that the strength of scientific understanding and inference is enhanced through careful vetting, deliberate calibration, and compounding multiple observations into an integrative whole. We thus propose that a robust science is one in which the pursuit of cumulative knowledge is reinforced by adopting an appropriate degree of intellectual skepticism toward novel propositions and appropriately adjusting those beliefs on the basis of accumulated evidence.

5. Robust Science Is Transparent and Open

Although we all contribute to and participate in multiple aspects of the broader scientific enterprise, a surprising amount of the system remains opaque. For example, authors are often the only ones who know whether time and effort was devoted to examining data supportive of their hypotheses, whereas less supportive data were relegated to the “file drawer” (Rosenthal, 1979). Editors and reviewers only evaluate a summary of the methodological and analytical decisions made in a research study rather than the full original source materials (Nosek et al., 2012). End consumers of research typically never see nor have access to the exchanges, disagreements, and revisions that occur during the review process that ultimately shape a manuscript (Nosek & Bar-Anan, 2012). Practitioners or researchers at institutions with limited resources often lack access to research disseminated in scientific journals because of high publisher subscription costs (Houghton & Oppenheim, 2010). As a result of such gatekeeping actions, very few (if any) individuals in a scientific community ever have full knowledge of “how the scientific sausage gets made.”

Many of the norms that guide the dissemination and communication of research products throughout the scientific enterprise emerged and persist for reasons that are valued, logical, and well-intentioned. Nevertheless, human history has taught us that increasing access to knowledge can be transformative on a grand scale, and organizational research has taught us that knowledge about the distributive, procedural, and interactional decisions made in a system is empowering and tends to promote positive behavior (Greenberg & Colquitt, 2005). As such, we consider a robust science one in which transparency and openness are embraced throughout the research process and scientific system. Activities that embrace these principles include more complete disclosure of data, materials, analyses, and hypotheses to the scientific community; promoting publication practices in which important questions answered well have a place in the literature regardless of results; and creating accessibility to the research process at all stages of production. We recognize there are challenges with data transparency and openness from corporate and private settings (e.g., Gabriel & Wessel, 2013; Jones & Dages, 2013). However, we wish to challenge I-O psychologists working in these settings to push for what can be shared with appropriate protections in place. Too often we fear the default is to not even engage in conversations about data sharing with clients or with I-O psychologists working internally. Ultimately, a robust science is one in which transparency across the scientific enterprise is the professional expectation and at the very least strived for in all that we do.

6. Robust Science Is Theory Oriented

In a manuscript discussing the limitations of quantum theory as the foundation for physics, Albert Einstein noted, “The whole of science is nothing more than a refinement of everyday thinking” (1936, p. 349). In this, Einstein reflects the notion that a critical goal of science is to take observations from our environment and transform them into accurate and useful explanations (i.e., theories) of how and why phenomena in the natural world occur (see also Feynman, 1969). The development of accurate, useful, and evidence-based explanations is a distinguishing feature of all sciences. Consistent with many recent discussions on the role of theory in the organizational sciences (e.g., Antonakis, 2017; Campbell & Wilmot, in press; Colquitt & Zapata-Phelan, 2007; Hambrick, 2007; McKinley, 2010; Ones, Kaiser, Chamorro-Premuzic, & Svensson, 2017; Spector, Rogelberg, Ryan, Schmitt, & Zedeck, 2014), we advance that a robust science entails treating theory as a means to an end, not an end in and of itself. That is, theory for theory's sake is not rewarded, and resistances against or devaluing of applied, inductive, and exploratory research are removed. Instead, a robust science is simply one in which its scientific pursuits contribute to explanation and “refinement of everyday thinking” by replicating, bounding, revising, falsifying, and, when appropriate, advancing new claims. By this definition, we propose that robust science is “theory oriented” (not theory driven or theory dependent) and promotes this tenet by describing, evaluating, and refining explanations. Genuinely accomplishing this goal requires research that reflects quantitative and qualitative methodologies across the full range of inductive, deductive, and abductive approaches.

An important addendum to this principle is the recognition that theory in robust science is not an informed hunch, logical speculation, or collection of plausible relations—it is “a comprehensive explanation of some aspect of nature that is supported by a vast body of evidence” (National Academies of Sciences & Institute of Medicine, 2008, p. 11). Consequently, a theory-oriented robust science is one in which explanations and empirical research emphasize precision. As a general definition, scientific precision reflects the degree to which explanatory mechanisms are specified with sufficient detail such that predictions/explanations regarding the specific magnitude, form, processes, and conditions that account for a set of relations can be advanced and tested (Edwards & Berry, 2010; Kozlowski, Chao, Grand, Braun, & Kuljanin, 2013). Note this definition does not require every aspect of a theory to be correct to be precise. The rightness of a theory is not determined by the clarity of its arguments or through formal logic ³ but by subjecting its claims to the gauntlet of empirical investigation. A science that strives for precise theories purposefully subjects its explanations to an increased “risk” of falsification to determine the level of confidence that should be placed in proposed relationships (Meehl, 1967). These higher evidentiary standards are commonplace in more well-developed areas of science, such as physics and chemistry, but are still lacking in the social sciences (Fanelli, 2010a). We thus submit that a theory-oriented robust science is one in which its explanatory accounts strive for the highest degrees of precision.

In sum, the culture of the modern scientific enterprise in which I-O psychology resides presents unique challenges for enacting changes that foster healthy and credible science. However, we believe that making progress toward these ideals can be facilitated through establishing common ground among members of our scientific community about the meaning of robust science and its foundational aspirations. To this end, we advanced six defining principles of a robust science (summarized in Table 1); specifically, a robust science is (a) relevant, (b) rigorous, (c) replicated, (d) accumulative and cumulative, (e) transparent and open, and (f) theory oriented. Endorsing a shared vision of robust science represents a commitment to promoting and reinforcing activities that emphasize positive scientific behaviors. More important though, it can also serve as a means for guiding and coordinating collective action that incrementally contribute to these changes. It is to this point that we now turn our attention.

Table 1. Summary of the Defining Principles of Robust Science

I-O Psychology's Scientific Ecosystem and Recommendations for Action

The scientific enterprise is a complex, interconnected ecosystem of stakeholders, processes, and functions (National Academies of Sciences, Engineering, & Medicine, 2017). We assert that creating and sustaining a robust science will require a scientific ecosystem whose members reflect five core intentions:

1. 1. Recognition that science is a public good and thus should be readily available for the benefit of everyone

2. 2. Provision of incentives and reward structures that value the publication of a full range of evidence (i.e., null results, replications, etc.) based on well-formulated research, regardless of the findings

3. 3. Developing and embracing research output platforms that move outside the traditional scientific communication system

4. 4. Engaging all stakeholder groups in adherence to the defining principles of robust science

5. 5. Reinforcing the defining principles of robust science at all stages of the research process

Many different actions could be undertaken to satisfy these goals in I-O psychology, and models for their implementation can be gleaned from other disciplines. However, successful efforts at promoting robust science made in other fields may not necessarily translate across different scientific disciplines, given variability in norms and practices (e.g., Evans & Reimer, 2009). This highlights two important needs for adapting the scientific ecosystem within I-O psychology. First, there is a need to understand the various stakeholders and contributors within our contemporary scientific enterprise so that the roles these entities play in promoting and sustaining a healthy science are better understood. Figure 1 provides a graphical heuristic of the most significant members we consider in this ecosystem. The stakeholders in this visualization are organized around two concentric rings reflecting the extent to which we believe the unique roles played by these entities are more (innermost circle) versus less (outermost circle) capable of directly influencing robust scientific practices, as well as the extent to which our scientific community has more (innermost) versus less (outermost) direct control over their norms, procedures, and operations. As any single individual may contribute to multiple of these stakeholder groups concurrently, it is useful to identify the different actions one could carry out to promote robust science when wearing any of these hats. Second, the only way in which the goals of a well-functioning scientific ecosystem can be achieved is through participation and action at all levels of the system. We believe small, meaningful changes that take place simultaneously and in a coordinated fashion across multiple members of our scientific enterprise can lead to self-reinforcing behaviors that gradually improve the field. As such, our goal for the remainder of this article is to identify the significant contributors to the scientific norms and practices within I-O psychology and recommend proactive changes that these entities can make to facilitate robust science. We fully recognize that our list of stakeholder elements is not exhaustive. It is not hard to identify other entities of relevance (e.g., organizational decision makers, universities). We chose the ones we did because we believe the field would be united around the central importance of each. Table 2 provides a summary of these recommendations.

Figure 1. Scientific ecosystem responsible for fostering robust science.

Table 2. Summary of Recommended Actions to Foster Robust Science by Stakeholders in I-O Psychology's Scientific Ecosystem

Authors

Authors are the primary generators and communicators of the research products in a scientific field. As the first point of contact with the collection, creation, and interpretation of inductively driven ideas and deductively guided empirical data, authors give voice to the explanations that are advanced for phenomena observed in the natural world. Consequently, authors are arguably the most influential source of bottom-up change in the scientific ecosystem for fostering robust science. We believe that the core responsibilities authors hold for fostering a robust science can be categorized into two broad sets of activities: (a) accurately, honestly, and clearly reporting all activities used in the production of scientific research; and (b) prioritizing publication practices that embody the defining principles of robust science. These responsibilities span many specific actions encompassing both the research and publication processes, but below we highlight three activities perceived as particularly critical.

First, authors should accurately describe and represent the capabilities of their methodologies and analyses when proposing inferences from their research. Science is not—and can never be—perfect. To borrow from McGrath's (1982) discussion of dilemmatics, research inevitably involves decisions that create tradeoffs among the precision, generalizability, and realism of results. In our field, it is common for authors to acknowledge these choices near the end of a manuscript when outlining the limitations of a research study. These sections have largely become placeholders for obligatory statements such as “the research should be tested using a different sample” or “the cross-sectional design does not allow for causal inferences,” which authors then feel motivated to defend to lessen the blow of their choices on conclusions they wish to draw. The role of an author in a healthy scientific ecosystem is not to convince the reader of how “exceptional” their study is, nor to generate a list of possible flaws and then systematically rebuke them. Instead, the author's responsibility is to openly acknowledge the extent to which the methods and analyses used in their research do and/or do not allow them to draw particular conclusions. This goal should be reflected throughout a manuscript rather than only after findings have been presented, including when hypotheses are developed (e.g., justifying hypotheses that do not propose the form, magnitude, or conditions for an effect; Edwards & Berry, 2010), methods and analyses are described (e.g., conveying extent to which a measure/technique permits one's predictions to be examined, degree to which results are robust to other plausible analytical choices; Simmons et al., 2011), and inferences are advanced (e.g. not overgeneralizing implications beyond the data; Hollenbeck & Wright, 2017).

Second, authors should be precise and transparent when describing the development of research questions and the rationale for the methodological/analytical choices used in their research. Much has been written concerning the statistical and epistemological concerns of HARKing and similar detrimental research practices (Banks et al., 2016; Hitchcock & Sober, 2004; Hollenbeck & Wright, 2017; Kerr, 1998). The specifics of those arguments are beyond the scope of this discussion, but their root cause is a lack of transparency from authors concerning conceptual rationale, methodological decisions, and analytical choices. Whether one identified a meaningful relationship after (faithfully) hypothesizing a finding a priori or it was identified post hoc does not change the fact that the observed relationship was present in the data. The natural world could not care less whether the scientist made an accurate prediction beforehand or not—but the manner by which scientific inference advances does. The attributions, interpretations, and implications that authors advance about their observations should change based on the process used to generate them. When a conceptual rationale is genuinely new, deviates from previous explanations, or is based on small sample sizes, it is important (and acceptable!) for authors to present interpretations from those studies with a healthy dose of skepticism. Similarly, it is important (and again, acceptable!) to openly describe the pros and cons of the methodological or analytical decisions made by the authors to pursue a scientific explanation. The goal of authors in a robust scientific ecosystem is to improve explanations and descriptions of how and why things in the natural world work the way they do. The ability to achieve this goal is significantly impaired when precision and transparency in conceptual rationale and methodology are lacking (Simmons et al., 2011).

Last, authors should commit to enacting the defining principles of robust science even when they are not rewarded for doing so. There are numerous avenues through which authors can attempt to meet the values of robust science. For example, authors can show a preference toward journals that actively reinforce robust science principles; make a point to have colleagues from both practice and academia collaborate and review manuscripts to engage multiple stakeholders in the research process; make materials, data, and/or analyses openly available when feasible; and place accepted prepublication manuscripts in open-access repositories. Whatever the ways in which these values could be fulfilled, they share the characteristic of being outside the current status quo and the route through which status and recognition are typically achieved in the current publication enterprise (Nosek et al., 2012; Nosek & Bar-Anan, 2012). These activities feel like extra work with little direct payoff to authors—because that is precisely what they are at present. However, a vibrant ecosystem can only be sustained if authors collectively agree that “what's good for the science” is at least as important and helps to achieve “what's good for the researcher.” Ironically, this situation shares many of the same characteristics with classic social dilemma and game theoretic scenarios that organizational scientists have studied for decades (Byington & Felps, 2017). The widely replicated inference from that line of research is that when individuals choose not to engage in socially beneficial behaviors, everybody loses in the end. If our field is to achieve the ideals of robust science, then all authors must contribute to the system they wish to create.

Editors

Editors-in-chief and associate editors of scientific journals (hereafter collectively referred to as editors) represent the most significant gatekeepers of published research in the current system of scientific communication (Nosek & Bar-Anan, 2012). Editors possess a number of avenues for influencing the quality and type of research published (Zedeck, 2008, 2017). Editors’ actions at a journal are generally perceived as communicating the requirements, criteria, and beliefs about what constitutes high quality and valued research. Furthermore, individuals also typically reach editorial positions because of their career successes and thus tend to be respected and widely connected figures in the broader science network. Editors thus possess multiple bases of power (e.g., reward, legitimate, referent, expert, and informational power; Raven, 1992) that make them instrumental levers for stimulating change in the scientific enterprise. We highlight a number of recommended activities in which we believe editors in I-O psychology and the managerial sciences should engage to foster robust science.

First, editors should establish review and reviewer norms consistent with robust science. To have a robust science, compelling research questions answered in methodologically sound ways should be published regardless of the actual pattern of results. We believe editors should explicitly communicate to their editorial teams and editorial boards that publication recommendations be made based on two principal factors: (a) the theoretical, conceptual, and practical relevance of the research questions explored and (b) the rigor of the methodology/analyses used to answer those questions. Along with this, editors should strongly communicate continuously to reviewers the importance of being open to scientific advancement through both inductive and deductive research, and that a “perfect study” (i.e., all hypotheses fully supported, all a priori theoretical rationale completely consistent with observed data, etc.) is not a requirement for publication per se (Simmons et al., 2011). After all, null findings found in statistically and methodologically powerful ways are essential to the advancement and self-correction of science (Landis, James, Lance, Pierce, & Rogelberg, 2014). This type of very frequent messaging facilitates alignment of the expectations and values expressed by reviewers.

Second, and very importantly, editors should provide regular and constructive feedback to reviewers. In many ways, editors represent the chief talent officers of their journals—they seek out reviewers they think will provide high quality services, wrestle with strategies for balancing reviewer workload, and are constantly looking for ways to diversify and maintain a productive crop of reviewers. However, one activity we believe could occur more frequently is provision of feedback and training opportunities to reviewers to help shape and/or correct behaviors in ways that promote robust science. For example, a reviewer who suggests dropping null results or creating new hypotheses after results are known should be informed by the editor of the inappropriateness of this recommendation and asked to revise the comment and/or signal in the decision letter that this action is inadvisable. Reviewer feedback and development by the editorial team not only plays an important role in facilitating more productive reviews but also signals that the practices of robust science are desirable and should be applied when reviewers adopt authorship roles with their own independent research. Editors must recognize their unique responsibilities in assuring that reviews and decision letters from their journal do not implicitly or explicitly encourage questionable research or work counter to robust science. If time for the above essential activities appears nearly impossible to find, editors must work strongly, in concert with their publishers, to expand the editorial team and/or streamline processes to recover time. A lack of time cannot be a reason to abdicate robust science responsibilities.

Third, editors should experiment with enacting nontraditional practices at their journal designed to promote robust science. Editors work hard to maintain fair and balanced practices that ensure the “rules of the game” implicitly and explicitly promote and result in dissemination of high quality research. Although the current system has been successful on a number of fronts, there are many new and emerging ideas about how these practices might be altered to better engender behaviors that facilitate robust science. Some visions for these practices represent novel and idealistic changes that would require substantial shifts in editorial infrastructure and journal management (e.g., Nosek & Bar-Anan, 2012). Others though are much more moderate in scope (e.g., Nosek et al., 2015). For example, attempts by editors to introduce results-blind submission paths, explicit journal sections welcoming replications, and options for authors to preregister studies are relatively straightforward, and good models for these practices are emerging. Journal editors could encourage authors to host their data, materials, measures, additional analyses, previous manuscript versions, review letters, and other materials in accessible and online repositories (such as the Open Science Framework, http://osf.io) to promote increased transparency and accumulation of science. Additionally, editors could pursue and reward regular reviewer training to promote an improved review process. In essence, editors should not only strive to be a steward of their journal but to also be a steward of science and use their unique role to experiment with practices that can make meaningful improvements for the scientific enterprise.

Last, editors should evaluate and, where appropriate, revise journal management practices to ensure they are consistent with advancing the principles of robust science. Beyond the many tasks that editors perform related to promoting the rigor and quality of research published in their journals, editors are also the captains and caretakers of a journal's reputation while they are at the helm (Hollenbeck, 2008). Consequently, engaging in thoughtful and diligent reflection on editorial leadership practices reflects an important piece of the puzzle. For example, much has been written about the importance of impact factors and “citability” as indicators of journal quality and editorial stewardship (e.g., Byington & Felps, 2017; Ireland, 2008). Although impact factors convey useful heuristic information about journal quality, narrowly focusing on them can encourage “gaming” behaviors (e.g., coercive citations) and discourage experimentation with interventions that may be good for robust science but potentially poor for impact factors. Embracing a multifaceted perspective on journal quality that extends beyond impact factors to characteristics such as engagement in open science, humanity in the review process, responsiveness, timeliness, and stakeholder satisfaction should also be considered. In addition to managerial practices focused inwardly on one's own journal, editors also play a key role as a liaison with publishers and other journal editors within and outside our specific discipline. Advocating for publication models that provide greater access to science and engaging in cooperative cross-journal initiatives that collectively promote positive scientific behaviors should be normative editorial behaviors. ⁴

Reviewers

There is widespread belief that the peer review process is one of the oldest and most venerable characteristics of science. Although critical dialogues and the defense of ideas among scholars have certainly existed since the earliest of scientific philosophers, the peer review process as we know it today did not exist until the mid-to-late 20th century (coinciding with the commercial availability of photocopiers and the ability to easily print multiple copies of a paper; Spier, 2002). Reliance on voluntary peer reviews of scientific research is now a staple of the contemporary scientific enterprise. Although editors still make the final determination on whether to publish a given research product, reviewers have become an increasingly significant quality assurance and gatekeeping mechanism of a field's research. We thus propose three recommended actions from reviewers within the field for promoting robust scientific practice.

First, reviewers should be open to diverse methodological approaches, research questions, and types of evidence in their evaluations. When evaluating manuscripts submitted for publication, importance/rationale of the research questions along with methodological rigor should continue to drive the recommendations made to editors regarding suitability. As noted previously, the rigor of a research study or theoretical proposition is characterized by precise operationalization, the extent to which the methodologies used permit collection of valid/informative observations, and the use of transparent and justifiable analytical strategies for generating inferences. There are clearly multiple ways in which these goals can be achieved. As such, the goal of reviewers is to evaluate the research product before them and not the research they would have preferred to see. On occasion, this may require reviewers to learn about unfamiliar methodological (e.g., inductive, qualitative, computational) or analytic (e.g., Bayesian statistics, semantic analyses, data mining) approaches to provide an adequate evaluation. Although such activities demand additional time, we believe it is important that reviewers and editors not perceive this as an unwanted burden but rather as professional development and an opportunity to learn something new (which may prove useful in one's own work) while simultaneously providing a service to their field and colleagues. This does not mean that reviewers cannot or should not be constructively critical and identify where improvement/clarity is warranted. However, recommendations should be sensitive to the fact that there are many different approaches through which research can be conducted and such diversity is conducive to robust science. Relatedly, reviewers also should not display a preference for the “novelty” of an idea but should be open to any sound study that leads us closer to understanding important phenomena. That is, reviewers and editors should consider the study as a whole (i.e., theory, methodology, importance of question) in terms of the extent that it helps contribute to an understanding of the phenomenon of interest.

Second, reviewers should be more accepting of “imperfections” in research. Imperfect research exists because it is the only kind of research there is (McGrath, 1982). Everyone who has conducted research knows there will always be instances where different decisions could have been made that might have altered the pattern of observed results and inferences drawn from a study. However, the expectation that every or even most data patterns should work as predicted (or be predicted a priori) is unreasonable, unrealistic, and likely contributes to the prevalence of many of the detrimental practices that take place in our research (Simmons et al., 2011). Results that fail to achieve statistical significance and hypotheses that are unsupported do not necessarily signal bad research. In fact, the occurrence of a null finding may be an important data point on the road to discovering truth. Such data should not be censured from our knowledge base if the observation is based on sound scientific practices. A review process that is more open to these “imperfections” promotes an accumulative science that allows for more accurate and precise explanations/predictions to be achieved. At the same time, it decreases the overall incentive for authors to implicitly or explicitly engage in detrimental research practices and submit papers that have been overly sanitized and cleaned.

Last, reviewers should promote and encourage authors to engage in practices that support the defining principles of robust science. Like editors, reviewers have significant opportunity for communicating positive norms about robust scientific practices. Whereas reviewer commentary should be directed toward evaluating the relevance and rigor of research, reviewers can also hold authors accountable for participating in open science by encouraging engagement in transparent reporting and sharing practices wherever possible during the review process. Additionally, papers that include or could include replications of previous findings and therefore contribute to cumulative knowledge should be positively recognized and drawn to the attention of editors. In short, reviewers should see themselves as empowered change agents that can help move us toward a more robust science. The reviewing process can serve as a leverage point to incentivize robust scientific behaviors and enhance the soundness of our scientific findings.

Professional Associations

There are typically multiple professional associations within any given scientific community. Some are small and regional; others are large and international. Some exist primarily to host conferences, and others provide certification, publishing, advocacy, and research support services. Irrespective of their differences though, all professional associations share the feature of promoting a common identity and network for members of a scientific community. As a result, professional associations provide a number of unique resources and opportunities for advancing initiatives to support robust scientific practices.

First, professional associations should publically endorse, openly advocate for, partner with, and reward initiatives intended to advance the defining principles of robust science. Whereas scientific journals and the publication review process implicitly communicate normative expectations about robust science, professional associations have the opportunity to communicate these values more directly and broadly. This activity should be seen as a core function of any scientific professional association, as it underlies the credibility of the field, the credibility of the academy, and the credibility of practice. Both the Society for Industrial and Organizational Psychology (SIOP; http://www.siop.org/mission.aspx) and the Academy of Management (AOM; http://aom.org/About-AOM/Vision,-Mission,-Objectives-Values.aspx), the two largest professional associations in our field, provide direct statements about the values, mission, and objectives held by their associations. We believe these mission statements could be updated to include declarative commitments to explicitly foster robust science. To be sure, both of these groups have engaged in some limited activities toward these ends. For example, in 2008 SIOP leadership entered into an Alliance for Organizational Psychology (AOP) with Division 1 of the International Association of Applied Psychology (IAAP) and the European Association of Work and Organizational Psychology (EAWOP). Eight years later, the first AOP summit meeting was held and a memorandum of understanding was established that identified several potential problems that may undermine the rigor, ethics, and relevance of our scientific enterprise and offered potential solutions for addressing these issues (Grote, 2016). Although it is still too early to know what outcomes and initiatives may arise from these and other similar actions, momentum and commitment to achieving the goals of robust science can be improved through direct endorsements, declarations, and advocacy by professional associations. In other words, we need our key professional associations to make robust science a top priority that is strongly addressed year in and year out.

Second, professional associations should provide resources for developing and implementing tools, techniques, and systems that encourage robust scientific practices. Openly supporting the values that foster robust science is an important step to promoting a healthy culture of science. However, change will only happen with boots on the ground. Professional associations can make a difference by dedicating time and monetary resources to members for instituting change. For example, the task force on Robust and Reliable Science (to which the authors on this article belong) requested and received financial support from both the SIOP Executive Board and Scientific Affairs Committee to back its efforts. In collaboration with the Executive Board and the SIOP Administrative Office, the 2017 SIOP Program Committee was able to create a new submission track and infrastructure allowing SIOP conference attendees to share data, analysis code, and papers presented at a session or poster with other members (see also http://www.siop.org/rr/). The allocation of resources to robust science activities creates accountability on the part of advocates (i.e., members) and supporters (i.e., professional associations) for making positive changes. We believe there are many additional initiatives where the commitment of resources by professional associations would be beneficial, such as (a) efforts to develop competency models, education, and certificates of training for reviewers; (b) repositories for training and/or standardized applications for using open-source statistical programs that promote analytical transparency; (c) introductory training and incentives for engaging in open science practices (e.g., through Center for Open Science, https://cos.io/); (d) using conferences and workshops as platforms for teaching open science elements to members; (e) developing and managing online indexed open-access journals/reports that can become locations for replication research; and (f) providing small grants and seed money for initiatives designed to promote robust science (e.g., “study swap” and other similar initiatives that encourage data sharing/partnerships between academics and practitioners, creation of a “measures chest” or similar repository for sharing materials to facilitate replications). We recognize that the success and implementation of any such activities will be conditioned on the determined actions of individual members and collaborations within a scientific discipline. However, these bottom-up changes can be greatly accelerated when support and resources are provided by institutions and professional associations that align with and value these goals.

Practitioners

I-O psychology is based on a scientist–practitioner model in which the knowledge generated through science ideally informs practice, and the front-line needs faced in practice ideally inform science. Although debates over the extent to which our discipline has been successful at (or is even capable of) achieving this vision are common (e.g., Rupp & Beal, 2007), those who apply the evidentiary base of organizational science to address the concerns, problems, and goals of organizations and workers represent integral members of the scientific enterprise. Depending on their roles and activities, practitioners may assume many of the same responsibilities as authors, reviewers, editors, consumers, and participants for fostering robust science. In addition to the recommendations advanced for those groups, we posit two unique actions through which practitioners can further contribute to improving the scientific ecosystem.

First, practitioners should place greater value on practices whose evidentiary base exhibits characteristics of robust science. Just as one places greater confidence in empirical research knowledge generated by robust scientific practices, the application of knowledge that is supported by robust science to create tools, services, and solutions will be more generalizable and reliable. Practitioners who make use of the knowledge produced in I-O psychology to address issues faced by organizations are responsible for interpreting the appropriateness of research and the evidentiary base they use to support their practices. In some respects then, empirical and conceptual work that adheres to the defining principles of a robust science creates a competitive advantage for practitioners, as the degree of uncertainty around the efficacy of practices supported by that work should be reduced. Furthermore, practitioners supporting and more actively applying knowledge that is transparent, replicated, precise, and rigorous can also provide motivation for researchers to engage in these practices to improve the relevance of their primary research. Displaying preference for robust science by relying on sources of well-established understanding benefits practitioners and simultaneously helps advance the relevance and rigor of the field's scientific base.

Second, practitioners should collaborate with academic researchers (and vice versa) to broaden the range of settings, data sources, and interpretations present in the scientific knowledge base. The call to bridge the science–practice divide is not a new one (e.g., Latham, 2007; Rynes, 2007; Rynes, Giluk, & Brown, 2007). Many of these appeals cite the importance of shrinking this gap so that scientists and practitioners are aware of the latest developments/needs and to stimulate new directions for empirical study. Perhaps more fundamentally though, partnerships between scientists in academic settings and scientist–practitioners are among the most direct and straightforward methods available for strengthening the rigor, precision, and relevance of our evidentiary base—an outcome that benefits both the accumulation and application of science. Through sharing measurement practices, collaborating with researchers on field studies, and making the results of practice applications more accessible and available to the scientific community, practitioners represent a potent force for challenging the generalizability of research and theory. Academic–practitioner collaborations thus do more than keep the cutting edge sharp; they serve as an essential barometer and foundational bedrock for determining the degree to which scientific findings may be considered legitimately robust. Although such partnerships are likely to face additional hurdles with respect to transparency of methods and data due to concerns over intellectual property, confidentiality, and legal protections, we do not see these challenges as insurmountable. Practitioners frequently conduct research during the implementation and evaluation of organizational programs and should thus be similarly documenting their research methodology, sample characteristics, and statistical procedures. The steps taken to produce the results of their analyses (e.g., data exclusion criteria, scale construction, etc.) should also be documented to enable accurate interpretation of findings and limitations due to sample conditions. Whenever possible and appropriate, efforts to share data and methods associated with research application and extensions should be made available across the profession. This goal is facilitated through collaboration among scientists who work across various settings (e.g., academic or field-based practice).

Educators

Science is a continual learning process. Many in academic and practice settings facilitate this goal through the provision of both formal and informal instruction, guidance, and mentorship. The most familiar of these educational roles is found in universities where professors provide training to both undergraduate and graduate students on the principles of I-O psychology. However, educational roles span a variety of contexts, including professional development workshops, developing online tutorials, and working as project managers. Educators and mentors across the board thus play a pivotal role in shaping and transmitting norms for robust science through teaching and modeling positive scientific practices. We suggest two actions in which educators should engage to promote principles of robust science.

First, educators should incorporate and emphasize the significance of the defining principles of robust science into explicit educational objectives. Virtually all undergraduate and graduate education in I-O psychology and related sciences exposes students to fundamental issues related to the scientific method, research design, validity, and research ethics. In our experiences, the primary focus of instruction on these topics is most often directed toward discussing tradeoffs among methodological or analytical choices (in the case of research methods) and issues surrounding the protection of research participants (in the case of research ethics). Both of these content areas are unquestionably important, relevant, and should continue to be emphasized as part of formal training. However, we believe these educational efforts should be supplemented with instruction on the principles of robust science and the responsibility that scientists have for ensuring such criteria are met. For example, discussing the meaning and identification of research misconduct and detrimental research practices, and why they should be avoided; understanding the difference between HARKing versus alternative forms of inference and their implications for advancing science (e.g., Hitchcock & Sober, 2004; Hollenbeck & Wright, 2017); understanding the difference between imprecise and precise theory (Edwards & Berry, 2010; Kozlowski et al., 2013); and describing the significance and means for conveying transparency in one's methodological and analytical choices (e.g., Simmons et al., 2011) all reflect topic areas for which formal educational objectives for training would foster robust science. The same is true for educational initiatives that occur outside of student education as well (e.g., ensuring that learners in a workshop on research methodologies or analyses understand how to apply new techniques in ways that promote transparency, rigor, and precision). We recognize that these topics are often discussed idiosyncratically as part of the mentorship system in science education; however, we believe there is great benefit in more systematic and consistently delivered educational instruction on these topics.

Second, educators should actively demonstrate, model, and provide learners with opportunities to engage in robust scientific practices. Research has reliably demonstrated that the retention and transfer of learning is greatly facilitated when training and education provide clear models for how to enact desired behavior and provide learners with opportunities to practice and receive feedback on these skills (e.g., Bell & Kozlowski, 2008; Taylor, Russ-Eft, & Chan, 2005). In other words, learning is facilitated through showing and doing, not just telling. Because a great deal of education in I-O psychology occurs through individualized mentor–mentee or supervisor–supervisee relationships, it is particularly important that robust scientific practices be modeled and reinforced during collaborative work with junior learners. Additionally, opportunities to practice behaviors conducive to robust science should be made available to learners. There are many activities that could be used to satisfy these goals: engaging learners in the scientific review process (e.g., sharing reviews received and generated by the educator, “co-reviewing” a paper with the learner); exploring tools and emerging products that facilitate open science (e.g., using open source statistical programs such as R, using repositories such as the Open Science Framework for theses/dissertations); and promoting efforts to replicate existing research alongside new/ongoing research all represent avenues through which educators can enact and engage learners in robust science.

Publishers

The publishing industry is experiencing a shifting economic landscape. On the one hand, publishers reap tremendous profit from the current business model supporting scientific publishing (Houghton & Oppenheim, 2010). On the other hand, business models have changed dramatically in recent years as the availability of electronic platforms continues to grow. One need only look at the impact of online reporting outlets on the newspaper and magazine industry to find examples of the rapid demise of once prominent publications and the dissolution of business enterprises that persevered within the traditional business model. ⁵ The scientific publishing industry is susceptible to similar patterns, though change has been slower. Revenue forecasts for scientific and technical journal publishers have shown flat growth in recent years (e.g., Ware & Mabe, 2015). Open-access vehicles are rapidly growing yet still capture only a small portion of the overall scientific and technical research output market (Neylon, 2013). With continued pressure on scientific publishers to compete with and adopt more open access practices, it seems likely the economics of publishing will favor the move toward digital formats. Nevertheless, so long as the current publication model remains advantageous to revenue generation from the sale of journal subscriptions (primarily to research libraries supported with public funding), the current publication model will remain steady. Consequently, we believe that publishers play a critical role in fostering robust science.

Most notably, publishers should recognize and be proactive in counteracting risks associated with the traditional publishing model that contribute to the prevalence of detrimental research practices. In theory, publishers should have a vested interest in the quality of the content they produce. If the status of published science erodes as a result of trends observed in the current publishing process (e.g., HARKing, dropping null findings to craft a better “narrative,” etc.), the value proposition offered by their most prestigious and profitable research outlets stands to be damaged. Furthermore, to the extent that authors and consumers within the scientific enterprise value the benefits to robust science that open-access journals provide (i.e., quicker time to print, greater data transparency, “crowd-sourced” peer reviews, more rapid dissemination of ideas throughout the scientific community, higher citation rates, etc.), shifts in submission rates and journal readership may change. Some have suggested that many of the publication practices observed today stem from constraints that emerged because of production costs associated with printing paper copies of journals (e.g., Nosek & Bar-Anan, 2012). Whatever the genesis, removing archaic page restrictions, ending restrictions on editorial team size, and providing latitude/encouragement to editors to engage in innovative initiatives opens the door for a variety of alternative journal publication and readership practices capable of fostering a more robust science. For example, new metrics for research quality that move beyond simple citation-based measures can be pursued. Research access models where impact is assessed by factors such as page views, the quantity and quality of active commentary, the number of studies that replicate and extend the original work, and the value conveyed by the provision of supporting materials and original data are more easily traceable in an open-access model and contribute to the accumulation of robust science. Increasing the variety and transparency of research outlets and publisher practices has potential to stimulate more and better scientific advances while maintaining the credibility of the scientific and technical publishing market. Like editors and reviewers, publishers should strive to be stewards of science as well.

Academic Hiring and Promotion Committees

The phrase “publish or perish” is a ubiquitous and anxiety-provoking maxim familiar to anyone within academia. Whereas the production and dissemination of scientific knowledge is a core activity of all researchers, publication numbers, citation counts, and other easily quantifiable metrics of research productivity remain the coin of the realm in higher education (De Rond & Miller, 2005). These norms have almost certainly been driven by increased competition over declining resources in the scientific landscape and the subsequent need to differentiate oneself and one's academic institution as respectable, distinguished, and desirable. Within academia, the proverbial rubber meets the road with the publish-or-perish adage in the form of hiring and promotion committees that determine who is hired, who is retained, and who is permitted to rise through the ranks. Much has been written and even limited empirical evidence exists linking the pressures to publish reinforced through hiring and promotion committees to detrimental research practices and other undesirable outcomes (e.g., Anderson, 2007; Fanelli, 2010b; Rawat & Meena, 2014). Hiring and promotion committees thus represent an obvious and important contributor for shaping the way in which the scientific enterprise operates.

We recommend that academic hiring and promotion committees should adopt a more holistic approach to evaluating candidates that reduces singular emphasis on “very elite” research productivity per se and rewards engagement in practices that foster robust science. Advancing scientific knowledge that is relevant, rigorous, replicable, cumulative, transparent, and theory oriented—that is, research that adheres to the defining principles of robust science—should be among the most important criteria for determining the contributions of a researcher's work. These evaluative markers can be incorporated into hiring decisions by efforts to evaluate an applicant's record of robust scientific practices and the extent to which these individuals have taken proactive steps toward demonstrating these principles in their contributions to the field (i.e., engage in transparent data sharing, attempt to replicate their findings in addition to advancing new understanding, demonstrate the capability to produce precise theoretical predictions). Likewise, promotion and tenure decisions are often influenced by the sheer quantity of an individual's publications and the extent to which these products appear in “A” journals; at the extreme, these decisions may rely very little on appraisal of the quality of an individual's work. At the same time, they can create a very high-stakes publishing game where authors will perhaps do everything it takes (including engaging in detrimental research practices) to publish in the journals that will get them their tenure and/or promotion. The achievement of robust science simply cannot be achieved through such a reward structure. Only through valuation of the extent to which a researcher's body of work embraces robust science, regardless of where such work may be published, will the scientific enterprise advance in a positive direction. We recognize that such shifts in the evaluation of a researcher's record places different and significant demands upon peers and administrators to more fully understand the nature of robust scientific contributions. However, we believe such incentives are critical and liberating to the pursuit of accurate and useful scientific knowledge.

Consumers

How and why the scientific knowledge generated by a field is used vary widely depending on the needs and context of different research consumers. The insights gained through I-O psychology are used to advance future research, guide practice, and to support policies/decisions of critical importance to individuals, teams, businesses, and society. Consumers of scientific research span a diverse range of stakeholders, including authors, reviewers, editors, practitioners, funding agencies, organizational decision makers, and the general public (e.g., media). Although each of these constituencies shares varying degrees of responsibility for ensuring that scientific research is appropriately conducted, all share responsibility for ensuring that this knowledge is properly interpreted and correctly utilized. As a source of demand for the supply of scientific knowledge generated within our field, consumers of research also play an important role in fostering a healthy scientific enterprise.

We believe consumers should place greater value on research that embodies the defining principles of robust science. Novel, counterintuitive, and “catchy” theories/research findings have always sparked great interest across the spectrum of scientific consumers (Antonakis, 2017), and likely will continue to do so. Why wouldn't they? Many who conduct research, and likely many more who do not, believe that the spirit of science lies in innovation, discovery, and intrepid exploration of the unknown (e.g., Astley, 1985; Davis, 1971; Pfeffer & Fong, 2002). Weick (1989) even famously equated the construction of theory in the managerial sciences as “disciplined” imagination. We acknowledge that these motives are central to the fabric of science and can be beneficial forces for scientific advancement. However, when consumers within the scientific enterprise too strongly favor “creative” science to the detriment of robust science, a positive feedback loop is created in which the number of ideas in need of testing/verification far surpasses the capacity to do so. ⁶ This state of affairs places the relevance, reliability, and credibility of a field's work at risk. Consumers of research can influence this trend by holding science accountable for producing robust science. The ability to distinguish robust science obviously requires understanding of the power and limitations of research methodologies as well as how to evaluate the degree of confidence that should be placed in research. However, the challenge in determining these criteria is attenuated when scientific evidence is supported by behaviors characterized by rigor, transparency, and precision. Consequently, by expressing preference toward journals and outlets that require and actively engage in robust scientific practices, consumers have potential to shape the scientific ecosystem.

Research Participants

Unlike many of the natural sciences, the foci of study for the social sciences do not uniformly adhere to well-known and clearly distinguishable principles of behavior. People and the psychological/social systems that they create are diverse, rich, and exhibit a free will of choice that makes efforts to understand how they function simultaneously fascinating and perplexing. As I-O psychologists, we rely on a variety of measurement and observational tools that attempt to capture the “hidden” experiences of people in laboratory and organizational contexts. The individuals who participate in our research thus carry significant weight in determining the quality and reliability of the evidentiary base. Although it can be difficult (and often undesirable) to directly influence the way in which research participants engage with the scientific process, they are integral elements of the scientific ecosystem whose actions clearly contribute to key aspects of a robust science.

Accordingly, we believe that research participants should be encouraged to recognize the importance of their contribution to science and participate in ways that reflect that status. Discussions about the use of college students (e.g., Greenberg, 1987; Sears, 1986) and more recently Mechanical Turk workers (e.g., Burhrmester, Kwang, & Gosling, 2011; Cheung, Burns, Sinclair, & Sliter, 2017; Paolacci & Chandler, 2014) as research participants frequently cite concerns over the trustworthiness and credibility of these data sources. Though similar critiques can just as easily be levied against samples gathered from organizational workers (Landers & Behrend, 2015), the more fundamental concern is legitimate: the validity of scientific inference is dependent on the quality of the observations and the evidence upon which it is based. For example, recent research has identified the pervasiveness of careless responding on the part of research participants (Huang, Curran, Keeney, Poposki, & DeShon, 2012; Meade & Craig, 2012). This lack of attention and effort can reduce the rigor, precision, and utility of our scientific conclusions (cf., Huang, Liu, & Bowling, 2015). Consequently, we believe it is important that research participants are cognizant that their actions can improve—and hinder—the quality of science. Beyond a role as reliable and honest contributors, research participants can also play an important role in reporting violations of participant rights. One emerging issue concerns the need to inform participants and for participants to be informed about what will happen with their data. Transparency is a central tenet of robust science, including data transparency. Researchers and practitioners have an important responsibility to ensure that appropriate identity and confidentiality safeguards are in place, but participants may be faced with a new normal where their data are likely to be available more broadly than they have been in the past, and thus participants should be encouraged to inquire about the protections and rights with which they are afforded.

Funders

Although funded research within I-O psychology and the managerial sciences remains much less common than in other fields, there are a number of government agencies and nonprofit foundations that support the advancement of scientific research. In the United States, the National Science Foundation, numerous agencies within the Department of Defense, the National Institutes of Health, the Alfred P. Sloan Foundation, and the Robert Wood Johnson Foundation, among others, all contain programs that provide financial resources to promote the accumulation of knowledge by extramural researchers in topic areas relevant to our discipline. Many industries and for-profit companies in the private sector also invest in their own internal research that can contribute to the accumulation of our science's evidentiary base. In addition to providing critical resources, the research agendas of many funders are crafted to promote research on topics of fundamental interest to society and humanity. Given that they represent a key source of support for establishing the relevance of research, funders can thus exert significant influence on what, how, and why research is conducted within the broader scientific enterprise.

Consequently, we believe that research funders should institute and enforce policies that promote greater transparency, rigor, and precision when evaluating and rewarding research proposals. In some respects, funders, particularly those within the federal sector, are already among the leading practitioners and endorsers of practices that support open and robust science. For example, a 2013 memorandum released by the United States Office of Science and Technology Policy directed all federal agencies whose budgets included at least $100 million to establish research and development plans to support increased public access to federally funded research (Holdren, 2013). As of 2016, 16 agencies established requirements for researchers to provide open access to any peer-reviewed publication resulting from funded research within at least 12 months of publication, and a number of these institutions have already created or been integrated into designated public repositories for housing this work. Building off that momentum, many federal funding agencies have also begun instituting infrastructure to support large-scale data storage and sharing agreements as well (Sheehan, 2016). These initiatives represent an impressive and positive step in the direction of fostering robust science. Infrequent participation by I-O psychologists in seeking research funding has likely contributed to our field being less impacted by these changes. However, as pressure to secure external support continues to mount in higher education, we suspect these practices will become more commonly followed by researchers in our discipline as well.

Scientific Tool Providers

The process of science is supported by a large marketplace of tools and software to aid researchers in the production and generation of scientific knowledge. The tools most familiar to researchers include those used to perform statistical analyses and survey hosting platforms, but there are also a growing number of products available for data processing, project management, real-time collaboration, storage, and the dissemination of methods, measures, and data. As these and other technologies are developed and adopted into the scientific enterprise, they have the potential to significantly impact the way in which research is conducted and scientific knowledge is accumulated. Consequently, scientific tool builders can play an important role in shaping the behaviors and expectations related to robust science.

To this end, we believe that scientific tool builders should pursue product designs in which the “default” behavior of a tool facilitates robust scientific practice. The end users of scientific tools will ultimately determine the utility of a particular product and its value for facilitating research. However, some ways in which tools are perceived as useful is if they are easy to use “out of the box,” accomplish their desired tasks accurately and reliably, and allow users to perform tasks essential to high quality research with less effort. Similarly, the capability to store one's measures, data, analytic syntax, manuscript documents, and review letters in a secure repository that can be directly accessed across any journal submission portal can increase the probability of authors, reviewers, and editors promoting more transparent scientific reporting. The democratization of tools (such as the move to open access, statistical programs like R or Python) used in research will also benefit robust science. When more research is conducted on common platforms with identical or similar tools, transparency of methods and analytic techniques will be enhanced, and these conditions will contribute to the ease with which findings may be replicated. By no means should this recommendation be taken as a justification for tempering scientific innovation and the use of novel analytical techniques when and as appropriate. Rather, we see the use of common tools where possible as an aid (not a mandate) for promoting transparency, replicability, and rigor that should be encouraged.

Conclusion

It is difficult to think of many issues more central to the health and identity of I-O psychology as ensuring the credibility of our science. The defining principles and recommended actions for both proximal and distal stakeholders that we have proposed represent a call to action for our scientific enterprise to collectively participate in efforts aimed at enhancing the quality and reliability of the scientific knowledge we produce. Although we have attempted to take a broad perspective on these matters, we recognize there will be disagreement and differing viewpoints about perceptions of the prevalence of detrimental research practices, who are the key stakeholders, what and how changes should be made, or perhaps whether changes are even needed in our field. Alternative and countervailing perspectives on the complex topics we have discussed are welcomed and hoped for; it is only through such debate that actionable consensus can be reached. As alluded to in the opening quotation of this article, though, we hope all will agree that a thriving science requires a culture whose values and behaviors promote trust from those within the scientific community and beyond. Just as there will always be room for improving the empirical research and evidence-based practices we pursue, there will always be avenues for bettering our scientific enterprise.