We lift our title from a joke dating back at least 50 years (Boulding, Reference Boulding1970): A physicist, a chemist, and an economist are stranded on a desert island with only a can of food. The physicist and chemist devise ingenious, discipline-grounded solutions for opening the can: Heat, pressure, force. But their high-abstraction colleague sees a better approach, “Let's think this through systematically – and start by assuming we have a can opener.”
The story comes to mind because we see integrative experiment design (IED) as a high-abstraction idea that is attractive in principle but that will be difficult to put to practical use. We see a tacit assumption underlying IED – namely, behavioral science is a rigorously self-correcting epistemic community regulated by CUDOS norms of science: Communal data-sharing, universalism, disinterestedness, and organized skepticism (Merton, Reference Merton1942/1973). In this optimistic view, the central obstacle to making behavioral science far more cumulative is essentially organizational. Investigators are too individualistic, insisting on pursuing their own trademark concepts and methods, which entails setting up false binary oppositions (playing 20 questions with nature) from which their side emerges victorious. If only we could subordinate rambunctious scientific egos to the greater epistemic good – integrative experiment design – rapid progress would follow.
We agree with the authors’ criticisms that behavioral science suffers from a validity crisis (in our view, a more devastating problem for behavioral scientists’ collective credibility than the better-known replication crisis). There are countless contradictory claims in the literature and no means of reconciling them because different research teams rely on different methods to study similar phenomena (Clark, Costello, Mitchell, & Tetlock, Reference Clark, Costello, Mitchell and Tetlock2022). Excessive individualism, however, is not the only problem; excessive conformity is as well. Truly thorough “research cartography,” or mapping out a comprehensive design space for a phenomenon requires investigators to engage with theories that seem to contradict their own previously published work, with variables that fall far outside their area of expertise, and with deeply dissonant possibilities (Tetlock, Reference Tetlock1994; Tetlock, Kristel, Elson, Green, & Lerner, Reference Tetlock, Kristel, Elson, Green and Lerner2000). But for a variety of personal, social, theoretical, and ideological reasons, investigators often balk at even considering certain categories of hypotheses (Clark & Winegard, Reference Clark and Winegard2020).
These pockets of collective closed-mindedness will bias – sometimes severely bias – the design space. Consider studies of poverty or educational attainment in which many investigators are unwilling to consider behavioral and genetic explanations (Harden, Reference Harden2021), studies of gender differences in which evolutionary hypotheses are taboo (Buss & von Hippel, Reference Buss and von Hippel2018), or studies of team dynamics in which investigators are reluctant to report results that cast doubt on the benefits of demographic diversity (Clark et al., Reference Clark, Fjeldmark, Lu, Baumeister, Ceci, German and Tetlock2023; Eagly, Reference Eagly2016). The list of “off limits” – yet perfectly plausible – explanations for many of the most societally important topics in behavioral science is long. And these tend to be the precise topics where scholars are most at loggerheads and thus most in need of progress.
Unlike cartography of the physical world, abstract spaces in social science cannot always be clearly identified and measured, and this ambiguity makes it easy for IED teams to leave out the strongest challenges to their pet theories and ignore socially costly hypotheses. This is especially true if IED teams are relatively homogenous in their theoretical orientations.
This is a challenging problem, and we doubt any big idea will solve it. However, not to let perfection be the enemy of improvement, we propose that IED will be most productive in the context of adversarial collaborations, in which teams of collaborators include scholars who have previously published from multiple competing theoretical perspectives (Clark & Tetlock, Reference Clark, Tetlock, Frisby, Redding, O’Donohue and Lilienfeld2023; Kahneman, Reference Kahneman2003). Traditionally, adversarial collaborations include pairs of disagreeing scholars (e.g., Abele, Ellemers, Fiske, Koch, & Yzerbyt, Reference Abele, Ellemers, Fiske, Koch and Yzerbyt2021; Killingsworth, Kahneman, & Mellers, Reference Killingsworth, Kahneman and Mellers2023; Mellers, Hertwig, & Kahneman, Reference Mellers, Hertwig and Kahneman2001), but adversarial-collaboration IEDs could include scholars from multiple or even dozens of formerly competing perspectives who study similar phenomena, such as poverty, educational attainment, or violence.
An adversarial approach helps address the problem of motivation. Many scholars are ambitious and want their scientific contributions to be distinctive, novel, important, and widely generalizable, and consequently, they lack the motivation to articulate a thorough design space. Indeed, scholars are often aware of alternative but equally relevant independent variables, dependent variables, and contexts from those they routinely test to support their theories, and they choose to avoid or file drawer these alternative approaches. Requiring scholars to work with theoretical adversaries would increase the likelihood that the research design space includes relevant parameters that might be rejected by or simply unknown to a team of theoretically homogeneous scholars. This would also help narrow the design space to the most relevant and high-quality parameters by eliminating those that a subset of the research team considers fatally flawed (thus increasing the feasibility of IEDs).
Additionally, our proposed approach could normalize explicit consideration of taboo and other alternative explanations and explicit inclusion of scholars who forward alternative conclusions. Adversarial-collaboration IEDs may be considered incomplete, or lopsided, or biased without relatively exhaustive sampling from relevant parameters and the scholars with expertise in those parameters. Although current norms of science threaten scholars with ostracism and other social sanctions for considering alternative conclusions and affiliating with the scholars who forward them, adversarial IEDs could require, and thus incentivize this more disinterested and sedulous approach to scholarship (Nemeth, Brown, & Rogers, Reference Nemeth, Brown and Rogers2001).
Adversarial IEDs may also motivate the search for genuine metatheories that can explain apparent discrepancies between leading scholars’ preferred theories. Rather than pitting seemingly contradictory hypotheses against one another in a “winner takes all” model of science (e.g., are political rightists more cognitively rigid than leftists or is cognitive rigidity symmetrical?), adversarial IEDs could lead to the development of metatheories that explain in which contexts different claims are true (Bowes et al., Reference Bowes, Clark, Conway, Costello, Osborne, Tetlock and van Prooijen2023). Over time, this could contribute to a more cooperative (and less acrimonious) scientific environment, in which intellectual adversaries are viewed less as enemies to be demolished than as colleagues in pursuit of truth.
We lift our title from a joke dating back at least 50 years (Boulding, Reference Boulding1970): A physicist, a chemist, and an economist are stranded on a desert island with only a can of food. The physicist and chemist devise ingenious, discipline-grounded solutions for opening the can: Heat, pressure, force. But their high-abstraction colleague sees a better approach, “Let's think this through systematically – and start by assuming we have a can opener.”
The story comes to mind because we see integrative experiment design (IED) as a high-abstraction idea that is attractive in principle but that will be difficult to put to practical use. We see a tacit assumption underlying IED – namely, behavioral science is a rigorously self-correcting epistemic community regulated by CUDOS norms of science: Communal data-sharing, universalism, disinterestedness, and organized skepticism (Merton, Reference Merton1942/1973). In this optimistic view, the central obstacle to making behavioral science far more cumulative is essentially organizational. Investigators are too individualistic, insisting on pursuing their own trademark concepts and methods, which entails setting up false binary oppositions (playing 20 questions with nature) from which their side emerges victorious. If only we could subordinate rambunctious scientific egos to the greater epistemic good – integrative experiment design – rapid progress would follow.
We agree with the authors’ criticisms that behavioral science suffers from a validity crisis (in our view, a more devastating problem for behavioral scientists’ collective credibility than the better-known replication crisis). There are countless contradictory claims in the literature and no means of reconciling them because different research teams rely on different methods to study similar phenomena (Clark, Costello, Mitchell, & Tetlock, Reference Clark, Costello, Mitchell and Tetlock2022). Excessive individualism, however, is not the only problem; excessive conformity is as well. Truly thorough “research cartography,” or mapping out a comprehensive design space for a phenomenon requires investigators to engage with theories that seem to contradict their own previously published work, with variables that fall far outside their area of expertise, and with deeply dissonant possibilities (Tetlock, Reference Tetlock1994; Tetlock, Kristel, Elson, Green, & Lerner, Reference Tetlock, Kristel, Elson, Green and Lerner2000). But for a variety of personal, social, theoretical, and ideological reasons, investigators often balk at even considering certain categories of hypotheses (Clark & Winegard, Reference Clark and Winegard2020).
These pockets of collective closed-mindedness will bias – sometimes severely bias – the design space. Consider studies of poverty or educational attainment in which many investigators are unwilling to consider behavioral and genetic explanations (Harden, Reference Harden2021), studies of gender differences in which evolutionary hypotheses are taboo (Buss & von Hippel, Reference Buss and von Hippel2018), or studies of team dynamics in which investigators are reluctant to report results that cast doubt on the benefits of demographic diversity (Clark et al., Reference Clark, Fjeldmark, Lu, Baumeister, Ceci, German and Tetlock2023; Eagly, Reference Eagly2016). The list of “off limits” – yet perfectly plausible – explanations for many of the most societally important topics in behavioral science is long. And these tend to be the precise topics where scholars are most at loggerheads and thus most in need of progress.
Unlike cartography of the physical world, abstract spaces in social science cannot always be clearly identified and measured, and this ambiguity makes it easy for IED teams to leave out the strongest challenges to their pet theories and ignore socially costly hypotheses. This is especially true if IED teams are relatively homogenous in their theoretical orientations.
This is a challenging problem, and we doubt any big idea will solve it. However, not to let perfection be the enemy of improvement, we propose that IED will be most productive in the context of adversarial collaborations, in which teams of collaborators include scholars who have previously published from multiple competing theoretical perspectives (Clark & Tetlock, Reference Clark, Tetlock, Frisby, Redding, O’Donohue and Lilienfeld2023; Kahneman, Reference Kahneman2003). Traditionally, adversarial collaborations include pairs of disagreeing scholars (e.g., Abele, Ellemers, Fiske, Koch, & Yzerbyt, Reference Abele, Ellemers, Fiske, Koch and Yzerbyt2021; Killingsworth, Kahneman, & Mellers, Reference Killingsworth, Kahneman and Mellers2023; Mellers, Hertwig, & Kahneman, Reference Mellers, Hertwig and Kahneman2001), but adversarial-collaboration IEDs could include scholars from multiple or even dozens of formerly competing perspectives who study similar phenomena, such as poverty, educational attainment, or violence.
An adversarial approach helps address the problem of motivation. Many scholars are ambitious and want their scientific contributions to be distinctive, novel, important, and widely generalizable, and consequently, they lack the motivation to articulate a thorough design space. Indeed, scholars are often aware of alternative but equally relevant independent variables, dependent variables, and contexts from those they routinely test to support their theories, and they choose to avoid or file drawer these alternative approaches. Requiring scholars to work with theoretical adversaries would increase the likelihood that the research design space includes relevant parameters that might be rejected by or simply unknown to a team of theoretically homogeneous scholars. This would also help narrow the design space to the most relevant and high-quality parameters by eliminating those that a subset of the research team considers fatally flawed (thus increasing the feasibility of IEDs).
Additionally, our proposed approach could normalize explicit consideration of taboo and other alternative explanations and explicit inclusion of scholars who forward alternative conclusions. Adversarial-collaboration IEDs may be considered incomplete, or lopsided, or biased without relatively exhaustive sampling from relevant parameters and the scholars with expertise in those parameters. Although current norms of science threaten scholars with ostracism and other social sanctions for considering alternative conclusions and affiliating with the scholars who forward them, adversarial IEDs could require, and thus incentivize this more disinterested and sedulous approach to scholarship (Nemeth, Brown, & Rogers, Reference Nemeth, Brown and Rogers2001).
Adversarial IEDs may also motivate the search for genuine metatheories that can explain apparent discrepancies between leading scholars’ preferred theories. Rather than pitting seemingly contradictory hypotheses against one another in a “winner takes all” model of science (e.g., are political rightists more cognitively rigid than leftists or is cognitive rigidity symmetrical?), adversarial IEDs could lead to the development of metatheories that explain in which contexts different claims are true (Bowes et al., Reference Bowes, Clark, Conway, Costello, Osborne, Tetlock and van Prooijen2023). Over time, this could contribute to a more cooperative (and less acrimonious) scientific environment, in which intellectual adversaries are viewed less as enemies to be demolished than as colleagues in pursuit of truth.
Competing interest
None.