I contest Almaatouq et al.'s claim that the integrative experiment design can help “develop a reliable, cohesive, and cumulative theoretical understanding” (target article, sect. 3.3, para. 1). This claim relies on three assumptions which I will challenge.
Assumption 1: Experiments in social and behavioral sciences test theories.
Challenge: Theory tests are, in fact, rare.
The authors assume that statistical null hypothesis tests in social and behavioral sciences involve a theoretical prediction, but this assumption has been widely challenged (e.g., Meehl, Reference Meehl1967, Reference Meehl1978, Reference Meehl1990). Recent scholarship in psychological science suggests that there is a theory crisis (e.g., Eronen & Bringmann, Reference Eronen and Bringmann2021; Muthukrishna & Henrich, Reference Muthukrishna and Henrich2019; Oberauer & Lewandowsky, Reference Oberauer and Lewandowsky2019; Oude Maatman, Reference Oude Maatman2021; Proulx & Morey, Reference Proulx and Morey2021; Robinaugh, Haslbeck, Ryan, Fried, & Waldorp, Reference Robinaugh, Haslbeck, Ryan, Fried and Waldorp2021; van Rooij, Reference van Rooij2019), rather than the “increasing theoretical maturity” (target article, sect. X, para. X) that the target article claims. Perhaps theoretical amnesia (Borsboom, Reference Borsboom2013) can explain this discord, where researchers can no longer tell what a theory is, and statistical models occupy the vacuum created by the absence of theories. Navarro (Reference Navarro2021) and Gelman (Reference Gelman2022) warn us against mistaking statistical models and inferences for scientific theories, just as Gigerenzer (Reference Gigerenzer1998) did a few decades earlier, pointing out that most statistical hypotheses being tested correspond to misleading surrogates for theory rather than genuine theoretical predictions. Almaatouq et al. provide no evidence to convince us otherwise, and their approach is more likely to integrate/reconcile such surrogates than to make real theoretical progress.
Assumption 2: Theories can be meaningfully reduced to a set of conditions represented by a set of experimental parameters.
Challenge: Theories are more than effects and their boundary conditions.
In rare cases where a well-specified scientific theory exists, we need to be explicit about what it entails and how it relates to the experiment. Almaatouq et al. assume that theories can be meaningfully captured by a set of (boundary) conditions that define experimental parameters in the design space. This reductionist view of theory overlooks the role of mechanisms, explanations, and understanding in scientific theory, and overemphasizes the mapping between experimental parameters and theory.
There is no universal consensus on what a theory entails (Winther, Reference Winther and Zalta2021) but a rudimentary framework à la Suppes (Reference Suppes and Morgenbesser1967) will serve here. A scientific theory comprises two distinct components: An abstract logical calculus using symbolic representations of a set of propositions, and a set of rules that give the logical calculus empirical content. The formal part of theory is used to represent, explain, and/or to predict an empirical phenomenon (Guest & Martin, Reference Guest and Martin2021). However, a formalism capturing aspects of a phenomenon without offering any mechanistic or causal scientific explanation does not automatically amount to theory (McMullin, Reference McMullin, Psillos and Curd2008; van Rooij & Baggio, Reference Van Rooij and Baggio2021). Theories gain explanatory power by isolating the causes and uncovering the mechanism generating empirical regularities (Craver, Reference Craver2006; Rohrer, Reference Rohrer2018). Yet experiments often aim to discover or confirm “effects” signifying empirical facts without providing an explanation. Per Cummins (Reference Cummins2010), McGurk effect captures a regularity regarding how speech sounds are perceived across senses, however, it cannot elucidate why the observed regularity occurs. As Poincaré (Reference Poincaré and Greenstreet1905) observes: “Science is built up of facts, as a house is built of stones; but an accumulation of facts is no more a science than a heap of stones is a house.” Scientific theories should go beyond accumulating empirical effects and their boundary conditions to inform us about the structure of systems they purport to explain (Van Rooij & Baggio, Reference Van Rooij and Baggio2021).
Using experiments to isolate effects and boundary conditions as a means to test theories would still face the issue of underdetermination of scientific theory by evidence, even if we only needed theories for description and prediction, not explanation (Stanford, Reference Stanford and Zalta2021). We depend on auxiliary assumptions to derive empirical consequences from a theory, and typically these assumptions neither uniquely identify distinct theories nor remain fixed over time. The integrative design purports to sample the set of experimental parameters comprising such auxiliary assumptions regarding experimental paradigm, context, population of interest, measurements, and so on. In fields where theories are rarely precise enough to specify these assumptions, any experimental design would be conceptually removed from the theory it is meant to test. When theory–experiment mapping is weak, we can define multiple empirically equivalent theories that can reasonably account for the observed data. Alternatively, a given dataset can be used to refute or confirm a given theory simply by altering how auxiliary assumptions are related to the theory.
We cannot simply assume that the design space for an integrative experiment effectively and exclusively captures key features of a theory; extensive theoretical development needs to precede explicit mapping of theory and experiment.
Assumption 3: Reconciling inconsistent experimental results may help reconcile incommensurable theories.
Challenge: Incommensurability of scientific theories is not an empirical problem.
The target article uses the term “(in)commensurability” (target article, sect. X, para. X) to characterize apparent inconsistencies or incomparabilities in experimental designs and results, suggesting that integrative design can effectively reconcile or compare incommensurable theories. In the philosophy of science, semantic incommensurability means there is no common measure between theories, and their fundamental concepts cannot be meaningfully translated or logically related to one another (Oberheim & Hoyningen-Huene, Reference Oberheim, Hoyningen-Huene and Zalta2018) while methodological incommensurability involves unattainability of shared, external, neutral methodological standards to perform a comparative evaluation theories (Chang, Reference Chang, Kindi and Arabatzis2012). Both kinds of theoretical incommensurability lead to underdetermination of theory choice. Empirical inconsistency described by the authors does not invoke theoretical incommensurability; rather it points to a lack of properly specified models or uncertainty associated with statistical inferences. True theoretical incommensurability cannot be reconciled with an integrative (or any other) experimental design, by definition. It has even been argued that forced reconciliation among incommensurable theories is not desirable and an independent pluralistic existence is necessary for theoretical progress (Chang, Reference Chang, Kindi and Arabatzis2012).
Integrative experimental design may serve a crucial exploratory role in the scientific landscape, by methodically narrowing down experimental conditions that are necessary to observe a phenomenon. Indeed the notion of systematic exploratory experimentation is not new (Burian, Reference Burian1997; Steinle, Reference Steinle1997) while largely underappreciated. The targeted theoretical aims, however, seem unfeasible if not impossible. There are no empirical shortcuts to theoretical progress.
I contest Almaatouq et al.'s claim that the integrative experiment design can help “develop a reliable, cohesive, and cumulative theoretical understanding” (target article, sect. 3.3, para. 1). This claim relies on three assumptions which I will challenge.
Assumption 1: Experiments in social and behavioral sciences test theories.
Challenge: Theory tests are, in fact, rare.
The authors assume that statistical null hypothesis tests in social and behavioral sciences involve a theoretical prediction, but this assumption has been widely challenged (e.g., Meehl, Reference Meehl1967, Reference Meehl1978, Reference Meehl1990). Recent scholarship in psychological science suggests that there is a theory crisis (e.g., Eronen & Bringmann, Reference Eronen and Bringmann2021; Muthukrishna & Henrich, Reference Muthukrishna and Henrich2019; Oberauer & Lewandowsky, Reference Oberauer and Lewandowsky2019; Oude Maatman, Reference Oude Maatman2021; Proulx & Morey, Reference Proulx and Morey2021; Robinaugh, Haslbeck, Ryan, Fried, & Waldorp, Reference Robinaugh, Haslbeck, Ryan, Fried and Waldorp2021; van Rooij, Reference van Rooij2019), rather than the “increasing theoretical maturity” (target article, sect. X, para. X) that the target article claims. Perhaps theoretical amnesia (Borsboom, Reference Borsboom2013) can explain this discord, where researchers can no longer tell what a theory is, and statistical models occupy the vacuum created by the absence of theories. Navarro (Reference Navarro2021) and Gelman (Reference Gelman2022) warn us against mistaking statistical models and inferences for scientific theories, just as Gigerenzer (Reference Gigerenzer1998) did a few decades earlier, pointing out that most statistical hypotheses being tested correspond to misleading surrogates for theory rather than genuine theoretical predictions. Almaatouq et al. provide no evidence to convince us otherwise, and their approach is more likely to integrate/reconcile such surrogates than to make real theoretical progress.
Assumption 2: Theories can be meaningfully reduced to a set of conditions represented by a set of experimental parameters.
Challenge: Theories are more than effects and their boundary conditions.
In rare cases where a well-specified scientific theory exists, we need to be explicit about what it entails and how it relates to the experiment. Almaatouq et al. assume that theories can be meaningfully captured by a set of (boundary) conditions that define experimental parameters in the design space. This reductionist view of theory overlooks the role of mechanisms, explanations, and understanding in scientific theory, and overemphasizes the mapping between experimental parameters and theory.
There is no universal consensus on what a theory entails (Winther, Reference Winther and Zalta2021) but a rudimentary framework à la Suppes (Reference Suppes and Morgenbesser1967) will serve here. A scientific theory comprises two distinct components: An abstract logical calculus using symbolic representations of a set of propositions, and a set of rules that give the logical calculus empirical content. The formal part of theory is used to represent, explain, and/or to predict an empirical phenomenon (Guest & Martin, Reference Guest and Martin2021). However, a formalism capturing aspects of a phenomenon without offering any mechanistic or causal scientific explanation does not automatically amount to theory (McMullin, Reference McMullin, Psillos and Curd2008; van Rooij & Baggio, Reference Van Rooij and Baggio2021). Theories gain explanatory power by isolating the causes and uncovering the mechanism generating empirical regularities (Craver, Reference Craver2006; Rohrer, Reference Rohrer2018). Yet experiments often aim to discover or confirm “effects” signifying empirical facts without providing an explanation. Per Cummins (Reference Cummins2010), McGurk effect captures a regularity regarding how speech sounds are perceived across senses, however, it cannot elucidate why the observed regularity occurs. As Poincaré (Reference Poincaré and Greenstreet1905) observes: “Science is built up of facts, as a house is built of stones; but an accumulation of facts is no more a science than a heap of stones is a house.” Scientific theories should go beyond accumulating empirical effects and their boundary conditions to inform us about the structure of systems they purport to explain (Van Rooij & Baggio, Reference Van Rooij and Baggio2021).
Using experiments to isolate effects and boundary conditions as a means to test theories would still face the issue of underdetermination of scientific theory by evidence, even if we only needed theories for description and prediction, not explanation (Stanford, Reference Stanford and Zalta2021). We depend on auxiliary assumptions to derive empirical consequences from a theory, and typically these assumptions neither uniquely identify distinct theories nor remain fixed over time. The integrative design purports to sample the set of experimental parameters comprising such auxiliary assumptions regarding experimental paradigm, context, population of interest, measurements, and so on. In fields where theories are rarely precise enough to specify these assumptions, any experimental design would be conceptually removed from the theory it is meant to test. When theory–experiment mapping is weak, we can define multiple empirically equivalent theories that can reasonably account for the observed data. Alternatively, a given dataset can be used to refute or confirm a given theory simply by altering how auxiliary assumptions are related to the theory.
We cannot simply assume that the design space for an integrative experiment effectively and exclusively captures key features of a theory; extensive theoretical development needs to precede explicit mapping of theory and experiment.
Assumption 3: Reconciling inconsistent experimental results may help reconcile incommensurable theories.
Challenge: Incommensurability of scientific theories is not an empirical problem.
The target article uses the term “(in)commensurability” (target article, sect. X, para. X) to characterize apparent inconsistencies or incomparabilities in experimental designs and results, suggesting that integrative design can effectively reconcile or compare incommensurable theories. In the philosophy of science, semantic incommensurability means there is no common measure between theories, and their fundamental concepts cannot be meaningfully translated or logically related to one another (Oberheim & Hoyningen-Huene, Reference Oberheim, Hoyningen-Huene and Zalta2018) while methodological incommensurability involves unattainability of shared, external, neutral methodological standards to perform a comparative evaluation theories (Chang, Reference Chang, Kindi and Arabatzis2012). Both kinds of theoretical incommensurability lead to underdetermination of theory choice. Empirical inconsistency described by the authors does not invoke theoretical incommensurability; rather it points to a lack of properly specified models or uncertainty associated with statistical inferences. True theoretical incommensurability cannot be reconciled with an integrative (or any other) experimental design, by definition. It has even been argued that forced reconciliation among incommensurable theories is not desirable and an independent pluralistic existence is necessary for theoretical progress (Chang, Reference Chang, Kindi and Arabatzis2012).
Integrative experimental design may serve a crucial exploratory role in the scientific landscape, by methodically narrowing down experimental conditions that are necessary to observe a phenomenon. Indeed the notion of systematic exploratory experimentation is not new (Burian, Reference Burian1997; Steinle, Reference Steinle1997) while largely underappreciated. The targeted theoretical aims, however, seem unfeasible if not impossible. There are no empirical shortcuts to theoretical progress.
Acknowledgments
I would like to thank Esther Mondragón and Erkan O. Buzbas for their insightful feedback.
Financial support
This work was supported by the National Institute of General Medical Sciences of the National Institutes of Health (Award No. P20GM104420).
Competing interest
None.