Integrative design for thought-experiments

Abstract

Integrative experiment design should be extended to thought-experiments. Thought-experiments are closely connected to “real” experiments. They are involved in devising the design space of theories and possible experiments. The latter may be partitioned into experiments to be really performed and mere thought-experiments. The proposed extension of integrative experiment design lends guidance to a more methodical performance of thought-experiments.

There are four reasons why integrative experiment design as developed by Almaatouq et al. should be extended to thought-experiments.

First, while thought-experiments differ from “real” experiments in being performed only mentally (Kuhn, 1977), there is a close connection: “thought experiment is experiment (albeit a limiting case of it)” (Sorensen, 1992, p. 3). One tempting idea is that a thought-experimenter performs an experiment on herself: She runs her own cognitive processes off-line with a certain hypothetical input and then reports the outcome. For instance, in the moral machine experiment (Awad et al., 2018), participants are faced with a certain decision situation. They take the hypothetical situation as an input for running their decision-making process off-line (without actually taking a decision) and report the outcome.

Second, the practice of thought-experimenting is widespread in science, including the social and behavioural sciences. Here, thought-experiments play a role in producing, justifying, and refuting scientific theories. Typical examples from the social sciences are Keynes's beauty contest scenario (Kornberger & Mantere, 2020), DuBois's colour line scenario, or Addams's scenario where only women are allowed to vote (Hill, 2005). Moral machine experiments started from ethical thought-experiments and extended them. Participants were invited to perform a thought-experiment, and their responses were treated as data in a real social experiment. The widespread use of thought-experiments calls for methodological reflection.

Third, the practice of thought-experimenting aggravates the incommensurability problems with real experiments. The limited description of a thought-experiment leaves many details unspecified. Thought-experiments are performed only occasionally, one after the other, without coordinating them with each other and with theorizing. At most they follow the scheme “question → theory → hypothesis → experiment → analysis → revision to theory → repeat,” as for instance Gettier experiments in epistemology (Gettier, 1963; Praëm & Steglich-Petersen, 2015). They are not systematically controlled for variables like implicit biases of the experimenter. There is no systematic variation in their explicit input. Completeness with respect to the target theory is not prioritized. The trolley case and its numerous variations are a clear example of how these problems affect thought-experiments (Dewitt, Fischhoff, & Salin, 2019; Foot, 1967).

These issues are not sufficiently discussed in the literature. The latter addresses what a thought-experiment is (Sorensen, 1992), how thought-experiments are processed (Nersessian, 2007), whether and how they can provide evidence without empirical observation (Brown, 1991), what their function and scope is (Praëm & Steglich-Petersen, 2015), how they relate to arguments (Norton, 2004), and how to formalize them (Dohrn, 2018; Williamson, 2007). If incommensurability problems are discussed, then mostly with a sceptical twist (Machery, 2017). What is lacking is a more constructive methodology for when and how to perform thought-experiments in a coordinated manner that contributes to building and testing theories.

Fourth, thought-experimenting can be a useful or even indispensable device for integrative experiment design. It arguably plays a role in coming up with such a design, and the perspective on thought-experiments naturally supplements the design space.

Having outlined four reasons for giving thought-experiments a role in integrative experiment design, we distinguish two directions of relevance. Thought-experiments are relevant to integrative experiment design and vice versa.

Integrative experiment design proceeds from a research question to a design space of theories and possible experiments with regards to relevant dimensions of independent variables. Thought-experiments are relevant to this procedure.

First, thought-experiments are heuristically useful in ruling out certain theories and focusing on others. For instance, Galilei's thought-experiments ruled out the Aristotelian theory of motion and oriented scientists towards Newtonian mechanics. Moreover, Almaatouq et al. suggest that candidate dimensions of relevant variables are taken “either from the literature or from experience” (target article, sect. 3.1, para. 6). Yet before the advent of integrative design, neither literature nor experiments were guaranteed to systematically survey relevant dimensions. Thought-experimenting provides an efficient heuristic for identifying these dimensions. Take the moral machine experiment: One has to check both variables explicitly fixed (rich vs. poor, old vs. young, etc.) and inexplicit variables that may have an influence on the decision (nationality, education, religion, etc.). One heuristic step in identifying such variables is a mental simulation of their impact on one's own decision making.

Second, integrative design proceeds by identifying a universe of possible experiments in a domain of inquiry. Since not all of these experiments can be really performed, one has to define an order of priority and a coordinating design plan for the experiments to be really performed. Beyond the plan of real experiments, there are those possible experiments in the universe which are dismissed as irrelevant, but there are also those which are relevant but not really performed for various motives: Too complicated, too costly, unethical, or simply too numerous. Sometimes one can reliably anticipate the result of an experiment without having to really perform it. Therefore, among the relevant experiments which are not part of the planned real experiments, one may select those to be performed as thought-experiments. These thought-experiments also should be ordered according to their priority and coordinated with regards to the explicit and implicit parameters to be controlled for.

The last paragraph already shows how integrative experiment design is relevant for the practice of thought-experimenting. Until now, thought-experiments have been largely performed in an anarchic manner. The proposed extension of integrative experiment design lends guidance to performing them methodically. It surveys potential thought-experiments, subjects them to an order of priority, and fixes parameters for explicit variation and parameters to be controlled for. It renders experimental settings comparable and reduces problems with incommensurability.

We note in closing that the anarchic mode of performing thought-experiments may sometimes be useful in playing a disruptive role (Stuart, 2020). Such a role may not be captured by integrative design, but integrative design does not exclude it either.