Representative design: A realistic alternative to (systematic) integrative design

Abstract

We disagree with Almaatouq et al. that no realistic alternative exists to the “one-at-a-time” paradigm. Seventy years ago, Egon Brunswik introduced representative design, which offers a clear path to commensurability and generality. Almaatouq et al.'s integrative design cannot guarantee the external validity and generalizability of results which is sorely needed, while representative design tackles the problem head on.

We share Almaatouq et al.'s concerns with the lack of commensurability and generalizability of experimental findings in the social and behavioural sciences. However, we disagree that a “lack of any realistic alternative” (target article, sect. 3, para. 1) existed, which prompted them to propose integrative design. Over 70 years ago, Egon Brunswik (1956b, p. 159) saw “intrinsic shortcomings” in “artificial, systematic [experimental] designs” regardless of whether or not these designs were implemented “one-at-a-time” (word in square brackets added). He proposed representative design as an alternative. This lays a path towards commensurability and generality as well as a clear vision for theoretically and practically valuable research in psychology.

Brunswik (1944, 1955b, 1956a) questioned the ability of systematic design to yield internally and externally valid results. He argued that variables may be artificially “tied” or “untied,” thus making it impossible to rule out the effect of the confound in the former case and making it impossible to study human functioning in a generalizable way in the latter case. His alternative, representative design retains the “causal texture of the environment” to which the human has adapted and to which the researcher intends to generalize (see Dhami, Hertwig, & Hoffrage, 2004, for a review). For Brunswik, the effect of specific variables should be disentangled at the data analysis rather than data collection stage. By contrast, Almaatouq et al. appear to accept systematic design and only critique its “one-at-a-time” implementation, arguing that results are difficult to compare, aggregate, and generalize. However, their solution to this problem suffers from the same limitations that Brunswik identified with systematic design.

Almaatouq et al.'s notion of the “design space” essentially comprises a large series of environments (combinations of various variables) from countless one-at-a-time experiments. As Brunswik (1955a, 1955b) noted, these will potentially include, at best, environments which are rarely encountered, and most likely, environments that do not (or cannot) exist in the real world. While Almaatouq et al. appear to accept Brunswik's view that the generalizability over situations is equally, if not more, important than that over participants, they fail to recognize the importance of representative stimulus sampling (and construction; see Hammond, 1966). There is no way to know which environments in the design space are representative and which are not. Instead, Almaatouq et al. are preoccupied with reconciling, replicating, or even opening the “file drawer” (target article, sect. 5.2, para. 4) of experimental studies that may lack generality because they were obtained under unrepresentative conditions.

Almaatouq et al. applaud Peterson, Bourgin, Agrawal, Reichman, and Griffiths's (2021) efforts to sample the “space of possible experiments [i.e., gambles] much more densely” (target article, sect. 4.2, para. 2) than before. Yet, they do not question the representativeness of the gambles studied and so the generalizability of the findings remain unknown. Brunswik's representative design (1956a, 1956b) on the contrary, tackles the problem directly; researchers must first define the “reference class” or “universe” of stimuli (tasks/situations, e.g., gambles) about which they want to draw a generalizable conclusion. One then either explicitly samples stimuli from this predefined set or constructs stimuli representative of it. One example where representative design has cast serious doubt over well-established conclusions based on systematic design is given by Juslin, Winman, and Olsson (2000) on the overconfidence phenomenon (for other examples, see Dhami et al., 2004). Representative design can also avoid potential pitfalls of Almaatouq et al.'s method such as the need to configure a “correct” or “relevant” design space, prioritization of aspects of the space, and keeping the number of possible experiments to a manageable level. Additionally, the use of representative design can be facilitated by virtual reality, and is not hampered by the need for large participants’ pools (since each individual performs multiple trials and data are analysed at the individual level). Simply stated, integrative design cannot guarantee the external validity and generalizability of results which the social and behavioural sciences sorely need, while representative design tackles the problem head on.

To us, the crux of the problem that ails the social and behavioural sciences, which Almaatouq et al. do not address, is: What is the overall goal? Indeed, before any researcher embarks upon designing a study, let alone a paradigm shift in doing research, one ought to consider what their goal is. For Brunswik, the method followed his goal. He envisioned psychology as a science of “organism–environment relationships” (Brunswik, 1943), and he provided the “lens model” framework (Brunswik, 1952, 1955a) for theoretically delineating how individuals are adapted to the environments in which they function (termed probabilistic functionalism). Note that we are not calling for an outright rejection of systematic design, but for it to be contextualized within representative design. In our view, psychology need not have one single goal or method, but we do agree with Brunswik that one's method should follow one's goal, and that generalizability is important.

Representatively designed experiments can reveal how humans are adapted to their environments. Experiments which then alter specific environmental properties can demonstrate how these adaptation processes are challenged. Thus, representative design requires researchers to delineate environmental properties to understand human environments – something researchers in the social and behavioural sciences rarely do, not even to determine the generality of an existing set of results. Understanding human cognition and behaviour as a function of environmental properties is also highly relevant for practically applicable research, and funding bodies and universities are increasingly rewarding researchers whose findings have impact, thereby providing further incentive for representative design.

In sum, Brunswik was ahead of his time in recognizing that systematic design means that researchers would need to be satisfied by “plausibility generalizations, … always precarious in nature – or [be] satisfied with results confined to a self-created ivory tower ecology” (1956b, p. 110). He provided a methodological solution to this problem, and a clear theoretical ambition. Unfortunately, his ideas have been largely ignored, forgotten, misunderstood, or even ridiculed (for a history and discussion, see Hammond, 1998; Holleman, Hooge, Kemner, & Hessels, 2020, 2021). By missing the opportunity to build on representative design, Almaatouq et al. themselves contribute to what they see as a fundamental problem in today's social and behavioural sciences, that is, not “putting things together.”