The miss of the framework

Abstract

The authors rightly critique existing social sciences approaches. However, they are too quick to dismiss the criticism that their proposed paradigm is atheoretical. Social and cognitive theories are indeed incommensurate, often due to the lack of a unifying framework. Without proper integration with theoretical frameworks, their proposal may merely produce a resource-intensive veneer of thoroughness without substantive improvements to understanding.

The authors have produced a valuable and timely critique of widespread approaches to social science research, and I found much to agree with in their essay. I agree with their claim that many problems in science are not solved by replicability, nor by any methods that improve the reliability of experiments (though these measures are still valuable, as reliability of results is a necessary but insufficient condition for robust science). I agree that experiments must be better integrated with theory, and that the cumulative advance of theoretical explanations is a fundamental goal of science (even if other goals can also exist simultaneously). And I agree that coherence across results and experiments is critical, and troublingly lacking in much of the social sciences (Smaldino, 2019). Nevertheless, I find their approach to theory development to be a bit hasty.

The authors toe a messy line in their critique of the “one-at-a-time” approach. Of course, all scientific explanations leave out large swaths of the complexity of real life. As von Uexküll (1921) noted over a century ago, it is only by doing “violence to reality” that science is possible. All scientific theories decompose their target systems into an artificial set of parts, properties and relationships. The trouble, in my view, comes not from trying to construct theories about social systems, but from overconfidence that the particular decomposition associated with a particular theory constitutes a satisfactory explanation of phenomena.

For the purpose of further elaboration, allow me to propose a distinction between hypotheses, theories, and theoretical frameworks (taken from Smaldino, 2023). A hypothesis is a prediction that if a particular set of assumptions are met, a particular set of consequences will follow. It is easy to see how problems arise if hypotheses are tested in isolation. A theory is a set of assumptions upon which hypotheses derived from that theory must depend. Strong theories allow us to generate clear and falsifiable hypotheses. However, different theories may decompose reality in different ways and may address qualitatively different questions about a particular system, making comparisons of competing theories challenging. A theoretical framework is a broad collection of related theories that all share a common set of core assumptions. An example of a theoretical framework is Darwinian evolution by natural selection, from which many subordinate theories have been derived. A robust framework provides the conditions for the accumulation of scientific understanding, because consistency between related theories must be constantly assessed. I think it is fair to say that there is not currently a single dominant framework for the social sciences. One likely reason is that there have been few incentives to develop one. Indeed, there may have been active selection against proclivities to do so, as that pursuit rarely leads to easily measured success in the increasingly cutthroat game of academic science. A single framework may also be undesirable, as it may preclude useful decompositions needed for certain theories (contra Popper, 1994).

The “integrative” approach proposed by the authors falls short in its overreliance on data and its dismissal of the importance of mechanistic or generative explanations. The approach provided does try to draw consistency across experiments, and this is laudable. But it underplays the value of consistent theoretical framework. This is demonstrated most clearly by the authors' implication that an interpretable, mechanistic model is essentially equivalent to a “surrogate model,” which is able to generate data that look like those collected empirically while remaining agnostic to similarities in the data-generating processes. I find this implication troubling. One reason is aesthetic – it is more satisfying to have a realistic explanation for a process than to simply produce an alternative process that generates similar outcomes. If the only objection were aesthetic, it would be easy to dismiss as mere preference. But the distinction is actually much more serious than this. A model that accurately represents the mechanisms that generate data is necessarily robust to changes in the contextual conditions under which the data are generated. This is because the assumptions of the model accurately map onto the conditions of the real world (within reason – all maps are ultimately imprecise). So the model can therefore be adjusted to match the new conditions, or at least will help us to identify the data needed to revise the model to match those conditions. A surrogate model, on the contrary, cannot do this, because the mapping between the model assumptions and the real world is fundamentally inaccurate. Consider how financial models failed to predict the economic crash of 2008, because their models were not mechanistic and therefore relied on correlations which suddenly failed to hold (this was not their only failure).

Thankfully, there are already theoretical frameworks that underpin some robust, testable, and coherent theories of human behavior. These include cultural evolution (Boyd & Richerson, 1985; Cavalli-Sforza & Feldman, 1981; Mesoudi, 2011) and human behavioral ecology (Nettle, Gibson, Lawson, & Sear, 2013; Smith & Winterhalder, 1992), which draw on insights from biological theories of evolution and ecology, as well as from related work in microeconomics and game theory. These frameworks give us good prior reasons for incorporating certain assumptions into our theories while excluding others, because they relate to fundamental aspects of social life, such as the presence or absence of particular social learning biases or the use of prosocial norms as mechanisms for dealing with uncertainty and risk. One advantage of these frameworks is that they do not discard, as many other approaches do, the troves of knowledge we have acquired about nonhuman species. Since humans are, after all, also animals, we are subject to many similar constraints and affordances that occur in other species. Consistency with these data is too often overlooked.