Triangulation, incommensurability, and conditionalization

Abstract

We present a new justification for methodological triangulation (MT), the practice of using different methods to support the same scientific claim. Unlike existing accounts, our account captures cases in which the different methods in question are associated with, and rely on, incommensurable theories. Using a nonstandard Bayesian model, we show that even in such cases, a commitment to the minimal form of epistemic conservatism, captured by the rigidity condition that stands at the basis of Jeffrey’s conditionalization, supports the practice of MT.

Introduction

Methodological triangulation (MT) is the practice of using different methods to evidentially support the same scientific claim. In the literature, different justifications for this practice were presented. Each of these justifications can be viewed as explicating a different sense in which MT can be epistemically valuable.

In this article, we present a new account for the epistemic value of MT which, we argue, manages to overcome some of the limitations of the existing accounts. However, our account should not be viewed as a rival account to the accounts found in the literature but rather as a complementary one. It explicates a different sense in which MT can be epistemically valuable and mainly applies to types of cases not covered by other accounts. Unlike other accounts, and more generally unlike other explanations one can find in the literature for the epistemic value of having a diverse set of evidence, our account mainly aims to capture cases in which the methods in question, and the types of evidence they provide, rely on partly incommensurable theories that, nevertheless, give predictions regarding the same phenomenon.

This type of case is quite common when it comes to evidence-based policy, so it is crucial to understand whether, and under which conditions, the claim that MT is epistemically valuable applies to it. Consider, for example, two studies, one in economics and one in sociology, that support the same hypothesis regarding the behavior of some social institution that is of interest to decision makers for public policy purposes. The economic study might be (as many economic studies are) theoretically committed to some version of rational choice theory (e.g., the empirical evidence it presents can be taken as evidence for the hypothesis in question only while relying on rational choice theory) while the sociological study might (as many sociological studies do) adopt a functionalist perspective. How should the mere fact that both studies, which rely on entirely different assumptions and employ different conceptual frameworks, support the hypothesis influence the degree of confidence a rational decision maker, whose decision relies on whether the hypothesis is true, assigns to it?

Some philosophers, such as Beach and Kaas (2020) and Runhardt (2021), have recently argued that MT has no epistemic value in such cases. We show that their arguments do not apply to the sense of the epistemic value of MT that our account explicates. According to our account, under plausible conditions, the mere agreement between the two studies should increase the decision maker’s degree of belief in the hypothesis, even though the two studies rely on different assumptions.

Adopting a (nonstandard) Bayesian point of view, we model such cases using different probability distributions that are defined over partly overlapping algebras. In the interpretation, each probability distribution represents a rational agent’s credence function when adopting the point of view of a different scientific theory. The agent, we assume, takes all theories to be reliable and still takes them to be incommensurable (this is why they are defined over algebras that are only partly overlapping). ¹

In our model, all the theories in question assign a probability value to some hypothesis, H, both before and after using the (different) methods associated with them to examine whether H is true. However, none of the theories assigns a probability value to the claim that one of the studies associated with one of the other theories predicts H with a given probability. We argue that in many such cases, it is natural to demand that after learning that a method associated with a given theory supports H to a given degree, a rational agent should not change her conditional credence in the reliability of other methods that are associated with other theories (incommensurable with the theory in question), given H. In our model, this last assumption, the rigidity assumption, amounts to a commitment to using Jeffrey’s conditionalization as an updating rule. We show that this leads to the conclusion that when all methods are reliable, rational credence in H increases with the number of methods used. We also provide a general formula for computing this credence and show that the order of using different methods does not change the agent’s final credence.

The simple formal model we use to develop our account is quite abstract. Still, we argue, it captures and justifies a common type of scientific practice important for public policy purposes. We demonstrate that by exploring the example of the research in criminology regarding different causal mechanisms that lead to women’s incarceration.

The relevant scientific literature identifies three different pathways to women’s incarceration. Some of the measures used in the research regarding each of these pathways are endogenous to the pathways. Thus, it is not always possible to assess the causal effect of one pathway given another. We argue that this should be understood as a type of partial incommensurability between the theories describing each pathway. Thus, our account can capture this case.

Specifically, we suggest that even though no statistical evidence regarding the effect of one pathway given another is available, the fact that a given woman follows more than one pathway should increase one’s confidence in the woman’s future incarceration. We point to a possible implication of this claim to decisions regarding resource allocations.

The rest of this article is organized in the following way. In Section 1, we review some of the existing accounts for the epistemic value of MT and point to their limitations. Section 2 introduces our example of pathways to women’s incarceration. In Section 3, we provide some necessary background regarding incommensurability, and argue that the different pathways to incarceration are grounded in incommensurable theories. In Section 4, we present our model and its results and discuss the sense in which they support MT. Section 5 concludes. The proofs are in the Appendix.

1. Methodological triangulation: Existing accounts

Let us call the claim that MT is, at least in some cases, epistemically valuable the “methodological triangulation thesis” (MTT). ² There are several ways to understand and justify MTT. As explained, the goal of this article is not to contest any of the existing accounts but rather to present a new account for triangulation that applies to cases in which different pieces of evidence support the same hypothesis using incommensurable theories. This section presents the main existing accounts, points to their limitations, and shows that they do not apply to the type of cases we have in mind.

1.1. Independence and robustness arguments

The most common way to justify MTT is by relying on the assumption that different pieces of evidence support the same hypothesis independently. Let us call this the independence assumption. The independence assumption is usually used to justify the robustness of the hypothesis. If multiple independent sources support a hypothesis, then even if one of those sources is unreliable, the hypothesis is still likely to be true. Different accounts in the literature explicate this general idea in different ways, each of which rests on a different interpretation of the independence assumption. ³ However, none of these accounts can support MTT in cases of incommensurability.

As we explain in Section 4, from a formal point of view, the account we present in this paper is closely related to some of the accounts in this family. Specifically, our main assumption, the rigidity assumption (which will be clearly defined in Section 4), plays a role in our account that is formally analogous to the role the independence assumption plays in some of the existing accounts. However, unlike in those accounts, in our account the rigidity assumption is not used to justify robustness (in the same sense).

An early argument for MTT, which explicitly relies on the independence assumption, was presented by Sober (1989). Sober identifies two pieces of evidence as independent if they are connected to the phenomenon in question through different causal routes. In his example (borrowed from Wittgenstein), a newspaper and a radio transmission are described as two independent pieces of evidence for the score of yesterday’s baseball match. In the example, the radio reporter and the newspaper reporter are two different people who have been in the match, and both report the score using different causal routes (unlike, e.g., two copies of the same newspaper which are not independent, as both rely on the same reporter).

Formally, Sober’s independence condition is defined in probabilistic terms. Let $S$ be a proposition that describes the score of the baseball match, ${e_1}$ a proposition that describes the newspaper report, ${e_2}$ a proposition that describes the radio report, and ${I_1}$ and ${I_2}$ propositions that describe the causal routes from S to the two pieces of evidence, respectively. According to Sober, ${e_1}$ is independent of ${e_2}$ If the likelihood of the evidence conditional upon the casual route from S to the evidence is not affected either by the likelihood of the other route or by the likelihood of the hypothesis: ⁴

$$P\left( {{e_1}|{I_1}} \right) = P({e_1}|{I_1}\;\& {I_2})\,{\mkern 1mu} {\rm{and}}{\mkern 1mu} P\left( {{e_1}{\rm{|}}{I_1}} \right) = P({e_1}|{I_1}\;\& S).$$

If the evidence is independent in Sober’s sense, the probability of a hypothesis increases as more reliable evidence supports it. This conclusion is based on the idea of robustness: Sober’s independence condition implies that even if it turns out that the casual route from the hypothesis to ${e_2}$ is unreliable, this does not imply anything about the reliability of ${e_1}$ .

The concept of independent evidence is also employed in accounts based on the Condorcet jury theorem. Based on List and Goodin’s version of the theorem (2001), Heesen et al. (2019) give an argument for MT in a setting in which each method (in a set of at least three methods) gives a binary accept/reject verdict regarding each hypothesis (from a set of at least two hypotheses, one of which—and only one—is true). They show that given two conditions, the chance of accepting the true hypothesis by relying on a majority rule (i.e., accepting the hypothesis that the majority of methods accept and if there is more than one such a hypothesis, then choosing one of them randomly) increases with the number of methods.

The conditions are:

1. 1. Reliability: Each method has a probability higher than 0.5 to accept the true hypothesis.

2. 2. Independence: The conditional probability of one method, M1, to accept a hypothesis, H, given that H is the true hypothesis, is equal to the conditional probability of M1 to accept H, given that H is the true hypothesis and that one of the other methods, M2, accepts H. ⁵

Both Sober’s independence assumption and that of Heesen et al. are explicitly formulated in terms of the methods’ chances, or objective probabilities, to give the correct predictions. This is so, for a good reason. Interpreting the independence assumption as applying to a subjective probability distribution over a set of propositions that includes the hypothesis in question and the predictions of the different methods, is problematic for two reasons.

First, a probability distribution that is defined over such a set must attach a probability value not only to the events of each one of the methods predicting (or not predicting) the hypothesis but also to the conjunctions of such events. While there seems to be nothing problematic with the assumption that an objective probability function (which might be completely inaccessible to the scientists) is defined on such conjunctions, the assumption that scientists’ subjective probability distributions are defined over such conjunctions seems highly unrealistic in many cases (and especially when it comes to evidence-based policy that relies on different studies from the social science). As we discuss in Section 3, when different pieces of evidence support the hypothesis using incommensurable theories, there is—often—no point of view thar a rational agent can adopt that justifeis assigning subjective probability values to such conjunctions.

Second, even in cases in which it makes sense to attribute to scientists a subjective probability distribution that is defined over such a rich set of propositions, the independence assumption seems either arbitrary or based on knowledge regarding the objective probability of the hypothesis and the evidence. To demonstrate, think of the following example.

Suppose I toss two fair dice. Let H, the hypothesis, be “I got a {5,6},” let one piece of evidence, E1, be “the first die fell on 5” and let another piece of evidence, E2, be “the second die fell on 5.” The independence assumption clearly fails in this case: Although each one of the pieces of evidence supports the hypothesis (i.e., the rational conditional degree of belief in H given each one of the pieces of evidence is higher than the unconditional rational degree of belief in H), the conditional rational degree of belief in each one of the pieces of evidence, given the hypothesis and the other piece of evidence is 0 (given that the score is {5,6} and that one die fell on 5, the other die did not).

What makes the independence assumption fail in this case is our knowledge of the relevant objective probabilities (which we have sneaked in the description of the case, using the assumption that the dice are fair). Thus, it is surely not the case that the independence assumption, understood as applying to a subjective probability distribution, is always justified. In cases in which scientists have knowledge regarding the relevant objective probabilities, the independence assumption, if justified, is justified in virtue of this knowledge. In such cases, there is no reason to understand the independence assumption in terms of subjective probability (it might also apply to the scientists’ subjective probability distribution but only in virtue of their knowledge of the relevant objective probabilities). In cases in which knowledge regarding the relevant objective probabilities is unavailable, there seems to be no good reason to adopt a subjective probability distribution that obeys the independence assumption because, as we have just seen, if such knowledge were available the independence assumption might not have been justified. Thus, in such cases, when understood as applied to a subjective probability distribution, the independence assumption seems arbitrary.

One way to understand our account (presented in Section 4) is as an explanation for why, in cases in which different pieces of evidence support a hypothesis through incommensurable theories, adopting the independence assumption, applied to a subjective probability distribution, is—after all—not arbitrary.

When understood as applying to objective probability distributions, independence assumptions are criticized in the literature. The main line of argument against these assumptions is based on the claims that (1) in many cases, the independence assumption fails (as we have just seen in the dice example), and (2) scientists are almost never in a position to know that the pieces of evidence available to them are independent In the relevant way. ⁶

Here we do not aim to show that the independence assumption—understood this way—always fails. It seems plausible that there are cases in which the independence assumption is satisfied. Our account covers, however, also cases in which it fails.

1.2. Reliability likelihood

Heesen et al. (2019) suggest a second sense in which MT is epistemically valuable. We can call this sense “reliability likelihood.” They argue that in some fields, particularly in the social sciences, there are no obviously reliable theories. Thus, in many cases, researchers have no information about which theory is reliable (a phenomenon they refer to as “Du Boisian indifference”). In such cases, they argue, the mere agreement between different methods about which hypothesis is true indicates that at least one of the agreeing methods is reliable.

They capture this idea in a formal model. In their model, there is—again—a set of available methods (at least three) and a set of possible hypotheses (at least two, only one of which is true). The methods give a binary accept/reject verdict regarding each hypothesis, and each method gives one and only one “accept” verdict. They show that if all methods are at least as reliable as a device that chooses a hypothesis randomly and at least one method is more reliable than that, choosing the hypothesis that is supported by the majority of methods is more likely to point to the true hypothesis than choosing a method randomly and accepting the hypothesis to which it points.

There are two main limitations to this account. First, it assumes that all methods are at least as reliable as a device that randomly chooses a hypothesis from the set of available hypotheses. This assumption seems plausible in some scenarios but is less plausible under Du Boisian indifference. Heesen et al. argues that the lowest level of reliability a method can have is the level of a randomizing device because any method with a lower level of reliability (i.e., any method that supports false hypotheses more often than a device that chooses a hypothesis randomly) can be used as a method with a higher level of reliability by interpreting it as supporting not the hypotheses in question but rather their negation. However, this argument depends on the researchers’ ability to identify the methods with a lower-than-random level of reliability. This ability is exactly what is missing in cases of Du Boisian indifference. Thus, Heesen’s et al. justification for their assumption that the lowest possible level of reliability is that of a randomizing device fails exactly in the type of cases in which they are interested.

Still, it seems plausible that in many scientific contexts, one can safely assume that none of the relevant methods is less reliable than a randomizing device, and some are more reliable (but which ones those are, is unknown). At least in such context, Heesen et al.’s argument seems to hold.

The second limitation of this account is, however, more serious. Heesen et al. compare the objective probability of a hypothesis to be true given that it was chosen by a triangulation procedure (which they interpret as choosing the hypothesis picked by the majority of methods and in cases in which there is more than one such a hypothesis, choosing randomly among them) to the objective probability of a hypothesis to be true given that it was picked by a randomly chosen method. This reference point is supposed to follow from the assumption of Du Boisian indifference: When a researcher has no grounds, whatsoever, to prefer relying on one method rather than another, a random choice of a method indeed seems to be the right reference point to adopt.

Their motivation for using the assumption of Du Boisian indifference is their attempt to keep their account free from the “Bayesian Baggage” (Heesen et al. 2019, p. 3076). However, it seems to us that by doing so, they only sneak a strong Bayesian assumption in the back door: Cases in which researchers have no grounds whatsoever to believe one method is more reliable than another are quite rare. Usually, researchers do have some grounds to believe that some methods are more reliable than others. In such cases, the natural reference point would be relying on the method that the researcher is most certain is reliable (or maybe relying on the method that maximizes expected accuracy). However, Heesen et al. cannot use this reference point because, in their model, the researcher’s degrees of belief in the reliability of methods are not represented. ⁷

Unlike Heesen’s et al. account, our account is explicitly Bayesian (albeit nonstandard). We agree with Heesen et al. that this always involves a commitment to some nontrivial assumptions. However, to the extent that by relying on these assumptions, we manage to shed new light on the epistemic value of MT in a family of cases not captured by existing accounts, our account provides indirect evidence for these assumptions.

1.3. Other accounts

Other philosophers (e.g., Schupbach 2018; McEvoy and Richards 2006, and especially Russo and Williamson 2007, that did not argue explicitly in favor MT but did argue in favor of a related thesis they called “methodological pluralism” ⁸ ) have pointed to other roles MT can play in more specific scientific contexts and while referring to specific methods and specific types of evidence (e.g., explanatory roles, roles it can play in the context of discovery, roles that are limited to establishing causal claims).

Unlike these accounts, our account is general and does not depend on the specific details of the scientific context or the methods in question and the types of evidence they provide.

2. Triangulation of gendered pathways to incarceration

To demonstrate the types of scenarios that our model is meant to capture, we examine a line of research concerned with gendered pathways to incarceration. The basic idea is that women who commit offending behavior are likely to experience a distinct set of causal factors leading to this behavior. The idea originated in qualitative research in the fields of psychology, addiction, and social welfare.

Salisbury and van Voorhis (2009) used these studies to identify three main pathways to incarceration, that is, causal mechanisms that lead women to commit offending behavior, resulting in their incarceration. They then used quantitative methods to measure the strength of each one of these mechanisms.

Two of the mechanisms, child victimization and relationship dysfunction, originated in feminist criminology studies. The child victimization mechanism suggests that victimization in childhood leads to mental illnesses, mental illnesses lead to substance abuse, and substance abuse leads to incarceration. The relationship dysfunction mechanism suggests that dysfunctional relationships lead to low self-efficacy, which causes women to be victimized, leading to mental illnesses, substance abuse, and incarceration.

The third mechanism, the social and human capital mechanism, is based on the strong correlations between incarceration and low capital. According to this mechanism, low educational strength and family support cause financial difficulties that limit the set of opportunities open for women, thereby causing them to choose illegal means to pursue their goals.

Salisbury and Van Voorhis assessed the effect of each mechanism on the likelihood of incarceration by utilizing data from the Women’s Needs and Risk Assessment Project in Missouri. The researchers used a path analysis technique, which involves running a series of OLS regressions from the assumed cause to the assumed effect. The advantage of this method is its ability to assess a whole mechanism rather than the correlation between two variables. However, its main disadvantage is that it requires the researchers to assume the causal direction of each mechanism a priori, so it relies heavily on prior qualitative evidence and theoretical background.

The study found that each of the mechanisms studied is a significant predictor of incarceration, so intuitively it seems plausible that women would be more likely to be incarcerated when affected by more mechanisms. However, assessments made using the pathway analysis technique presupposed the variables and the direction of causality. Thus, If the evidence for the operation of the causal variable according to the different mechanisms relies on incommensurable theories (as we explain in the next section), it is impossible to compare their causal effect on the same metric. Consequently, none of the existing accounts of MT can justify the claim that the effect of two mechanisms is stronger than the effect of one.

Having similar cases in mind, some philosophers have recently argued that incommensurability might undermine the value of triangulation. For instance, Runhardt (2021) argued that in some cases, assumptions employed by a theory that supports using a given method might undermine the justification for using another method (supported by another theory). When this is the case, she argued, triangulation seems epistemically useless. Beach and Kaas (2020) and Kuorikoski and Marchionni (2023) argued for similar conclusions.

All these arguments are based on the idea that adopting the assumptions of one theory entails doubting the reliability of another. In our example, for instance, one might argue that to determine the effects of the relational mechanism, we must accept that a woman’s criminal activity is a result of the social condition she is trapped in and not the result of a rational decision in the sense of rational choice theory. However, if we accept this assumption, it is unclear how to assess low social capital’s effect on women’s incarceration-related choices. ⁹ However, as we argue in the following sections, MTT can be justified even in such cases.

3. Justifying incommensurability

We discuss two notions of incommensurability that our model captures, taxonomical and methodological (we rely here on Sankey’s [1994] extensive review of incommensurability). The two notions are not mutually exclusive: Theories may be incommensurable in both ways. We also show that the different mechanisms discussed by Salisbury and Van Voorhis can be seen as grounded in incommensurable theories, according to both notions.

The discussion aims to provide the background needed to justify the main assumption we employ in our model (presented in the next section), the rigidity assumption. The rigidity assumption demands that in cases in which an agent takes two different theories, T1 and T2, to be reliable with respect to some hypothesis, H, but incommensurable with each other, after learning a piece of evidence, E1, associated with theory T1 (but not with T2), although the agent might change her credence in H, from the point of view of T2, she should not change her conditional credence in any piece of evidence, E2, which is associated with T2 (but not with T1) given H.

3.1. Taxonomic incommensurability

The term “incommensurability” was first popularized by Kuhn (1970), but the idea of taxonomical incommensurability precedes Kuhn. It was initially suggested by Feyerabend (1962), who discussed “impetus” as an example of a concept from Aristotelian physics that is untranslatable to the Newtonian paradigm. While both the concepts of “inertia” and “impetus” refer to the force that acts on moving objects, impetus refers specifically to an internal force that originates in the object. Thus, according to Feyerabend, a statement such as “impetus does not exist” lacks a truth value when considered from the point of view of the Newtonian paradigm.

In some places in his writings, Kuhn seems to interpret incommensurability as taxonomical. He argued, for example, that incommensurable theories “cut out the world differently” (Kuhn 1983). Thus, even when they refer to the same phenomenon, some claims that can be made about the phenomenon in one paradigm cannot be translated to another paradigm due to the paradigms’ different conceptual frameworks.

Taxonomical incommensurability is sometimes seen as a type of reference class problem. For instance, In Salisbury and Van Voorhis’s study, the relational mechanism utilizes the reference class of women involved in dysfunctional relationships. The mechanism entails that belonging to this class increases the probability of incarceration. The human and capital model entails the same thing about the reference class of women who lack family support. The problem arises because women can be classified as having dysfunctional relationships for various reasons. In some cases, part of the dysfunction is lack of family support. However, family support is attached to a dysfunctional relationship in other cases. ¹⁰ Therefore, the joint effect of “dysfunctional relationship” and “family support” on incarceration can only be assessed if we know how many cases of dysfunctional relationships overlap with (lack of) family support.

The most natural solution to this problem is to redefine the reference classes. To evaluate the impact of these mechanisms, we can classify women into four groups based on their level of relationship dysfunction and family support. This refinement will help us determine the extent of overlap among these variables. ¹¹ However, in many cases, there is insufficient information to make a statistically significant conclusion based on the fine-grained division.

Although this is a practical rather than conceptual limitation, it is sometimes a very serious limitation. Moreover, there is a related conceptual problem: In some disciplines (particularly in the social sciences), it can be argued that it is always possible to divide the population into more fine-grained reference classes based on some possible relevant features (Runhardt 2017). However, to do any statistical research, we must commit ourselves to some division of reference classes before assessing any mechanism.

We do not deny that theoretically, by incorporating all relevant evidence, we might find a taxonomy that can be used to assess all the relevant evidence. Nevertheless, such a theory is still unavailable in many cases. In these cases, scientists and policy makers still must make decisions based on theories that use incommensurable taxonomies. Thus, it seems reasonable that, in some cases, scientists and policy makers will view all the theories in some set of taxonomically incommensurable theories as reliable.

3.2. Methodological incommensurability

Kuhn articulated incommensurability as a phenomenon that is not only taxonomical but also methodological. There are several ways to understand methodological incommensurability. Based on Sankey (2013), we understand this phenomenon as occurring when two theories are based on different assumptions in a way that prevents an independent standard for comparing them. Thus, a researcher who adopts the assumptions of one paradigm cannot estimate the implications of learning a piece of evidence associated with other paradigms.

In the gendered pathways case, it can be argued that the social and human capital mechanism is methodologically incommensurable with the two other mechanisms. According to this mechanism, offending behavior is assumed to be intentional; indeed, a rational decision. In the other models, it is assumed to be a symptom of pathological states such as addiction and depression. It would be a mistake to claim, we believe, that the offending behavior is unintentional according to these models. Whether it is intentional or not is a delicate matter that might get different answers by different theories in the set of theories that constitute the theoretical foundations that support the child victimization and/or relationship dysfunction mechanisms. It is clear that even these theories take some offending behavior to be intentional. However, the sense in which it is intentional is not the same sense of intentionality as the one captured by the social and human capital mechanism.

Consequently, there is a problem in directly assessing the joint influence of the human capital mechanism and one of the other two mechanisms (or both). The relational mechanism, for example, assumes that some actions of individuals are directly caused by a pathological disorder such as substance abuse. From the perspective of theories that support the human capital mechanism, substance abuse can only be a cause of actions as far as it affects intentional actions. However, the relational mechanism does not explicate how substance abuse affects or is affected by intentional actions. Because pathway analysis assessments presume the direction of causality, an agent cannot evaluate the influence of substance abuse from the perspective of theories that support the human capital mechanism.

4. A (nonstandard) Bayesian model of MT in the case of incommensurability

4.1 The model

We are interested in cases in which different incommensurable theories give predictions regarding the same phenomenon, and an agent (a decision maker or a scientist) takes all these theories to be reliable (to some degree) with respect to this phenomenon. As noted in the introduction, we think such cases are common when it comes to usages of research from social science in evidence-based policy.

In such cases, we model the agent’s epistemic state in the following way. When considering the phenomenon, the agent can adopt the point of view of each one of the theories. As we try to justify the practice of MT, we assume the agent is rational. We adopt a Bayesian approach and assume that when adopting the point of view of each theory, the agent’s degrees of belief are probabilistic and updated using Bayesian conditionalization.

Thus, for each theory, i, there is a corresponding credence function, ${C^i}\left( . \right)$ , that represents the agent’s beliefs when adopting the point of view of the theory. To capture the incommensurability, we depart from the standard Bayesian model and assume that these credence functions are defined over algebras that are only partly overlapping. This assumption captures both cases of taxonomic incommensurability, in which some of the propositions one theory refers to are inexpressible in the conceptual framework employed by another theory, and cases of methodological incommensurability, in which a theory is uninformative with respect to evidence that another theory considers.

Still, we assume that at least one proposition, H, is included in the algebras of all the theories. H represents the hypothesis in question, that is, the one with respect to which the agent considers all theories to be reliable to some degree. Because the agent takes all the theories to be reliable to some degree with respect to H, we assume that for each theory, i, there is a proposition, ${E_i}$ that represents a piece of evidence (or the entire body of evidence) that supports H, according to theory i, such that the agent’s degree of belief in H, from the point of view of theory i, after learning, ${E_i}\ C_{{E_i}}^i\left( . \right),$ ¹² is higher than her degree of belief in H, from the point of view of theory i, before learning ${E_i}$ :

For each theory i,

$$C_{{E_i}}^i\left( H \right) = {C^i}\left( {H{\rm{|}}{E_i}} \right) = {{{C^i}\left( {{E_i}|H} \right)\;{\rm{*}}{C^i}\left( H \right)} \over {{C^i}\left( {{E_i}} \right)}} \gt {C^i}\left( H \right)$$

To capture the incommensurability, we assume that for each theory, i, there is at least one piece of evidence represented by a proposition, ${E_i}$ such that each one of the other credence functions, $\;{C^j}\left( . \right)$ associated with each one of the other theories, is not defined over ${E_i}$ .

Thus, all credence functions (each one associated with a different theory) assign some probability value to H, each one of them assigns some probability value to some proposition, ${E_i}$ that represents evidence for H from the point of view of the corresponding theory, and none of the credence functions assign a probability value to a proposition, ${E_i}$ which is associated with another theory.

What we are interested in is the effect of learning a piece of evidence associated with one theory, i, ${E_i}$ on the degree to which a piece of evidence associated with another theory, j, ${E_j}$ , evidentially support H, from the point of view of theory j. Thus, we are interested in $C_{{E_i}}^j\left( {H|{E_j}} \right)$ —the agent’s conditional degree of belief in H, given ${E_{j}}$ from the point of view of j, after learning ${E_i}.$

In a traditional Bayesian framework, in which there is only one credence function which is defined over an algebra that includes both, ${E_i}\;and\;{E_j}$ this probability value would be equal to the conditional credence of $H$ given ${E_i}\;$ and $\;{E_j}$ , i.e., $C\left( {H|{E_i}{E_j}} \right)$ . However, in our (nonstandard) framework, this last expression is undefined because none of the theories is defined over an algebra that includes both ${E_i}\;$ and $\;{E_j}$ .

Our model shows that by relying on two natural assumptions, we can derive the value of $C_{{E_i}}^j\left( {H|{E_j}} \right)$ . The two assumptions express the two defining features of the type of cases we are interested in:

1. 1. The theories are incommensurable, and

2. 2. The agent takes all theories to be reliable.

While these two features are not contradictory (as incommensurability is not incompatibility), they do, in some sense, not sit well with each other: The point of view the agent must adopt when judging all theories to be reliable cannot be the point of view of any of the theories (because from the point of view of each theory, it is not the case that the other theories are either reliable or unreliable with respect to the hypothesis), but because the agent takes the theories to be incommensurable, there can be no “meta point of view” the decision maker can adopt that allows her to compare the reliability of the different theories.

Still, as noted, when it comes to evidence-based policy, scientists and decision makers often find themselves in cases that at least seem to have these two features. Our model points to a natural way to make sense of feature 2, given feature 1 (more on this in the following text).

Let us start with the latter feature and then move to the former: ¹³

Same credence for H

1. 1. Before learning any evidence, the agent assigns to H the same probability, regardless of the perspective from which she considers H.

2. 2. After updating on a piece of evidence, ${E_i}$ , from the perspective of theory i, the agent adopts the new credence value of H, according to the perspective of theory i, when considering H from the perspective of theory j:

$${C^i}\left( H \right) = {C^j}\left( H \right),\;C_{{E_i}}^i\left( H \right) = C_{{E_i}}^j\left( H \right),\;$$

The assumption expresses the idea that the agent considers all theories to be reliable. Because this is the case, at any evidential state (including the one captured by the first condition of the assumption, in which no evidence was learned), the agent has the same degree of belief in H from the point of view of all theories.

On an intuitive level, the idea is that because the agent takes i to be a reliable theory (from every point of view), after raising her degree of belief in H, from the point of view of theory i (as a result of learning $\;{E_i}$ ), the agent adopts this new degree of belief also from the point of view of theory j.

This degree of belief can be taken to represent the agent’s “all-considered” degree of belief, that is, the one the agent uses when making decisions based on her scientific beliefs. When all is said and done, the agent must make decisions and if these decisions are rational (according to most accounts of practical rationality) they can be represented as resulting from a maximization of expected (or—according to some—rank-dependent expected) utility for some utility function and a unique probability function. This probability function can be seen as the agent’s “all considered” credence function. ¹⁴

Another way to understand the “same credence for H” assumption, that is not committed to the behavioral interpretation of credence, is to take ${E_i}\;$ (i.e., evidence for H, according to i) to be nonpropositional evidence for H according to j. According to this interpretation, when learning, ${E_i}$ although the agent does not learn any proposition in the algebra over which ${C^j}\left( . \right)$ is defined, she does receive an input, which is nonpropositional from the point of view of j, that makes her raise her degree of belief in H also from the point of view of j, without raising her degree of belief in any proposition in the algebra to 1. This interpretation naturally leads to the thought that the right way for the agent to update her beliefs in such cases is by using Jeffrey’s conditionalization. As we argue in the following text, this conclusion follows directly from our second assumption, the rigidity condition. ¹⁵

Rigidity:

The conditional credence the agent attaches to a piece of evidence associated with theory i, given H, from the perspective of theory i, does not change after learning a piece of evidence associated with theory j: ¹⁶

$$C_{{E_j}}^i\left( {{E_i}|H} \right) = {C^i}\left( {{E_i}|H} \right)\;and\;C_{{E_j}}^i\left( {{E_i}|\neg H} \right) = {C^i}\left( {{E_i}|\neg H} \right)$$

Given incommensurability, the rigidity assumption expresses a relatively modest form of epistemic conservatism. Learning a piece of evidence associated with theory i teaches the agent something about H, even when considered from the perspective of j (as the previous assumption demands), and so might also teach her something about $\;{E_j}$ (because $\;{E_j}\;$ is indicative of H, and so when the probability of H goes up, the probability of $\;{E_j}$ should also go up). However, it should not teach the agent anything about how likely $\;{E_j}$ given H is (from the perspective of j). Learning $\;{E_i}$ does not teach the agent anything about the internal structure of theory j because $\;{E_i}\;$ is not even expressible in j. Particularly, it does not teach the agent anything about how indicative $\;{E_j}$ is to H, from the point of view of j. Thus, because by learning $\;{E_i}$ the agent did not learn anything relevant to the probability ${C^i}\left( {{E_i}|H} \right)$ —according to the type of epistemic conservatism the rigidity assumption expresses—the agent should not change this conditional degree of belief. The same is true, of course, regarding $\sim H$ .

The sense of epistemic conservatism that the rigidity condition captures is, then, the following: If you did not learn anything that should make you change your degree of belief in a proposition (or conditional degree of belief in one proposition given another)—do not!

It might be useful to compare the rigidity condition to the independent condition used in other accounts for MT (and discussed in Section 1). If the agent’s credence function was defined over on algebra that includes H, $\;{E_j},$ and, $\;{E_i}$ , the natural way to express the idea that after learning $\;{E_j}$ , although the credence of $\;{E_i}$ might go up (in virtue of the credence of H going up), the conditional credence of $\;{E_i}$ , given H should not change, would be to use the independence condition of Steganga and Menon (2017):

$$C\left( {{E_i}{\rm{|}}H} \right) = \;C\left( {{E_i}{\rm{|}}H{E_j}} \right)$$

However, as explained in Section 1, in such a setting the independence assumption seems to be unjustified: In cases in which no knowledge regarding the objective probabilities of $H$ , ${E_j},$ and ${E_i}$ , is available, the assumption is arbitrary and in cases in which such knowledge does exist, there is no reason to formulate the assumption in a subjective probability setting. In the type of cases our model covers, however, the assumption expresses—as just explained—no more than a commitment to a modest type of epistemic conservatism.

In other words, although from a formal point of view the rigidity condition plays an analogous role to the independence condition, it is based on a completely different justification.

4.2. Updating using Jeffrey’s conditionalization

When the two assumptions hold, the agent updates her beliefs using Jeffrey’s conditionalization (JC). JC is a generalization of classical Bayesian conditionalization. In the standard Bayesian picture, when an agent learns a proposition, she learns it with certainty (i.e., she raises its probability to 1). Richard Jeffrey (1992) argued, however, that in some cases, agents learn something without learning anything with certainty. This is exactly what happens in our model. When the agent raises the credence she assigns to H, as a result of learning, ${E_i}$ from the perspective of theory j, the agent has learned something (that made her raise her credence in H also from the perspective of j), without learning anything with certainty (because ${E_i}$ is not a proposition in the algebra over which the credence function associated with j is defined).

According to JC, the credence function, ${C_\alpha }\left( . \right)$ , an agent adopts after learning some input, α, that made her change her credence in a proposition, H, is such that for any proposition, E, the following holds:

$$\left( 1 \right){\rm{\;}}{C_\alpha }\left( E \right) = C\left( {E{\rm{|}}H} \right)\,{\rm{*}}\,{C_\alpha }\left( H \right) + C\left( {E{\rm{|}}\neg H} \right)\,{\rm{*}}\,{C_\alpha }\left( {\neg H} \right)$$

(When c(.) indicates the agent’s credence function before learning α).

It is easy to see that rigidity is a sufficient condition for JC: When rigidity holds,

$${C_\alpha }(E|H){\rm{ }} = C(E|H){\mkern 1mu} {\rm{and}}{\mkern 1mu} {C_\alpha }(E|\neg H) = C(E|\neg H)$$

Thus, (1) reduces to the law of total probability. Notice also that when ${C_\alpha }\left( H \right)$ = 1, JC reduces to standard Bayesian updating.

Applying JC to our model, we get:

$$\left( 2 \right){\rm{\;}}C_{{E_j}}^i\left( {{E_i}} \right) = {C^i}\left( {{E_i}{\rm{|}}H} \right)\,{\rm{*}}\,C_{{E_j}}^i\left( H \right) + {C^i}\left( {{E_i}{\rm{|}}\neg H} \right)\,{\rm{*}}\,C_{{E_j}}^i\left( {\neg H} \right)$$

However, recall that we are not interested in, $C_{{E_j}}^i\left( {{E_i}} \right)$ but rather in, $C_{{E_j}}^i\left( {H|{E_i}} \right)$ that is, with the effect of learning ${E_j}$ on how indicative ${E_i}$ is to, $H$ from the point of view of $i$ . To calculate this probability value, all one has to do is use JC on $C_{{E_j}}^i\left( {H|{E_i}} \right)$ . Our two assumptions allow us to do that.

When there are only two theories, the agent’s credence in H, from the point of view of theory $i$ , after learning both ${E_i}$ and ${E_j}$ is:

$$\left( 3 \right)\ C_{{E_j},{E_i}}^i\left( H \right) = C_{{E_j}}^i\left( {H|{E_i}} \right) = {{C_{{E_j}}^i\left( {H{E_i}} \right)} \over {C_{{E_j}}^i\left( {{E_i}} \right)}} = {{C_{{E_j}}^i\left( {{E_i}|H} \right){\rm{*}}C_{{E_j}}^i\left( H \right)} \over {C_{{E_j}}^i\left( {{E_i}{\rm{|}}H} \right){\rm{*}}C_{{E_j}}^i\left( H \right) + C_{{E_j}}^i\left( {{E_i}{\rm{|}}\neg H} \right){\rm{*}}C_{{E_j}}^i\left( {\neg H} \right)}}$$

Rigidity implies that:

$$\left( 4 \right)\ C_{{E_j}}^i\left( {{E_i}{\rm{|}}H} \right) = {C^i}\left( {{E_i}{\rm{|}}H} \right){\rm{\;}}and{\rm{\;}}C_{{E_j}}^i\left( {{E_i}{\rm{|}}\neg H} \right) = {C^i}\left( {{E_i}{\rm{|}}\neg H} \right)$$

Therefore:

$$\left( 5 \right)\ C_{{E_j},{E_i}}^i\left( H \right) = {{{C^i}\left( {{E_i}|H} \right){\rm{*}}C_{{E_j}}^i\left( H \right)} \over {{C^i}\left( {{E_i}{\rm{|}}H} \right){\rm{*}}C_{{E_j}}^i\left( H \right) + {C^i}\left( {{E_i}{\rm{|}}\neg H} \right){\rm{*}}C_{{E_j}}^i\left( {\neg H} \right)}}$$

Using Bayesian conditionalization, we can get:

$$\left( 6 \right)\ {C^i}\left( {{E_i}|H} \right) = {{{C^i}\left( {H|{E_i}} \right){\rm{*}}{C^i}\left( {{E_i}} \right)} \over {{C^i}\left( H \right)}} = {{C_{{E_i}}^i\left( H \right){\rm{*}}{C^i}\left( {{E_i}} \right)} \over {{C^i}\left( H \right)}}$$

$$\left( 7 \right)\ {C^i}\left( {{E_i}{\rm{|}}\neg H} \right) = {{{C^i}\left( {\neg H|{E_i}} \right){\rm{*}}{C^i}\left( {{E_i}} \right)} \over {{C^i}\left( {\neg H} \right)}} = {{\left( {1 - C_{{E_i}}^i\left( H \right)} \right){\rm{*}}{C^i}\left( {{E_i}} \right)} \over {1 - {C^i}\left( H \right)}}$$

By substituting these results in the former equation, we get:

$$\left( 8 \right)\ C_{{E_j},{E_i}}^i\left( H \right) = {{{{C_{{E_i}}^i\left( H \right){\rm{*}}{C^i}\left( {{E_i}} \right)} \over {{C^i}\left( H \right)}}{\rm{*}}C_{{E_j}}^i\left( H \right)} \over {{{C_{{E_i}}^i\left( H \right){\rm{*}}{C^i}\left( {{E_i}} \right)} \over {{C^i}\left( H \right)}}{\rm{*}}C_{{E_j}}^i\left( H \right) + {{\left( {1 - C_{{E_i}}^i\left( H \right)} \right){\rm{*}}{C^i}\left( {{E_i}} \right)} \over {1 - {C^i}\left( H \right)}}{\rm{*}}C_{{E_j}}^i\left( {\neg H} \right)}}$$

$$ = {{C_{{E_i}}^i\left( H \right){\rm{*}}C_{{E_j}}^i\left( H \right)} \over {C_{{E_i}}^i\left( H \right){\rm{*}}C_{{E_j}}^i\left( H \right) + \left( {{{{C^i}\left( H \right)} \over {1 - {C^i}\left( H \right)}}} \right)\left( {1 - C_{{E_i}}^i\left( H \right)} \right){\rm{*}}\left( {1 - C_{{E_j}}^i\left( H \right)} \right)}}$$

Same credence for h implies that

$$\left( 9 \right){\rm{\;}}C_{{E_i}}^i\left( H \right) = C_{{E_i}}^j\left( H \right)$$

Therefore, we get:

$$\left( {10} \right)\ C_{{E_j},{E_i}}^i\left( H \right) = {{C_{{E_i}}^i\left( H \right){\rm{*}}C_{{E_j}}^j\left( H \right)} \over {C_{{E_i}}^i\left( H \right){\rm{*}}C_{{E_j}}^j\left( H \right) + \left( {{{{C^i}\left( H \right)} \over {1 - {C^i}\left( H \right)}}} \right)\left( {1 - C_{{E_i}}^i\left( H \right)} \right){\rm{*}}\left( {1 - C_{{E_j}}^j\left( H \right)} \right)}}$$

Notice that the learning order of the different pieces of evidence (the different ${E_i}$ propositions) does not matter due to same credence for h implying:

$$\left( {11} \right){\rm{\;}}C_{{E_i},{E_j}}^i\left( H \right) = C_{{E_i},{E_j}}^j\left( H \right)$$

And our model, which implies:

$$\left( {12} \right)\ C_{{E_j},{E_i}}^i\left( H \right) = C_{{E_i},{E_j}}^j\left( H \right)$$

This feature is preserved when we expand our model to n methods. In such a case, our assumptions imply the following equation (proof in the Appendix): ¹⁷

(13) for any n methods (1 < n):

$$C_{{E_1}:{E_n}}^n\left( H \right) = {{\prod\nolimits_{i = 1}^n {C_{{E_i}}^i} \left( H \right)} \over {\prod\nolimits_{i = 1}^n {C_{{E_i}}^i} \left( H \right) + {{\left( {{{{C^n}\left( H \right)} \over {1 - {C^n}\left( H \right)}}} \right)}^{n - 1}}\prod\nolimits_{i = 1}^n {\left( {1 - C_{{E_i}}^i\left( H \right)} \right)} }}$$

Observation 1 (Proof in the Appendix):

$$for\;any\;n\;\left( {1 \le n} \right),\;C_{{E_{n + 1}}}^{n + 1}\left( H \right) \gt {C^{n + 1}}\left( H \right)\;iff\;C_{{E_1}:{E_{n + 1}}}^n\left( H \right) \gt C_{{E_1}:{E_n}}^n\left( H \right)$$

In words: As long as a piece of evidence supports a hypothesis, from the point of view of a theory that the decision maker takes to be reliable with respect to this hypothesis, learning this piece of evidence makes any other piece of evidence that supports the hypothesis from the point of view of another theory that is incommensurable with the first theory, more indicative to the hypothesis.

A natural way to interpret this result is the following one. Remember that we are interested in cases characterized by two main features:

1. 1. The decision maker takes several theories to be reliable with respect to some hypothesis.

2. 2. The decision maker takes these theories to be incommensurable with each other.

As explained in the preceding text, although these two features do not contradict each other there is a tension between them due to the lack of a “meta point of view” the decision maker can adopt that can allow her to compare the reliability of different theories. How can, then, one makes sense of feature 1, given feature 2?

A seemingly naïve suggestion is that in such cases, the decision maker’s commitment to the reliability of theory i, when considered from the point of view of theory j, is expressed in that every piece of evidence associated with theory j, but not with i, supports the hypothesis, from the point of view of j to a higher degree than what j itself implies. The commitment to i’s reliability makes evidence, according to j, stronger than what j takes it to be (and, of course, vice versa). ¹⁸

Our results show that this seemingly naïve suggestion is implied by a commitment to the minimal type of epistemic conservatism the rigidity condition expresses (the “do not change anything in your credence function unless you have to” sense). Being epistemically conservative in this way enables one to be committed to 1 and 2 simultaneously.

How strong is the effect of learning each additional piece of evidence? Equation 14 gives the exact answer to this question (which is similar to the answer given by the first account offered by Heesen et al. 2019).

Figure 1 demonstrates considering the effect of learning pieces of evidence, when the decision maker starts with a credence of 0.2 in the hypothesis and learning each piece of evidence, on its own, makes the decision maker raise her credence in the hypothesis to 0.4 (when considered from the point of view of the relevant theory).

Figure 1 The credence of an agent occrding to our model given different numbers of methods, when $C^{{i}}(H)=0.2$ and for each method $i C_{{E_i}}^i\left( H \right) =0.4$ .

5. Conclusion: Back to gendered pathways to incarceration

Salisbury and van Voorhis’s study shows that when each mechanism is considered from the perspective of the theory that supports it, it provides evidence for incarceration. For example, evidence that a given woman is affected by the social and human capital mechanisms $\left( {{E_{SHC}}} \right)$ should increase a decision maker’s credence in her incarceration (I), when she adopts the perspective of the social and human capital theory:

$${C^{SHC}}\left( {I{\rm{|}}{E_{SHC}}} \right) \gt {C^{SHC}}\left( I \right)$$

The study also implies that the same result holds for the relational (R) and child victimization (CV) mechanisms:

$${C^R}\left( {I{\rm{|}}{E_R}} \right) \gt {C^R}\left( I \right)$$

$${C^{CV}}\left( {I{\rm{|}}{E_{CV}}} \right) \gt {C^{CV}}\left( {I{\rm{|}}CV} \right)$$

However, as we argued, we cannot assess the epistemic value of ${E_R}$ from the perspective of the theory underlying the SHC mechanism. Thus, the epistemic value of one mechanism, when seen from the perspective of a theory that supports another mechanism (e.g.,) ${C^{SHC}}\left( {I{\rm{|}}{E_R}} \right)$ , cannot be determined. However, given rigidity and same credence for H, our model shows that the more evidence from incommensurable theories an agent learns, the higher their credence in I should be:

$${C_{{E_{CV}}}}\left( I \right),{C_{{E_D}}}\left( I \right),{C_{{E_{SHC}}}}\left( I \right) \lt {C_{{E_{CV}},{E_D}}}\left( I \right),{C_{{E_{CV}},{E_{SHC}}}}\left( I \right),{C_{{E_D},{E_{SHC}}}}\left( I \right) \lt {C_{{E_{CV}},{E_D},{E_{SHC}}}}\left( I \right)$$

Consider now an organization that aims to decrease the rates of women’s incarceration. If the organization is committed to evidence-based policy, then Salisbury and van Voorhis’s study seems like a great source of information for this organization on which to rely. According to this source, each of the three mechanisms provides good evidence for incarceration. However, the study does not provide any “meta point of view” that the organization can adopt to assess how multiple mechanisms affect incarceration, even though many women are affected by more than one mechanism.

Suppose that the organization has a limited set of resources and is interested in allocating them in a way that best serves the goal of minimizing incarceration. Given the incommensurability we argued for, there is no straightforward way to calculate the probability of incarceration of women affected by multiple mechanisms (which is needed when trying to find the best way to allocate resources). Without our model, the organization would presumably have to use (as input to its decision-making process) the chance of incarceration of such women according to just one of the mechanisms. By doing so, however, it will ignore available evidence.

Our model introduces another option. By committing ourselves to epistemic conservativism (in the minimal sense explained previously), we can justify the claim that women affected by more than one mechanism are more likely to be incarcerated than women affected by only one mechanism.