Evidential Variety and Mixed-Methods Research in Social Science

Abstract

Mixed-methods research (MMR)—the combination of qualitative and quantitative data within the same design to strengthen causal inference—is gaining prominence in the social sciences but its benefits are contested. There remains confusion over which methods to mix and what is the point of mixing them. We argue that variety of evidence is what matters, not of data or methods, and that distinct epistemic principles underlie its added value for causal inference. The centrality of evidential variety also implies that strong causal pluralism is untenable as a foundation for MMR.

1. Introduction

Mixed-methods research (MMR) is increasingly popular as a methodology in social science. It combines different methods of data production and analysis within a single research design, most often with the purpose of increasing the reliability of causal inference. Its purported novelty is that the methods represent different methodological traditions, usually qualitative and quantitative social research, and that the mixing should happen in a coordinated, integrative fashion within a single research design. MMR designs aim to direct the selection of methods and the units of study as well as the formulation of research questions and hypotheses in such a way that the different methods produce mutually supporting evidence. The attractiveness of MMR, the justification for trying to do many things at once instead of one thing well, is thus based on some version of the variety-of-evidence thesis, according to which the epistemic whole is more than the sum of its evidential parts, provided the parts are varied in some relevant way.

Several MMR designs have been proposed. For example, set theoretic multimethod research combines case studies with small-n comparative studies subjected to qualitative causal analysis (Rohlfing and Schneider 2018). Take, for example, the causal relation between elites holding liberal ideas and democratic peace (a candidate explanation for the democratic peace phenomenon): process tracing involves identifying the mechanism through which liberal ideas causally contribute to democratic peace and then applying qualitative comparative analysis to generalize to other cases. Nested analysis, instead, combines observational quantitative studies (regression analysis) with case studies carried out within the sample used in the quantitative study, hence the word “nested” (Lieberman 2005). For example, Coppedge (2005, discussed in Lieberman 2005) first applied general models (statistical causal hypotheses) to account for the deterioration of democracy in Venezuela, identified what remained to be explained, and then “intensively” studied the case of Venezuela to conclude that patriarchy was an overlooked and yet important explanatory factor. Humphreys and Jacobs (2015) proposed a unifying Bayesian framework intended to apply to different combinations of statistical and case-study methods.

In spite of its popularity, however, there remains considerable skepticism concerning whether MMR is superior to single-method research. Here is how one commentator summarizes the concern:

[w]hile multimethod research is evidently experiencing a surge of popularity, there are reasons to worry about whether multimethod applications are in fact producing more grounded, justified, and persuasive inferences than studies using a single method. (Seawright 2016, 4)

The confusion is due in part to a lack of consensus about the epistemological rationale of MMR, and in part to differences across MMR designs in terms of which methods are combined, and what the purpose of combining them is.

The aim in this article is to clarify the epistemic rationale of MMR for causal inference in the social sciences. Our main claim is that MMR should be designed and evaluated from the perspective of what hypotheses the evidence produced by different methods is evidence for and of how these hypotheses are inferentially interlinked, not in terms of whether the methods or data are diverse as such. This is so because it is the body of evidence and its features that give reason to believe in the hypothesis of interest. We therefore proceed as follows. In section 2 we explain why a variety of methods does not suffice to produce a variety of evidence and, consequently, why MMR affords epistemic benefits only if, and when, evidence is varied in some relevant way. Drawing on the philosophical literature on the variety-of-evidence thesis, we argue in section 3 that it is possible to understand MMR according to three distinct principles of evidential variety. Section 4 addresses some undesirable consequences of the pluralist claim that different methods in MMR embody different conceptions of causality. Section 5 concludes the article.

2. From methods to evidence

It is often assumed that MMR is about combining data obtained and analyzed by different means (typically qualitative and quantitative) for the purpose of better supporting a causal hypothesis compared to data obtained from a single method. This idea is behind what we call simple methodological pluralism, according to which different methods produce different kinds of data that, when combined in the right way, provide better support for causal inferences. The problem with simple methodological pluralism is that it remains unclear why combining data produced by different methods should in itself improve causal inference, over and above there being more data in support of the hypothesis (with the obvious proviso that the data agree).

Let us think of a method as the causal process whereby the data are generated, together with the inferential practices applied to them, and then further distinguish data from evidence. Data comprise the causally produced, concrete, and public artifacts that carry information from the phenomenon of interest (Bogen and Woodward, 1988; Leonelli 2015). As such they are always of, or from, something, their identity and type determined by the process through which they were created. Evidence, in turn, is a claim or a fact (or some other propositionally structured entity) about the data, usually grounded on a systematic pattern of some sort, which speaks in favor of, or provides a reason to believe in, some specific claim (e.g., Achinstein 2001; Cartwright 2013). As such, evidence is always for something: Its identity as evidence is relational to a claim or a hypothesis.

The distinction between data and evidence is not mere philosophical sophistry. The epistemic benefits conferred by variety reflect the relevant differences in the evidence, not in the data or their production as such. The point of bringing together diverse evidence is to substantiate the reasons for believing a hypothesis (in this case a causal claim or a claim about a mechanism). Data in themselves do not speak in favor of a claim, nor do they count as reasons to believe; they do so only when some of their features are interpreted as evidence for a specific claim. As material artifacts, data simply exist, whereas evidence lives in the space of reason. As Leonelli’s (2015, 2019) work convincingly shows, data can also be reused and cited as evidence for claims other than the one for which they were originally produced. The distinction between data as material artifacts and evidence as a reason to believe means that it is a category mistake to equate kinds of methods with kinds of evidence, and methodological variety with evidential variety. We should point out that this is not to claim that the methods whereby data are produced are irrelevant to evidential relevance: For example, as we show later on, features of the causal processes whereby the data are produced are relevant to achieving the variety of evidence required for triangulation (Kuorikoski and Marchionni 2016).

We should also point out that if different methods always produce relevantly different evidence, then the epistemic justification of MMR could be straightforwardly explicated in terms of exploiting their complementary strengths and weaknesses, therefore simple methodological pluralism would suffice as a foundation for MMR. For example, it is a widespread belief among social scientists that knowledge of mechanisms can only be generated by case-based methods. This argument is also well rehearsed in the MMR literature, in which it is presupposed that case-based methods and statistical analyses produce essentially different kinds of evidence. For example, most MMR frameworks simply assume that whereas quantitative data yield evidence for population-level claims about average causal effects, qualitative data provide evidence of the process that connects the cause to the effect in each particular case (e.g., Lieberman 2005). Derek Beach, an influential methodologist in the field of political science, insists (e.g., 2020) that both statistical studies, including randomized controlled trials (RCTs) and cross-country comparisons, and case studies produce different kinds of evidence: difference-making evidence and mechanistic evidence, respectively. ¹

However, differences in methods and evidence do not always go hand in hand. Evidence about mechanisms may be obtained in different ways, including RCTs and statistical analyses (Illari 2011; Marchionni and Reijula 2019). Conversely, the same method may generate different kinds of evidence. For example, as discussed in the fields of development economics and criminology, RCTs may generate evidence for the efficacy of an intervention as well as evidence about the mediating mechanism (so-called policy and mechanism experiments, respectively; see Ludwig et al. 2011). Methodological diversity does not guarantee the kind of difference in evidence that is necessary for exploiting the epistemic benefits of evidential variety.

3. Varieties of evidential variety

If simple methodological pluralism is not a sufficient epistemic foundation for MMR, then what is? MMR rationale is sometimes articulated in terms of methodological triangulation (Johnson et al. 2007): the idea that bringing different methods each with its own characteristic weaknesses and strengths to bear on the same causal claim helps to remedy those weaknesses (see also Crasnow 2010). This idea has been questioned, however (Seawright 2016, 4–10), and other MMR researchers seem to hold different conceptions of the purpose of mixing methods (see Johnson et al. 2007).

As Ahmed and Sil (2012) point out, motivating MMR in terms of triangulation is problematic because in many cases such as nested analysis the different methods clearly address different questions. Rather than a matter of triangulation, Plano Clark and Ivankova (2016) describe MMR as “the process of integrating quantitative and qualitative research to more completely address a study’s purpose or research questions” (emphasis added). Accordingly, the pieces of evidence integrated into MMR address different questions, not the same question as in the case of triangulation. Shan and Williamson (2021) argue that evidential pluralism, which they understand as the claim that causal inference typically requires evidence of both difference making and mechanisms, provides a suitable epistemic foundation for MMR.

As we have already argued elsewhere, there are multiple distinct sources of this value, and distinct epistemic rationales for combining evidence, namely triangulation, security, and integration (Figure 1) (Kuorikoski and Marchionni 2022).

Figure 1. A schematic representation of kinds of evidential variety. P is a proposition about a phenomenon, E_n is evidence, and M is a mechanism; arrows represent causal relations whereas lines represent evidential relations.

These forms of reasoning are sometimes presented as mutually exclusive alternative rationalizations for the variety-of-evidence thesis, but this need not be case. Because the epistemic rationales for these different forms of the value of variety appear to make sense in their own right, we propose that, depending on what is the inferential problem at hand and/or what characteristics a certain piece of evidence has in relation to a given causal claim, there are simply different ways in which evidence can be combined and different nonaggregative benefits that can accrue. Furthermore, different self-described MMR procedures appear to realize different principles. Hence, dividing the value of variety in this way has analytic value (Kuorikoski and Marchionni 2022).

Triangulation refers to the use of multiple independent lines of evidence for a single claim about a phenomenon such as the existence of a causal relation (e.g., Heesen et al. 2019; Kuorikoski and Marchionni 2016; Schupbach 2018). Accordingly, the added value of evidential variety derives from the fact that concordance between multiple independent lines of evidence indicates that the common result is unlikely to be an artifact of errors and biases characteristic of any of the individual methods.

Successful triangulation therefore requires that the different pieces of evidence are evidence for the same causal claim, and that they are suitably independent. Philosophers of science have debated how this independence should be understood (e.g., Fitelson 2001; Stegenga 2009; Claveau, 2013),² but two things are clear. First, the diversity of methods by which data are produced is not, in itself, sufficient for obtaining evidence that is independent in the right way. Second, the independence condition may be hard to meet in practice and other epistemic considerations, such as the reliability or informativeness of individual sources of evidence, may outweigh the value of its variety (Claveau and Grenier 2019). This matters, because at least some of the skepticism related to MMR arises from its being regarded as realizing triangulation. For example, multimethod large-N qualitative analysis is popular in political science and international relations, its aim being to combine statistical analysis with case studies to provide independent evidence for population-level claims about average causal effects. As Rosa Runhardt (2022) argues, however, the statistical evidence does not add anything to the evidence about mechanisms obtained from case studies. If this is the case, then it is clearly not a suitable design for realizing triangulation.

Security constitutes another source of value related to evidential variety. Here, one piece of evidence remedies the deficiencies of another piece of evidence relative to a given causal claim (Staley 2004; Zahle 2019). The main difference with the case of triangulation is that the different pieces of evidence are not direct evidence supporting the same claim. Variety brings security when there is a causal claim, first-order evidence in its support, and then second-order evidence in favor of the first-order evidence being reliable and relevant for the causal claim in question. To clarify the difference between triangulation and security, we suggest thinking of triangulation as the support afforded by two independent eyewitnesses each testifying to having seen the suspect at the crime scene. Independence here could mean that the eyewitnesses did not know each another and were in different spatial locations with respect to the suspect, for example. According to this analogy, security is involved when the testimony of an expert placing the eyewitness in a position to see the suspect clearly enough counts as second-order evidence in support of the reliability of the eyewitness as first-order evidence (for the claim about the culprit). As in triangulation, second-order evidence can only strengthen the inference if it is not subject to the same errors and biases as the first-order evidence.

Zahle (2019) suggests that the combination of observational evidence and interview evidence within a case study sometimes follows the logic of security: Under proper conditions, evidence obtained through participant observation provides second-order evidence that interview evidence is reliable first-order evidence for the phenomenon. Evidence obtained through interviews, for example, may confirm that the researchers’ observations are in fact relevant evidence. Extending this idea to MMR is relatively straightforward, even though skeptics might claim that the challenge of incommensurability that besets it does not arise in this case. For example, in arguing for the use of MMR in impact evaluations of development projects (a kind of causal inference), Michael Woolcock (2019) lists multiple uses for qualitative methods, most of which are about producing evidence that the primary quantitative (experimental or quasi-experimental) evidence is indeed reliable.

The third source of added value arises when evidence for distinct claims about the same phenomenon together produces a fuller picture of it such that the epistemic whole is better supported by the totality of evidence than the sum of the individual hypotheses (Lloyd 2009; Cartwright 2021). Integration differs from triangulation and security because the evidence need not be independent, nor does it need to be about the same claim. To continue with the trial analogy, two experts produce integrating evidence when they testify on different aspects of the case, one being a ballistics expert testifying about the way the gun was fired and the other being a criminal psychologist testifying about the suspect’s motives.

A useful way of understanding how integration works is to use causal graphs. One could think of the mechanism as the full causal graph and the different pieces of evidence as being about different components of the same graph. Again, diverse evidence in this sense is often produced using different methods, but it need not be. For example, evidence for a mechanism mediating an effect in a field experiment could be produced by a mechanism experiment.

Integration may be realized in a number of ways in various epistemic contexts. Cartwright’s (2021) proposal for MMR comes closest to our notion of integration. Cartwright’s argument is that a mix of methods may well be needed if the aim is to support different kinds of subsidiary claims (such as claims about the presence or absence of support factors) that must hold true for the overall claim to be true. Likewise, Shan and Williamson (2021) point out that “to establish a causal claim one normally needs to establish the existence of an appropriate conditional correlation and the existence of an appropriate mechanism,” proposing this as a foundation for MMR. Combining evidence for the existence of a correlation and evidence for the existence of a mechanism fits both the logic of integration and some instances of MMR. For example, with regard to nested analysis, statistical studies provide evidence for average causal effects, and case studies evidence for case-specific causal mechanisms. However, the nested analysis framework connects these two types of analysis in a relatively restricted way: case studies either provide (dis)confirmatory instances of the mechanistic hypotheses behind the statistical analysis, or serve as a heuristic source of new hypotheses for subsequent statistical analysis.

4. Causal pluralism and mutually supporting evidence

Some defenders of MMR support the idea that different methods of data production and analysis embody different notions or conceptions of causality, which in turn correspond to alternative philosophical theories of causation (e.g., Cartwright 2007; Reiss 2009; Illari and Russo 2014). Whereas quantitative methods are usually assumed to presuppose a dependence/difference-making concept of causation, qualitative methods are more in tune with production and process conceptions. In turn, it is sometimes claimed, differences in conceptions of causation imply differences in the rules of causal inference. Johnson et al. (2019, 155) write, for example, that evidential pluralism, which holds that “many sources of evidence are needed and used in our making of causal claims,” entails conceptual pluralism, namely that “there are different kinds of causes at the conceptual level.” The implication is that there are also different kinds of causes on the level of reality, in other words ontological pluralism. However, if evidential pluralism entailed conceptual and especially ontological pluralism, it would create more problems for MMR than it solves.

First, we have already questioned the idea that having methods of different kinds automatically results in stronger evidence.³ Second, as Beach and Kaas (2020) have argued, strong conceptual pluralism entails that the causal claims supported by different methods are really incommensurable, and therefore that MMR is not feasible to begin with. Our position that it is the variety of evidence, not of methods or data, that is epistemically important further strengthens Beach and Kaas’s argument: If the causal claims for which the distinct methods produced evidence were built on incommensurable concepts of causation, there could not be any well-defined inferential relations between them, and thus no epistemic gains from a variety of evidence.

Finally, conceptual causal pluralism has undesirable practical consequences. If different methods produced evidence for claims that embody different concepts of cause, they could also legitimately support apparently contradictory causal conclusions. For example, in qualitative studies it is claimed that active labor-market policies have a negative impact on employment (through demotivation and exit from the market), whereas quantitative studies indicate that their impact is positive. For the end user of this information (e.g., policy makers) this is deeply unsatisfactory, but for the conceptual pluralist there is not necessarily anything particularly puzzling about it. The apparent contradiction could be attributed to the different methods implying different concepts of “impact” (presumably process and difference making).⁴

As we cannot accept that all MMR research is doomed to either failure or practical irrelevance because of conceptual incommensurability, we have to deny genuine pluralism on either the conceptual or the ontological level. For MMR to work, different kinds of causal claims built from different causal concepts have to be inferentially relatable to one another. This requires at least some degree of conceptual unity and coherence in causal reasoning and inference, which conceptual pluralism does not give. Fortunately, it is one thing to point out that social scientists make use of different causal concepts (e.g., average treatment effect, causal relevance, causal capacity, mechanism, process) with distinct inferential properties, and another to claim that these causal concepts correspond to fundamentally different concepts of causation. Conceptual, let alone ontological, pluralism cannot be motivated by the plurality of causal concepts alone (Reiss 2009).

Even though we cannot conclusively argue in favor of any specific causal account that unifies the causal concepts relevant to MMR, we agree with Rohlfing and Zuber (2021) that the interventionist theory of causation utilizing causal graphs is promising in terms of analytic and expressive power. The interventionist account of causation could arguably be used to clarify and regiment inferential relations between unit- and population-level claims, between type and token claims, as well as between causal-effect and mechanism claims. The interventionist framework is not fully neutral with respect to the epistemology of causation, however. Most importantly, it denies the implicit assumption that causal-process observations obtained using case-based research methods could somehow sidestep the fundamental problem of causal inference, namely that the relevant counterfactual values of the variables cannot be directly observed. Beach (2020, 167–68), for example, regards counterfactuals as one possible test for causality that could be achieved by experimental means rather than through analyses of the meaning of the claim being tested. The idea seems to be that if your concept of causation is singularist and process-oriented, you can simply observe causation if you look closely enough.⁵

This is not to say that purely observational case-based causal inference is impossible. What makes it possible, however, is not a distinct concept of causation together with some kind of causal sixth sense. It is rather that the detailed documentation of a series of interactions could ground causal claims when, and insofar as, background causal knowledge is sufficiently reliable. It is quite common in social science for this background knowledge to be based on everyday reasoning and common folk-psychological interpretational practices, and not on explicit social or psychological theories. This may make it easy to forget that reading causation into everyday social interactions does require causal assumptions.

5. Concluding remarks

We have proposed an understanding of MMR in terms of the variety-of-evidence thesis in support of causal inferences, arguing that different MMR procedures can produce and benefit from three distinct kinds of variety of evidence. We also addressed the concern that MMR is not feasible because some methods of data production and analysis embody different, and incommensurable, conceptions of causality. The upshot is that it has the potential to be a powerful methodology for studying causal inference in social science, but its added value can only be secured given the right kind of variety of evidence and the right procedure for bringing it together. The focus on the differences between methods in current methodological discussions on MMR risk distracting scholars from substantial questions concerning the claims for which the diverse evidence is evidence of, how it is diverse over and above being produced by different methods, and in what way different kinds of diversity work in support of causal inference.