Qualitative Replication as a Pedagogical Approach to Teaching Research Methods

Abstract

The importance of replication has been a major matter of discussion among political scientists for decades. However, in the past five years, the issue has gained greater traction, with many major journals adopting official standards for Data Access and Research Transparency (DA-RT). At the same time, scholars suggest the pivotal role that replication might play in methods training for students. Unfortunately, these conversations have been limited in that they emphasize quantitative methods and training of graduate students. This article seeks to fill this gap by offering commentary on the author’s experience in introducing a qualitative replication project in an upper-division undergraduate course. The students in the course replicated Ross’s (2004) influential article on the “resource curse,” but the assignment framework can be adopted for various topics and contexts.

In the political science classroom, faculty are charged with increasing students’ substantive knowledge while also encouraging them to think deeply about “how we know what we know.” Emphasizing research transparency in the classroom can serve both purposes: allowing students to learn more about the political world and to glimpse the process of research. This article brings together two threads in recent discussions of political methodology: norms and practices around Data Access and Research Transparency (DA-RT)¹ and pedagogical approaches to research methods education—specifically, qualitative methods education (Elman, Kapiszewski, and Kirilova 2015; Moravcsik 2014).

Much of the conversation surrounding research transparency in political science has been focused on professional researchers in academia (see Golden 1995 and King 1995 for early examples). Although this is important work, few scholars have considered how it might change our approach to teaching students about research. Greater transparency leads not only to better research overall; it also can demystify the process of research. Allowing students to “see under the hood” and understand the process of research in a lower-stakes setting may change their perspective on their course of study and inspire some to pursue independent research in the future.

Greater transparency leads not only to better research overall; it also can demystify the process of research.

This article describes my experience in designing a course around a qualitative replication project, specifically having students replicate Ross’s (2004) influential article on the “resource curse” and civil conflict. The assignment was developed for a small, month-long, intensive course in research design and resource conflict for students at a large private research university.² Although the experience was shaped by the course parameters, this type of research project can be applied more broadly for different topics and a range of course contexts, including more quantitative-oriented research methods courses at the undergraduate or graduate level. The article concludes with suggestions for how instructors can adapt the exercise to these contexts.

QUALITATIVE REPLICATION AS A PEDAGOGICAL PRACTICE

Replication is a practice identified by many as being a key modality for students learning research methods (Janz 2016). It functions through learning by doing; because students are using materials that have been vetted in some way (usually through publication), they are less likely to chase down infeasible projects. The practice also is useful in that it allows students who may be hesitant about the research process to do so with “training wheels” (Druckman 2015).

Although replication has largely been advocated when teaching quantitative methods (King 2006), this does not mean that it cannot or should not be applied to qualitative work. In fact, several authors advocate for the development of projects in which undergraduate students are trained explicitly in qualitative methods (Elman, Kapiszewski, and Kirilova 2015) and/or that are expressly intended to replicate previous qualitative work (Moravscik 2014).

I took these suggestions seriously, devising a project that would allow undergraduate students to perform a qualitative replication in the context of a research design course. Qualitative replication, although fairly rare, seemed ideal from a pedagogical perspective: students taking the course had varying levels of preparation in quantitative methods, and I wanted to keep the focus on the evaluation of information—not learning a particular programming language or statistical technique (Bernstein and Allen 2013). Focusing on qualitative work allowed for greater emphasis on research design and causal mechanisms. This emphasis also would make it appropriate for a research design course or one that is more focused on quantitative analysis; the identification and coding of observable implications of causal mechanisms is critical in any study designed to evaluate causal claims.

CREATING THE REPLICATION ASSIGNMENT

The assignment was designed to serve three purposes: (1) improve students’ understanding of research design and what creating a piece of academic research entails; (2) give them the experience of evaluating a research project, both substantively and in terms of the extent to which it engages in transparent research practices; and (3) allow them to work firsthand with qualitative case data to assess hypotheses of substantive importance.

For this exercise, I used several criteria to choose the study that the students would replicate. I specifically looked for an article that:

• • covered a topic of likely student interest

• • was part of an ongoing, robust scholarly debate

• • was sufficiently explicit in terms of case selection, operationalization of important concepts, and coding of cases

• • used a “medium-N” approach, providing a variety of cases for students to explore

Ultimately, I chose Ross’s (2004) article, “How Do Natural Resources Influence Civil War? Evidence from Thirteen Cases.” The article considers the relationship between natural resource wealth and civil war, an issue of great importance generally and one that, from experience, grabs students’ attention. It is a foundational work in the scholarly debate around this issue, and thoughtful critiques have been published about its methodology (e.g., Weller and Barnes 2016). Furthermore, with 13 cases and 13 hypotheses, there is sufficient material for students to work with in the replication exercise.

After the article was chosen, it was necessary to determine an appropriate process for the project and delineate questions for students to address at each stage. This involved creating an assignment and an assessment that capture students’ mastery of the material and application but are not dependent on a predetermined outcome for the replication process.

INTRODUCING THE ASSIGNMENT

First, I needed to orient the students to the issue under study and help them gain an understanding of qualitative research methods and research transparency. The level of preparation among students varied: some had worked on faculty research projects before, others were seeing the research process in action for the first time. This required a carefully thought-out approach to support the gathering of evidence and the coding more generally.

The opening module of the course focused on teaching the students about research design, with a particular focus on evaluating the different types of validity that might be claimed by an author of a study. It also was imperative to ensure that students felt comfortable reading and engaging with academic journal articles. They were required to read chapters from Powner’s (2014) book on how to read political science and how to engage with and in qualitative research. To facilitate their reading, I created an “empirical article checklist” (EAC) to guide them through a piece of research, ensuring that they identified the important aspects of the study’s research design and evidence. (The EAC is included in the online supplemental materials.)

We also discussed the importance of transparency and replication, highlighting Lupia and Elman’s (2014) categorizations of data access, production transparency, and analytic transparency. This framework is particularly useful for undergraduates because it lends itself to asking critical questions: Do authors make their data accessible? Do they provide a full account of the procedures used to collect or generate the data? Do they clearly explain the links between their empirical data and their conclusions? Students were asked to answer these questions to evaluate Ross’s article in terms of research transparency and to include their evaluation in their final paper.

This framework is particularly useful for undergraduates because it lends itself to asking critical questions: Do authors make their data accessible? Do they provide a full account of the procedures used to collect or generate the data? Do they clearly explain the links between their empirical data and their conclusions?

To facilitate the students’ work in a practical sense, they were given opportunities to meet with subject librarians at our university. Students were advised on the resources available to them through the university library, and they had a separate session on strategies for research that they might use in their replication.

Students also needed to be oriented to the substantive issues addressed in Ross’s article. This involved having them read several articles on the relationship between natural resource wealth and civil war, particularly those to which Ross was explicitly responding (Collier and Hoeffler 2004; Fearon and Laitin 2003) and those that responded to Ross (Humphreys 2005; Weller and Barnes 2016).

FACILITATING THE REPLICATION PROCESS

When the students felt more comfortable with the reading and became familiar with the literature on resources and conflict, I introduced the replication assignment, making clear that the purpose of the assignment was to engage in the process of research. I emphasized that this was not a “gotcha” mission and that their grade was not tied to a particular outcome but rather to their demonstration that they made a full-faith attempt to replicate the original study. (The full assignment is included in the online supplementary materials.)

The first step of the actual replication process was for students to read the Ross article individually and complete the EAC to the best of their ability. They were expected to come to class with their EAC completed and be ready to discuss their assessment with their classmates. Some parts of the EAC are more straightforward than others to answer; when there were differences among the students, they were encouraged to discuss their choices and come to a consensus. As the professor, I was prepared to act as final arbiter if a consensus could not be reached or if the students went off path, but I did not need to intervene as much as I had anticipated. There certainly was spirited debate! Instructors who want to apply this method in their own classroom should intervene as they see fit; however, a light hand is recommended so that students feel ownership in the project.

After the students had broken down the original article into its component parts, they thought about how to construct a codebook. Ross did not write a codebook for his article, but what clues could be found about how he made his coding decisions? When students reviewed original case material, how would they know which causal mechanisms were present and which were not? They painstakingly reviewed all 13 causal mechanisms covered by Ross and determined what counted as evidence of each mechanism. They tried to be as faithful to Ross as possible. As in the initial round of the EAC, the codebook was constructed based on consensus, with limited professor input. I weighed in when clarification or additional specificity in the coding rules was needed.

With the codebook set, I assigned specific cases to the students. Each student was asked to replicate a certain number of the cases (i.e., for our project, two cases from the 13 included in the original study). This way, multiple students were working on each case. Students were told explicitly not to work together on the casework, and I introduced the concept of intercoder reliability to explain the prohibition. Cases were assigned considering student interest in a particular case or region. For the most part, students were assigned at least one of their first-choice cases.

After they had been working on their own cases for a week,³ we arranged to meet with Ross.⁴ Although it may not be possible in all cases, instructors interested in designing a similar exercise should consider asking study authors if they are willing to share materials or meet with students. Our class found the interaction to be incredibly valuable, and several mentioned it in course evaluations as their favorite part. Before the meeting, I asked them to devise a series of questions for Ross, which we then vetted. We were fortunate in that Ross was willing to answer all of their questions with great candor, which gave them greater insight into the process of research. He also shared his case notes, which provided an additional source of material for the replication. The students’ experiences were greatly enriched by Ross’s cooperation.

Although it may not be possible in all cases, instructors interested in designing a similar exercise should consider asking study authors if they are willing to share materials or meet with students.

To facilitate the students’ work on their final project, I held regular progress meetings, both individually and with the entire class. This was helpful not only for completing the replication but also for thinking through implications of various research design choices and devising creative solutions for finding more data to include in future analysis. As students gained confidence in their grasp of the material and the research design issues, they generated ideas for how we might extend Ross’s analysis to an additional set of cases, including those ideas in their final project reports.

Because of the course emphasis on research transparency, one major point that I stressed was documentation. Students were required to create a digital data appendix that included the information informing their coding choices to provide data access and production transparency. They also were asked to submit a set of coding notes that explained their choices in evaluating each of the 13 hypotheses, providing analytic transparency. We discussed how to organize and present the data appendix so that it would be useful for others, and students reviewed sample appendices.

As they completed the replication, students were asked to prepare a one- or two-page reflection on the experience as part of their final project submission to encourage metacognitive skills and to preserve the stated focus on process over outcome. They were assessed based on how well their final product achieved the goals of data access, production transparency, and analytic transparency (Lupia and Elman 2014).

APPLICATION IN OTHER CONTEXTS

Instructors interested in using this method of teaching about research should apply the process to research in their own areas of expertise. The assignment is agnostic in terms of the topic of the paper; however, for practical purposes, I recommend that the chosen study have at least some of the following attributes. First, qualitative work that is “medium-N” is more likely to engage students because they can work on different cases and compare and contrast their experiences in coding different countries, time periods, and so forth. However, it ultimately is up to instructors to decide how many cases are appropriate for their class size.

Second, the work should be on an issue currently under debate with “real-world” implications. Although few academic debates are definitively “settled,” students are energized by the idea of contributing to our knowledge of a topic—particularly if it has direct policy implications. Instructors would do well to seek topics within their academic expertise that are more likely to engage students.⁵

Third, instructors should ensure that any potential replication does not require access to specialized sources that are unavailable to students. For this reason, studies covering more contemporary cases may be preferable because students may not have access to archival materials. Furthermore, instructors are advised to verify that the documents necessary to complete the study are available in English or in an easily accessible translated form. This is an area in which university librarians can be excellent resources because they can direct instructors to resources, archives, and other special collections on campus that students can use for their project.

Not every piece of research considered for replication will have all of these characteristics. In my areas of research, Fortna’s (2004) book about variation in ceasefire agreements in civil wars and how they might lead to a more or less durable peace, and Haggard and Kaufman’s (2012) article on democratic transitions, may be fruitful source material for qualitative-replication projects. Ultimately, it is incumbent on instructors to choose the piece that they believe is most appropriate for their students.

Additionally, not all classes are the same in size, time span, and level of preparation among students. Again, my course was a small, one-month intensive upper-division course, which meant that I had sufficient time to work with students to ensure that they were well prepared for the task.

Nevertheless, the project is applicable in various course contexts. For example, an instructor might want to apply the assignment in a larger class. The assignment is sufficiently flexible to allow for students to work in teams, which should not jeopardize their educational experience.⁶ For example, the part of the assignment dedicated to devising a codebook could be assigned to teams of four to six students, or it could be an exercise in a TA-led discussion section.

This project also may be applied to a lower-division class or adjusted for students with varying levels of preparation. One advantage of the assignment is that it can be used modularly, dividing tasks as appropriate. Instructors may ask students to use the EAC as they read or apply the concepts described by Elman, Kapiszewski, and Lupia (2018) to evaluate the level of research transparency of a particular article. Instructors might ask students to read a codebook and assess a particular set of coding decisions made by an author or create a digital data appendix for a paper.

This method of teaching students about the practice of qualitative research through replication also may be useful for graduate study (Moravcsik 2014). Graduate students are unlikely to have had training in qualitative methods during their undergraduate career because the major requirements of political science programs vary greatly in terms of whether they require a research methods course and the type of methods that are taught in it (Turner and Thies 2009). Having early-career PhD students engage in replication can generate fruitful discussions on case selection,⁷ coding, and analysis that can help students develop the skills needed for their own future research projects.

Whatever the context or specific topic of study, qualitative replication is a pedagogical practice that can open up the process of research to students, empowering and inspiring them to see themselves as researchers and contributing to our collective knowledge of the political world.