Social Science Mechanics: A Graduate Training Module that “Looks under the Hood” at Innovative Research Designs

Abstract

Scholars in the fields of instructional development and pedagogy note that learning outcomes can be improved when teachers use “narratives” to communicate how complex processes work or how problems are addressed. In this article, the authors describe a narrative-centered approach to graduate-level instruction in research methodology. This approach is intended to supplement, not replace, conventional graduate seminars in quantitative or qualitative methods. In a series of lectures, scholars reflected on how their published articles originally were framed, the trade-offs that were necessary to advance the investigation, the methodological challenges and non-findings that had to be addressed—but may not have been printed—and the evolution of a piece as it progressed through the peer-review stages. This approach to exposing graduate students to the entirety of the research process is termed Social Science Mechanics: A Look under the Hood at Innovative Research Designs. Surveys used to evaluate the series confirmed that graduate students who attended the presentations found them to be highly engaging and beneficial. Many faculty members also attended and found the lectures to be equally instructive.

Methods-training courses in political science graduate programs typically follow a standard formula. Instructors focus primarily on building proficiency in selected quantitative or qualitative analytic tools rather than fostering a comprehensive understanding of the research process itself, from the identification of interesting questions, to the creation of a research design, to the ultimate dissemination of findings. An inventory of the graduate methodology syllabi posted at the Political Methodology website archive (available at http://polmeth.wustl.edu/index.php) indicates that the tools imparted in graduate seminars are designed primarily to equip students with the skills needed to understand the empirical literature and to add to it: time-series analysis, structural-equation modeling, Bayesian inference, panel-data methods, discourse analysis, and so on. While an extensive toolkit is essential, it is equally important for course participants to understand how a given methodological technique is applied in practice and which adjustments must be made when gathering data, estimating parameters, and—ultimately—developing a persuasive case for readers. Choosing how to specify a time-series function, for example, may involve dozens of operational decisions, any one of which might be debated when the work is presented and published. How to navigate this process—that is, the specification of a model, its presentation to various audiences, and the give and take that is inherent in any reviewing process—is truly news that can be used in graduate school. However, these challenges often are not covered in doctoral-level methodology training programs. Instead, most academics likely learn these lessons through “the school of hard knocks.”

Consequently, even if graduate students have “aced” all of their methodology seminars, they often have not acquired an understanding of the vagaries of conducting research. Takata and Leiting (1987, 144) noted that “the traditional research methods course is extremely limited because…it does not explain the realities of research.” This problem likely is compounded by the limited scope of many methods courses. Typically led by a single instructor who is highly specialized in certain methodological frameworks, courses may lack firsthand guidance in utilizing a wide variety of methods—and could not possibly expose students to the multitude of methodological choices available in a specific context. As a result, key information about the process of selecting and implementing methodologies often is missing in graduate methodology seminars.

In this article, we discuss an attempt to address this gap in graduate-level pedagogy through an approach based in the retrospective assessment of a major research piece. In a series of lectures, scholars reflected on how their published articles or books were framed originally, the trade-offs that were necessary to advance the investigation, the methodological challenges and non-findings that had to be addressed—but may not have been printed—and the evolution of the piece as it progressed through peer-review stages. This approach to exposing graduate students to the entire research process is termed Social Science Mechanics: A Look under the Hood at Innovative Research Designs.

LITERATURE REVIEW

Relatively little has been written about teaching methods for graduate-level methodology courses in political science. The limited work in this area tends to focus on what—rather than how—graduate students are taught (e.g., Bennett, Barth, and Rutherford 2003). Although these discussions are useful in understanding methodological standards and expectations in the discipline, they are less beneficial for understanding the impact of various pedagogical choices. Another body of literature focuses on the challenges of teaching undergraduate methods (e.g., Andersen and Harsell 2005; Bernstein and Allen 2013; Bos and Schneider 2009; Brandon et al. 2006; Brown, Blankenship, and Kramer 2007; Centellas 2011; Thies and Hogan 2005). This is a noteworthy body of work; however, to be successful in the discipline, graduate students must become knowledge generators and disseminators—a point not often at issue with undergraduates, who most commonly find methods to be irrelevant or, in regard to quantitative methods, mathematically overwhelming (e.g., Bos and Schneider 2009).

Almost every graduate student in political science, however, has read the ubiquitous Designing Social Inquiry (King, Keohane, and Verba 1994), often called “KKV” by disciplinary insiders. While KKV’s recommendations for qualitative research led to a wide-ranging debate, we highlight the following observation that would generate little controversy:

Once an investigator has collected data as provided by a research design, he or she will find an imperfect fit among the main research questions, the theory and the data at hand. At this stage, researchers often become discouraged. They mistakenly believe that other social scientists find close, immediate fits between data and research. This perception is due to the fact that investigators often take down the scaffolding after putting up their intellectual buildings, leaving little trace of the agony and uncertainty of construction. Thus, the process of inquiry seems more mechanical and cut-and-dried than it actually is. (King, Keohane, and Verba 1994, 13)

Why, then, in graduate methods curricula does the discipline typically focus on the elegance of the final publication rather than on all of the mistakes it took to arrive at that point?

This perfunctory treatment of the research process aptly describes the perception of many graduate students who commonly begin their methodological training by reading the published, polished work of scholars who have erased all of the trials and tribulations from their end product. How, a graduate student may wonder, can such a level of scholarly perfection ever be reached? Of course, such a perspective does not reflect the reality of social science research. As KKV (1994, 182) noted, “[P]erfection is unattainable, perhaps even undefinable.” Instead, research often is messy and requires creativity more than perfection. Why, then, in graduate methods curricula does the discipline typically focus on the elegance of the final publication rather than on all of the mistakes it took to arrive at that point? Perhaps it is because most researchers are reluctant to reveal their gaffes or, perhaps—as we discovered—they simply have not been asked about these challenges. The goal of the seminar series described in this article was to uncover the missteps, the challenges, the dead ends, and the insights that lead to innovative research. We believe that such a narrative will better prepare graduate students for their own research journey.

To understand the potential benefits of such a pedagogical approach, we need to consider the body of literature on narrative-centered learning. The construction of a story is one of the most fundamental ways to create shared meaning, an activity central to all aspects of learning (Bruner 1986; Clark and Rossiter 2008; Connelly and Clandinin 1990; Wells 2009; White 1980). A narrative-centered learning environment has two elements: transportation and performance (Mott and Lester 2006). By transporting learners into a plot full of characters and ideas, this learning environment enables students to participate in active problem solving and to draw their own inferences (Mott and Lester 2006, 676). Furthermore, the communication of a narrative plays a central role in memory by providing an organizing structure for new experiences and knowledge (Mandler 1984). As Schank and Berman (2002, 293) stated, “for communication, memory, and learning purposes, stories are likely to be richer, more compelling, and more memorable than the abstracted points we ultimately intend to convey or learn when we converse with others.”

According to Mott and Lester (2006), a narrative-centered learning environment is particularly suited to inquiry-based learning that emphasizes concept building, hypothesis formation, data collection, and testing. Mott et al. (1999) noted that “in the sciences, an inquiry-based curriculum featuring dynamic narratives of the highly nonlinear process of scientific discovery could foster an in-depth understanding of how real-world science plays out.” For graduate students who must learn how to conduct their own research in the disorderly reality of social science, such an endorsement seems particularly relevant. Mott et al. (1999) discussed three types of narrative-centered learning activities: co-construction, exploration, and reflection. In our seminar series, we focused primarily on reflection (i.e., activities that engage post-hoc analysis) and exploration (i.e., activities that consider how intentions affect actions in the evolving narrative). Co-construction (i.e., activities that allow participation in narrative creation) also may be a useful strategy in graduate-level instruction—for example, by creating alternative stories about how a potential research question might be addressed. However, this particular type of narrative-centered learning was beyond the scope of our seminar series, which asked scholars to relate their own version of events and thereby bring to life to the research process. In hearing and reflecting on the narratives of these scholars, we expected that attendees would be more cognizant and open about their own evolving research path.

BACKGROUND OF THE SEMINAR SERIES

In the fall of 2011, the Political Science Graduate Student Association at Purdue University convened a working group, which included a faculty member who regularly taught required methods courses. The purpose of the working group was to discuss issues in the current methodological training of graduate students as well as possible remedies. From the discussions, it became clear that the current methods curriculum did not offer graduate students a panoramic view of the research process; neither did it provide access to scholars with diverse research experiences. This gap was not being filled by research mentorships with faculty. As is the case in many departments, most students were teaching assistants, independent instructors, or on independent fellowships. Although some students had participated in limited projects with faculty (e.g., data collection), only a few had participated in the entire research process of any project. With limited resources, how could this deficiency be remedied?

We devised a lecture series to provide a more comprehensive “look under the hood” at social science research. The goal was to provide opportunities for students to examine the research process through actual experiences of established scholars. We did not aim for a conventional lecture series in which a researcher presents a work-in-progress for audience consideration and feedback. Instead, the focus of the “Look under the Hood” colloquium was to be on a piece that had already been published in a major peer-reviewed journal. Using the publication as the end point, authors presenting at sessions would be asked to describe their research process from the beginning, giving particular attention to the conceptual genesis of the project, the rationale for key methodological choices, and the misjudgments made along the way. Such a narrative meant that speakers covered material that went well beyond the traditional research presentation—and even beyond a typical class session in a graduate methodology seminar. Our expectation—which was confirmed when the series began—was that after authors had successfully published a piece in a visible scholarly outlet and had received all due professional recognition, they would be open to having a frank conversation about the “scaffolding,” “agony,” and “uncertainty” that surrounded the piece.

Our expectation …was that after authors had successfully published a piece in a visible scholarly outlet and had received all due professional recognition, they would be open to having a frank conversation about the “scaffolding,” “agony,” and “uncertainty” that surrounded the piece.

Given the unusual nature of these lectures, a rubric was prepared for the authors who were invited to present a piece. Under this rubric, each presentation was to be structured as a retrospective account of a specific research project. Speakers would begin by citing their source of inspiration, whether it was the result of—for example—a debate with a peer or an attempt to fill a gap in the existing literature. At this point, researchers could share the relevant literature; however, an extensive overview of the literature was discouraged. Instead, we encouraged them to move from a discussion of the formation of the research question to a detailed explanation of how the approach to analyzing it was chosen. Here, speakers would address method selection and application. Although the series was not structured as a class, we asked authors to discuss the virtues of the particular methods used in their article as well as how empirical analysis and interpretations changed in response to comments from colleagues, panel discussants at professional conferences, editors, and anonymous peer reviewers.

With these goals in mind, several examples of innovative work in the social sciences were selected. Care was taken to ensure that the group of articles represented a variety of different methods as well as different academic fields. Some of the methods (e.g., social-network analysis and experimental methods) are not regularly taught in our current curriculum, which increased the value of these seminars. The seven-part series was held in the spring of 2012 on the Purdue University campus. Almost all of the authors who were invited to present at a “Look under the Hood” colloquium agreed to come and present the back story for their already-published piece. ¹ Each session was scheduled for three hours, which included a brief reception for the presenter, the retrospective narrative, and a lengthy informal question-and-answer (Q&A) period. The series was well attended not only by graduate students but also by many faculty members.

ASSESSMENT OF THE SPEAKER SERIES

To assess the seminar series, attendees completed a brief anonymous survey after each lecture. The survey consisted of two parts. In the first part, respondents evaluated their general level of agreement with the following five statements (i.e., 1 = completely disagree; 10 = completely agree):

• S1 I perceive this seminar as useful for professional development.

• S2 This seminar is relevant to my research interests.

• S3 I enjoyed today’s presentation.

• S4 This presentation is appropriate for this seminar.

• S5 I feel that the methods used in this seminar will be useful for future personal research projects.

The second part of the survey solicited open-ended feedback. A total of 91 questionnaires were returned for the seven presentations. The mean scores for the five evaluative statements were generally positive, particularly with respect to professional development (i.e., M = 9.15); other averages were above 7.5 with relatively little variation in responses (figure 1). In the open-ended section of the survey, one student wrote that he or she attended to learn “how methods are applied in real research.”

Figure 1 Average Survey Response for Seminar Series by Statement

Note: There are 91 total responses for each statement, with the exception of statement 5, which includes 88 responses.

We note that the seminar taught more than research methods. During the series, speakers addressed professionalization, conference presentations, and—in some cases—the lengthy journey from research question to published article. Students expressed great interest in these topics. In the Q&A period, they frequently asked for more “insider” details on working with coauthors (e.g., “How were the tasks divided?”); the targeting of particular journal outlets (e.g., “Was there a reason you did not attempt to publish in _____?”); and challenges when revising for resubmission (e.g., “Was there a point when you considered pulling the manuscript, given the seemingly impossible-to-satisfy Reader 3?”). In one survey response, a graduate student emphasized that “it was valuable to hear from published authors and what they did in order to get a major publication through the review process.” Several faculty members who attended the seminar mentioned how reassuring it was to hear that their own challenges in publishing were not unique. One wrote in the assessment survey that the “Look under the Hood” series offered “encouragement and help” at his or her particular career stage.

DISCUSSION AND IMPLICATIONS

Overall, this series exposed students to the process of discovery in political science, “warts and all,” and provided opportunities to discuss methodological techniques that might not have arisen in our standard graduate training curriculum. As one graduate student remarked in the survey, “I have been struggling with a problem in my current research project and I believe that today’s method [i.e., bootstrapping] can help me fix this problem.”

…authors of major scholarly works generally enjoy discussing the evolution of a piece and that these reflections offer valuable lessons for novice researchers.

Although some students believed that certain highlighted methods were not applicable to their own research program, the series provided useful information on professionalization and publishing. In some cases, survey responses indicated that the opportunity to learn methods was not the primary reason for attending the seminar—some attendees came to learn about submitting work for publication and others to find solutions to common problems in conducting research programs. One student wrote that he or she attended to “learn about problems with writing research papers, especially the publishing process.”

Educators throughout the academic disciplines agree that to improve learning outcomes, traditional “chalk-and-talk” instruction should be supplemented with “active learning”—that is, opportunities to learn while doing. When teaching graduate students about the principles of research design and causal inference, the selection of appropriate tools for empirical analysis, the write-up of results, and the challenges of publishing, the retrospective narratives in the “Look under the Hood” series constituted a form of vicarious active learning. Our experience demonstrates that authors of major scholarly works generally enjoy discussing the evolution of a piece and that these reflections offer valuable lessons for novice researchers.