Shadowing as a Tool for Studying Political Elites

Abstract

This article offers a description and discussion of “shadowing” as a data collection and analytic tool, highlighting potential research opportunities related to the direct observation of individuals—principally political elites—in their normal daily routine for an extended period of time, often between one day and one week. In contrast with large-scale data collection methods, including surveys, shadowing enables researchers to develop detailed observations of political behavior that are not limited by the availability of administrative data or the constraints of a questionnaire or an interview guide. Unlike more in-depth qualitative methods, such as ethnography, shadowing is scalable in a manner that allows for larger sample sizes and the potential for medium-N inference. I provide a detailed account of how to design and conduct a shadowing study, including sampling strategies, techniques for coding shadowing data, and processes for drawing inferences about the behavior of shadowed subjects, drawing on examples from a completed shadowing-based study. I also discuss ways to mitigate selection and observer biases, presenting results that suggest these can be no more pronounced when shadowing political elites than in other forms of observational research.

1 Introduction

Political elites—elected officials, government appointees, and public activists—are the focus of substantial work in political science. Researchers analyze the decisions politicians make over legislative strategies, the character of bureaucrats’ interactions with the public, and the techniques of social movement leaders. Yet, detailed, direct observations of behavior by political elites are rare and, where they occur, often constrained on multiple dimensions. Large-N observations, such as roll call votes and data from audit experiments, offer important insights into specific domains of behavior yet provide limited information on a broader range of elite activities. Small-N, qualitative observation techniques, such as ethnography and interviews, instead offer the opportunity to observe and investigate a wide set of behaviors, but typically on a much smaller sample of individuals, thus limiting the potential generalizability of inferences. ¹ This suggests that there may be benefits from adopting tools of observation and data collection that can feasibly leverage the benefits of studying many individuals at a time, while maximizing the amount and quality of data collected on each individual.

A particularly valuable tool for achieving this goal is in-depth shadowing. The shadowing technique involves direct observation of individuals in their normal daily routine for an extended period of time, often between one day and one week. During this time, the subject’s activities are recorded in a regular and meticulous fashion, at prespecified intervals. The resulting field notes are subsequently coded and analyzed, and the de-identified data can be made available to the public through standard data-sharing platforms. Shadowing enables researchers to develop detailed observations of political behavior that are not limited by the availability of administrative data or the constraints of a questionnaire or an interview guide. Simultaneously, the intermediate time frame entailed by the shadowing model allows for observation of a larger sample of individuals than is feasible with traditional ethnography.

In this article, I provide a description and discussion of shadowing as a data collection tool, including strategies for coding shadowing data and drawing inferences. Shadowing is not new to political science: Fenno’s work on US legislators in the 1970s provides perhaps the best-known example of related methods, and much other work utilizes elements of a “shadowing” methodology. ² Yet, there does not currently exist a guide for conducting shadowing in a rigorous and transparent manner. ³ In addition, existing methodological attention to shadowing fails to highlight the explicit benefits and costs of shadowing for the observation of political elites. ⁴

My contribution therefore differs from the existing work on several important dimensions. Overall, I offer a comprehensive set of standards for the design and implementation of shadowing studies, including unique elements intended to maximize the potential for drawing inferences from shadowing data. First, I provide a model for study designs that enables data collection by multiple shadowers to increase the potential for medium-N analysis. Second, I present coding standards for field materials that increase rigor in the analysis and inference phases of a study. Third, I offer recommendations to increase opportunities for transparency in field notes and replication of results. I also include a comprehensive discussion of why shadowing is particularly appropriate for studying political elites. As a whole, this standardized but flexible model for data collection and analysis through shadowing offers new prospects for improving our understanding of, and inferences about, elite political behavior. ⁵

In the following sections, I elaborate on shadowing as an analytic technique. I first discuss the contributions of shadowing, before moving to a framework including study design, shadower training, fieldwork practicalities, coding, analysis, and writing of narratives. I then consider challenges to the implementation of shadowing, emphasizing issues of selection bias and observer effects, and present strategies for evaluating and mitigating these risks. Throughout the discussion, I draw on multiple examples of shadowing-based research, including a study of Indian political elites, provided as a companion to this article, to illustrate various points. ⁶ I conclude with a discussion of how shadowing integrates multiple research methods and can be integrated with other research tools.

2 Shadowing as a Methodological Technique

Fundamentally, shadowing involves the observation of an individual during her regular, daily activities. The “shadower” is generally a passive observer, rather than a participant, with some exceptions discussed below. The goal of this observer is to document in a comprehensive manner the subject’s activities, the substance of the subject’s conversations and meetings, and the reasons the subject offers for why she does some things, or makes some choices, and not others.

Shadowing emerged in studies of individuals within organizations and has been used across a range of disciplines, including management, education, and communication. ⁷ A recent review discusses more than forty studies using variants of a shadowing methodology. ⁸ This work has frequently focused on business and social elites, such as CEOs, managers, and religious leaders, across a range of geographic locations, including the United Kingdom, Canada, and Sweden. ⁹ Politicians have also been the subjects of shadowing studies. ¹⁰ Thus, shadowing has a rich history across a large scope of elite occupations and geographic contexts.

The research questions posed in shadowing studies, while reflecting persistent topics of inquiry in the social sciences, are also quite diverse. How do politicians acquire information for policy-making and what are the implications for decision-making? ¹¹ How does the implementation of a new policy change individual behavior? ¹² How do a set of actors’ actions compare with their officially defined roles? ¹³ How do executive managers plan, strategize, and act? ¹⁴ How do institutional rules shape interactions in a hierarchical organization? ¹⁵ How do constituency visits contribute to politicians’ electoral strategies? ¹⁶ These topics also suggest that there is a wide range of unanswered questions about elite political behavior—in domains such as principal-agent dynamics or the relationship between institutional constraints and strategic behavior—that could be productively addressed through application of shadowing methods.

As a technique for data collection, shadowing offers unique benefits to the researcher relative to a range of other strategies. With large-n studies based on large surveys, administrative data, or public documents, the content of the dataset is limited by the survey instrument, the topic of the bureaucratic data collection, or choices over what information to release publicly. While techniques such as open-ended questions in surveys can generate unexpected insights, shadowing may allow for the gathering of rich, detailed insights, including about those areas with which the researcher was not already familiar. Where a researcher is concerned with a wide range of potential activities, shadowing can thus be particularly helpful. In addition, shadowing can be used to gather information on how different aspects of subjects’ activities relate to each other, which is often difficult with preset questions and administrative data.

Relative to other qualitative data collection techniques, such as ethnography and interviews, shadowing can be especially useful when researchers are interested in the generalizability of their findings. This is most feasible when a rigorous sampling technique is used to generate a medium-N sample, with the sampling frame identified in advance and a random sampling procedure used to generate the proposed set of respondents. Where this is done, the findings from a medium-N shadowing study can reasonably be generalized to a broader population. Where the unit of observation is respondent-time, as discussed further later, shadowing field notes can especially fruitfully be analyzed using techniques for large-n data.

Shadowing can be used at any research stage. Early in a research project, shadowing can help to establish a descriptive understanding of the context and individuals of interest. Alternatively, shadowing can be used later in a project to explore previously established hypotheses, such as those derived from patterns in large-n observational or experimental data. ¹⁷

Finally, even as shadowing has been productively used to study private sector elites, it is particularly appropriate for the study of political elites, with regard to their professional activities conducted in both public and private. While in other subject areas potential shadowees operate primarily outside the public eye, such as in a corporation or private institution, political elites, and especially public officials, spend a considerable portion of their time in public. ¹⁸ The substance of elites’ public behavior is often of great interest to political scientists. As Fenno showed in his study of US legislators, choices that a politician makes over which events to attend and where to spend time in the constituency can be quite illuminating regarding a politician’s personal preferences and electoral strategies. ¹⁹ Thus, shadowing offers entrée into activities that are substantively relevant but that generally cannot be observed through the use of administrative data. ²⁰ In addition, it is not unusual for politicians to have multiple individuals with them throughout the course of their daily professional activities. In this sense, shadowing may often be less intrusive for politicians than it might be for other types of professionals.

At the same time, politicians also conduct substantial activities in private. Shadowers can, theoretically, continue their observations during the portion of an official’s time when they are not in public. While subjects may not allow an observer to join them in all private meetings, often the shadowing relationship generates a dynamic that results in increased access for the shadower. This potential access to nonpublic domains of a subject has two important substantive implications. First, the shadower will be able to gain information that is traditionally more difficult to observe than a politician’s public events, which may reasonably be studied through attendance at public functions in ways that do not require direct engagement with a subject. For example, who attends private meetings and what are the topics that are discussed? Second, the choices that a subject makes over which private events the shadower may not attend are themselves potentially relevant to the researcher. Does the subject allow her observer to attend meetings with bureaucrats, but not those with party workers? What does this imply about the subject’s preferences? What explanations do shadowees give for these choices? ²¹

Despite these potential benefits of shadowing in general, and in particular for studying political elites, we lack a framework for implementing such studies rigorously, and with attention to current standards for transparency in empirical research. ²² A methodological literature on shadowing, including typologies of existing shadowing-style work and examinations of trends, best practices, and ethical considerations, does inform my discussion here. ²³ Yet, this work lacks guidelines for designing, conducting, and analyzing a shadowing engagement. ²⁴ It also does not adequately address key issues and potential areas of concern, such as selection and observer bias.

3 Guidelines for Shadowing Studies

To address these limitations, I now turn to my recommendations for designing and reporting a shadowing study, drawing on examples from shadowing-based literature. The core of these recommendations is a formal study design, which is key to achieving research goals through shadowing. A study design should include at least five core elements: research question(s), definition of proposed subjects, sampling plan, implementation strategy, and data analysis plan. This design can serve as a pre-analysis plan and be registered in advance of implementation. ²⁵ While previous work has discussed some of these elements, none has described how they should be prepared and connected in a study design prior to initiating research. I conclude the section with a discussion on drawing inferences and writing up a shadowing study.

The example Study Design provided in the Supplementary Materials introduces a study of politicians in India that serves throughout this article as an example of the use of shadowing. This study concerns how local political leaders in India engage as politicians in their constituencies and draws on data from the shadowing of fourteen village council presidents (VCPs) in the state of Uttar Pradesh. I present here extracts from this study, referred to hereafter as the “VCP study,” while providing preliminary materials and a completed paper in the Supplementary Materials. Readers who would like to review the details of a “start-to-finish” shadowing study alongside this article are encouraged to refer to those additional documents. ²⁶

3.1 Research Question(s)

While shadowing allows for the collection of a broad range of qualitative data, these data should nonetheless be amassed within the context of defined research objectives. The study design provides an opportunity to lay out specific questions to be answered with shadowing data. For example, in his study of management strategies, Noël asks how CEOs influence corporate strategy formation through their daily activities. ²⁷ In the VCP study, I ask: do local presidents prioritize different types of professional activities according to personal interests? ²⁸

3.2 Definition of Subjects

A viable shadowing study must clearly define the proposed subjects of observation. For example, “local politicians” is likely to be insufficiently specific, unless the researcher intends to study locally elected representatives globally. A more precise definition, such as village council politicians in India, can be linked both to a study’s goals and to the sampling plan, as discussed in the next subsection. In the VCP study, the subjects are the leaders of gram panchayats (village councils) in a single Indian state. ²⁹ In Orton et al.’s study of politicians’ information-seeking behaviors, the subject population was Members of Parliament in the United Kingdom. ³⁰

3.3 Sampling Plan

The type of sample to be collected is dependent on the goals of the researcher and the characteristics of the study population. A researcher may utilize a census, random sample, or snowball sample to acquire the desired subject pool for observation.

If the proposed population is relatively small, shadowing a census of all relevant actors may be feasible—for example, city council members in a single city, or mayors in US cities with populations over one million people. The benefit of this strategy is that it should allow for inferences about the total population of interest, conditional on response rates. ³¹ The conditions under which a complete population of interest may be small enough to shadow may, however, be limited, thus suggesting the need for alternative sampling strategies.

To the extent that a study has generalizability as a goal, and to the extent that there is a known population that is too large to be included in its entirety and from which individuals can be sampled, it is most appropriate to select potential subjects via a randomized sample. In Uttar Pradesh, there are approximately 52,000 local village councils. Shadowing all council presidents is not feasible, but a list of those individuals can be used to take a random sample, which, if reasonably large, can enable sufficient statistical power for generalizing to the population.

The benefit of a random sample, as in other forms of data collection, is that a researcher may often reliably draw inferences from the sampled population to the population as a whole. Risks associated with nonresponse hold in this context as well as elsewhere, and I consider these risks below. That said, random samples offer the strongest model for drawing general inferences to a large population when using shadowing for data collection. In the VCP study, I paired shadowing of village council presidents with that of state legislators, and randomly sampled local presidents from within the sampled legislators’ constituencies. In other cases, such as Fenno’s study of US state legislators, researchers attempt to build a “representative” sample in an ad hoc manner, based on characteristics of interest to the researcher, but this is unlikely to produce a final sample from which inferences can reliably be drawn to a broader population. ³²

Where the population is large but does not facilitate random sampling, such as when the bounds of the population are unknown or there is no available list from which to sample, a researcher may instead use a snowball sampling approach. In this model, the researcher identifies a few initial “typical” subjects to shadow, based on any available information about the subjects of interest, and then uses these subjects’ contacts to build a larger pool of subjects. Here, inferences can only readily be made to those individuals who were shadowed. Nonetheless, snowball sampling may offer a useful strategy when the population is not easily identifiable.

3.4 Implementation Strategy

An implementation strategy includes both when and where shadowing will take place and who will do the shadowing. The shadowing environment should reflect the research questions. For example, politicians can be shadowed in the legislature or in their constituency; shadowing can be targeted at particular political periods, such as the middle of a term or the lead-up to an election; and politicians can be shadowed during specific legislative moments, such as periods of debate over a policy. Thus, the context of a study can, and should, be altered to increase the likelihood of observing behaviors of interest. Designs that do not maximize observation time in the relevant context will result in inefficient and unnecessarily costly data collection.

Core to a shadowing implementation strategy is also a plan for who will be collecting data and a set of instructions for doing so. ³³ First, while many researchers may choose to conduct shadowing engagements themselves, shadowing projects are also feasibly implemented with a team of shadowers. For studies targeting a medium-N sample of respondents, this is at least in part because achieving scale can be quite difficult for an individual researcher. Each shadowing study is, by definition, going to take at least a few days in the field, if not a week or more. This implies an extended period in the field, which has both practical and substantive implications. Practically, many researchers may find the time frame for fieldwork difficult, due to work or personal obligations. One solution to this limitation would be to intersperse shadowing across multiple trips. Yet this strategy reduces a researcher’s ability to collect data on all subjects during the same time period. This prevents the researcher, at a minimum, from drawing inferences about the behavior of politicians in response to the same external events.

I suggest that the use of a shadower team is often key to alleviating challenges to achieving scale in shadowing studies. As with surveys, using a team for data collection greatly maximizes the amount of work that can be done in a given time frame. For example, a study with twenty subjects to be shadowed over one week each would require nearly five months in the field for a single researcher versus only a month and a half by a team of three shadowers.

Additionally, utilizing a shadower team can reduce overall costs, particularly in environments where the primary researcher would require a translator. In these conditions, a shadower team comprised of native speakers can complete the studies at half the overall cost of a researcher plus translator team. While training costs can add to the overall budget of the project, in many cases this will not offset the overall financial benefits of using a team.

A full implementation plan should be outlined in a personal fieldwork guide, if a single individual is conducting the shadowing, or a detailed training guide, if there will be a team of shadowers. A training guide is particularly important for ensuring the standardization of data collection across shadowers. I offer an example training guide in the Supplementary Materials, with guidelines here, and these recommendations hold equally well for developing a personal guide. ³⁴ These guides should include, at a minimum, background on the project, an overview of the shadowing technique, logistical information, guides for note-taking, guides on asking questions, and advice for dealing with unexpected and/or difficult events.

At the core of a training guide is the set of instructions for how to collect data. An example of general instructions on note-taking, drawn from the VCP study guide, is as follows:

Written notes are the primary form of documentation for shadowing. It is fundamental to the success of the project that investigators take detailed and comprehensive notes while they are in the field. Thus, it is important to make observations on any and every thing that the investigator sees and experiences while shadowing. This includes, at a minimum:

• ∙ What are the settings where you are observing during the day (e.g. politician’s house, public market, village, constituent home)? Make note of the location and any relevant details (e.g. the name of the village).

• ∙ What is the content of an interaction? What are people talking about? Does the conversation stay on that topic, or does the topic change? If someone is making a request of the politician, who are they, what are they asking for, and for whom? What is the politician’s response to these discussions/requests?

• ∙ What is the mood? Are people friendly, confrontational, respectful, etc.?

• ∙ What is your own impression of an interaction? How does it make you feel?

  What are your initial impressions of the politician and his/her associates? Do these impressions change during the shadowing period? If so, how and why? ³⁵

The full set of instructions for note-taking should include (1) the minimum intervals at which observations must be made, (2) the format in which notes should be taken, and (3) any specific substance that is required in the notes. For example, shadowers may be asked to document activities at a maximum of every fifteen minutes in a format that includes columns for time, activities, and any comments or questions from the shadower, as shown in Figure 1. ³⁶ Researchers may also request additional data, such as information on other actors who are present, and there should be a specific format for collecting these data. While the format for data collection can differ across shadowing studies, lack of a common format within a study will lead to substantial difficulties in later coding processes.

Figure 1. Example Data Collection Format with Hypothetical Content (VCP Study—Training Guide).

An additional element of a training guide is a set of directives for the conditions under which a shadower may insert themselves directly into a subject’s activities. For example, are shadowers empowered to ask the subjects substantive questions and, if so, on what set of topics? Can or should shadowers take photographs or collect other forms of documentation while they are in the field? ³⁷ Failure to address these topics may lead to substantial variation in data collection across shadowers. The example training guide offers illustrations of how these topics can be addressed up front when instructing a shadowing team.

A final recommendation for the guide is a set of instructions for how shadowers should deal with the kinds of unexpected and potentially ethically, physically, or mentally challenging events that are often inevitable in fieldwork. At the very least, there should be a statement about health and safety, which outlines the researcher’s expectations about the conditions under which shadowers should remove themselves from the fieldwork location, or temporarily pause observations. Even for unlikely situations, it is useful to have expectations set out in advance. ³⁸ The guide may also include recommendations for logistics during shadowing. In general, as shadowing is a form of fieldwork, resources more generally focused on the practice of fieldwork should also provide a useful complement to the shadowing-specific discussion offered here. ³⁹

3.5 Data Analysis Plan

Analysis of the material collected via shadowing involves, where necessary, translating and transcribing the shadowing observations, coding the observations, and then analyzing these materials. ⁴⁰ If observations are handwritten, they will need to be transcribed into an electronic format that facilitates data analysis. In the VCP study, observations were simultaneously translated into English and transcribed into a spreadsheet format that replicated the format for data collection. The closer the match in formatting between these two documents, the easier the transition between data collection and analysis.

3.5.1 Coding Strategies

A rich literature exists on coding techniques for qualitative data. I propose two general models, which may or may not be combined in a single project, based on a researcher’s overall objectives: researcher-driven qualitative coding and machine learning-based topic modeling. ⁴¹

The researcher-driven, qualitative coding model draws on the techniques described by Emerson et al., as well as Saldaña’s “first cycle” model, which I extend to support a multiple-shadower environment and the potential for quantitative analysis of observations drawn from shadowing. ⁴² This coding strategy involves one or two steps, depending on the stage of the research project. For an early stage research project, the researcher conducts an initial step, known as “open coding,” involving a preliminary review of the field notes to develop a sense for themes and topics. This involves taking a random sample of observations from the field notes, representing 10–20% of the overall corpus, ⁴³ and reading “field notes line-by-line to identify and formulate any and all ideas, themes, or issues they suggest, no matter how varied and disparate.” ⁴⁴ Two examples of randomly selected observations from the VCP study are:

“Pradhan and a ward member are walking and after few steps they stop in front of someone’s house and Pradhan said to the ward member to note down his name in the list of Prime Minister House Scheme. I asked Pradhan whose house is this? He told me whose house it is.” ⁴⁵

“Pradhan is distributing drought relief packet.” ⁴⁶

During this process, researchers should note words and phrases that characterize their interpretation of the material in the field notes. These “codes” may be informed by the goals of the study, but they may also emerge from the field materials themselves, allowing for new ideas and themes to be coded even where they were unanticipated by a researcher. This will result in a set of codes (words and phrases), which the researcher can, and often should, subsume into larger categories related to particular themes. These categories and subcategories should then be organized into a comprehensive list with descriptions and numbers, to serve as a coding guide. An extract from the coding guide for the VCP study is shown in Figure 2.

Figure 2. Example of Coding Instructions (VCP study).

The second stage involves applying this coding guide to the full set of shadowing materials. When a project is at a later stage, and the themes of the study are already established, this will be the first step of coding. This may be done by one or more coders, and the coding guide should be used as a set of instructions to enhance intercoder reliability in a coding team. ⁴⁷ A straightforward manner for doing this is to add a set of columns—one for each level of codes in the guide—to the spreadsheet with the shadowing observations. The coder then adds the appropriate code(s) for each observation in the columns for each row, as in Figure 3. This also allows for simple manipulation and reorganization of the codes for analysis. In the VCP study, there are five major themes for activities, seventeen subthemes, and multiple types within six of the subthemes. In Figure 3, observation #27 of subject P(H), a request for help to acquire a gas connection, is coded as 2 (Shared activity), 2.1 (Distribution), 2.1.3 (LPG connection).

Figure 3. Example of Coded Field Notes (VCP Study)⁴⁸.

The second coding strategy draws on machine learning tools to extract patterns of information from shadowing field notes. Specifically, topic modeling can be used to determine what configurations of words and phrases tend to appear together, and in what frequency, to illuminate trends in activities, interests, and behaviors of the observed individuals. As in the analysis of speeches, news reports, or social media, automated machine learning tools can be used to examine a large corpus of shadowing notes and produce a set of “topics” that summarize and classify substance in the text. ⁴⁹ Example topics from the VCP study are shown in Figure 4.

Figure 4. Representative Topics (VCP Study).

3.5.2 Analyzing Results

Analysis of shadowing data can involve examinations at the level of the observation (a moment in time), the shadowee, and the topic. The structure of shadowing field notes, which include information on dates, times, and locations of observations, also lend themselves to the use of structural topic models. ⁵¹ This technique allows for inclusion of metadata on each timed observation. In this way, each observation within a set of field notes can be analyzed on its own, while retaining information such as the identity of the shadowee and the time of the observation.

The VCP study offers examples of how a single type of activity—such as a president’s role in providing assistance to citizens in accessing subsidized foodstuffs (rations)—can be examined in multiple ways. At the observation level, one can generate descriptive statistics on any codes included in a coding scheme. For this example, 5% of observation time concerns official documentation, 60% of which relates to ration cards. ⁵² One can evaluate similar patterns at the subject level. Based on the qualitative coding, presidents in the VCP study spent an average of 35 minutes on activities related to ration cards during shadowing, relative to an average of 1,093 total observation minutes per president. At the topic level, evaluation of the structural topic model shows that time spent on ration-card-related requests (topic 17) differs only minimally across subjects. Together, these findings suggest that providing ration cards requires only a small amount of time but is a common activity among council presidents.

Additional analytic techniques for topic models can provide deeper insights into the relationships between activities—such as by analyzing the proportion of a text corpus made up by each of the topics. ⁵³ Word clouds, as shown in Figure 5, highlight the relative prevalence of different words within a topic. In this example from the VCP study, the word cloud brings out the potential importance of ward members—elected village council members other than the president—in the process of developing beneficiary lists for government welfare programs.

Figure 5. Word cloud for topic 19 “Beneficiary Lists” in VCP Study.

Overall, qualitative coding strategies can give the researcher somewhat more specific insights into a subject’s behaviors, through reading of the detailed field notes. Topic models, in contrast, offer a “hands off” strategy for analyzing overall patterns. Thus, topic models can help to validate themes uncovered in qualitative coding and provide a check on potential researcher biases that can shape coding outcomes, while a researcher-driven qualitative strategy can importantly help to “ground truth” insights drawn from a topic modeling approach.

An associated aspect of analysis is writing up initial observations from coding field notes and/or the analysis of a topic model in the form of coding memos. Coding memos can be used to develop “as many ideas, issues, and leads as possible,” ⁵⁴ and “to try to identify and explore a general pattern or theme that cuts across a number of disparate incidents or events.” ⁵⁵ These narratives are not intended for publication, but serve as an intermediate step for the researcher to explore topics emerging from field observations, the ways in which they relate to each other, and implications for the study’s questions of interest. One goal for these memos is to establish general themes from a review of the field materials. ⁵⁶ Another goal can be to document “day in the life” narratives for potential inclusion in narratives for publication. ⁵⁷

3.6 Drawing Inferences and Reporting Results

The final stage of shadowing is to use all of the coded and analyzed materials to draw inferences and write-up observations to convey findings in a publishable form. The analytic questions for the data are informed by the substantive goals of the overall project. The purpose of this process is to determine the key findings of the study, by examining observations from the empirical analyses and their consistency. For the VCP study, I ask, inter alia: how does the allocation of time across autonomous and shared activities compare to our theoretical expectations? ⁵⁸ Are there unanticipated patterns of behavior that require further examination?

For work using multiple analytic techniques, such as qualitative coding combined with a topic model, it is necessary to adjudicate between the findings from each. One should ask, at a minimum, did the coding scheme sufficiently account for all of the types of activities and relevant themes that emerged from the topic model? Did the coding memos sufficiently account for observations that showed up as frequent in the quantitative analyses? In the VCP study, the combination of coding methods allows me to identify via the topic model the importance of certain types of activities—in particular the predominance of tasks over which presidents have “autonomous” versus “shared” authority—and then to verify these patterns and examine specific characteristics via the qualitative coding. Thus, comparing results across analytic techniques can allow the researcher to revisit the primary findings and reduce bias prior to writing up results.

3.6.1 Writing Up Observations

Researchers will typically want to have a narrative form of shadowing results to convey the character of observations, if not fully summarize the themes of the study, for the purposes of publication. Multiple narrative forms may be used in a single write-up of a study, to achieve different goals. There are three main forms this can take: single case, multicase, and thematic.

The first narrative style is a detailed, single case model. This format allows researchers to draw rich, textured insights into the life of a subject by drawing on field notes to elaborate on specific events. The discussion is likely to focus on the subject’s primary activities and may consider how these activities fit into major themes drawn from the broader coding exercise. Often case analysis respects a temporal logic, “following” the subject through their day(s). ⁵⁹ This model can be particularly useful for introducing readers to the context of a given study.

The multicase model focuses on the “typical” experience of the shadowing subjects, creating a “day-in-the-life” (or other time period) drawing examples from many individuals, rather than only one. Here, the focus is on activities, rather than individuals. This narrative can have a temporal logic while also considering themes, allowing the researcher to showcase many examples. ⁶⁰ This model is often most appropriate for showcasing overall findings from a study.

The thematic model abstracts away from attention to the subjects themselves and shifts primarily to a discussion of topics that emerged from the shadowing observations. Here, the narrative concerns an elaboration of the main themes, which may be demonstrated with examples from specific shadowing observations. As with the multicase model, thematic narratives are often most appropriate for summarizing the findings of a study.

In addition, any style of narrative can incorporate discussion of quantitative analyses. In the VCP study, this is done by buttressing the thematic narrative discussion with presentation of descriptive statistics on the frequency of described activities among the full sample. ⁶¹ Similar models can be used to present a variety of quantitative data within narrative accounts.

Overall, shadowing offers tools for collecting, analyzing, and reporting data that can elucidate unique insights into the behavior of political elites. In the VCP study, this includes new evidence that council presidents dedicate a disproportionate amount of time to activities for which they can individually claim credit, rather than those for which they share responsibility with other council members. ⁶² This finding has important implications for understanding the nature of local distributive politics that have not previously been raised by ethnographic or survey-based research on these same elites. ⁶³ By providing a new perspective on choices over actions and activities, shadowing thus generates new opportunities for pushing forward research.

4 Risks to Successful Shadowing Studies and Techniques for Mitigation

While shadowing offers a substantial range of benefits to researchers interested in elite political behavior, there are challenges to the successful execution of a shadowing study. I considered many practical issues in Section 3. In this section, I focus on inferential risks associated with bias in the sample of subjects or in the observations recorded in the field.

Selection Bias in the Subject Sample

In Section 3, I described strategies for developing a rigorous sampling plan. Implementing this plan in practice will, as with surveys and interviews, involve negotiating the risks of nonresponse, or selection bias in the subject sample. There are two primary causes of nonresponse in shadowing: accessing subjects and gaining their participation.

Accessing subjects involves both acquiring their contact information and being able to contact them to make the request for a shadowing engagement. Acquiring the official contact information for politicians’ offices is typically straightforward, but may require visiting the legislature. Accessing direct or personal contact information often involves more steps, and can be quite time-consuming if visiting individual constituency offices becomes necessary. In recent years, this has typically become more feasible, with the increasing prevalence of online sources for contact information. In some cases, such as in India, politicians’ mobile phone numbers are a matter of public record, accessible via government and privately operated websites.

Once contact is made, it is then necessary to secure the participation of proposed subjects. This is perhaps the most substantial challenge to most shadowing studies. Even for public officials, who are typically accustomed to being observed, it is a considerable additional step to allow a shadower to follow one around for an extended period. Thus, researchers should prepare accordingly, particularly with regard to the information that is shared with potential subjects.

As in most studies involving human subjects, potential participants should be introduced to the study via a process of informed consent. With shadowing, this should include what will be expected of the subject, such as time spent with the shadower, general activities to be covered, and the expected time frame. The subject should also be given reasonable information about the purpose of the study, to the extent that this does not compromise the study’s objectives. Where feasible, the subject should also be assured that this is a research study and is not affiliated in any way with organizations or individuals that might hope to harm the subject.

Managing sample bias also involves documenting the results of this process, whether or not one is utilizing a random sample. A record should be kept of all sampled individuals and whether they agreed to be shadowed. This record can then be used to compare those individuals who were shadowed with those who did not agree, on the basis of available observational data, akin to balance tests used in experimental research. The results of these tests can then be used to evaluate the degree to which findings can viably be generalized to a broader population.

Bias in Observations

A second concern for shadowing studies involves bias in observations, with regard to whether what is documented during a shadowing engagement reflects what is typical outside the study. Observations may be biased either because the moment at which a study takes place is unusual relative to a subject’s normal routine, or because a subject behaves differently than she would when not being observed. Both of these risks are a concern in most research that involves an investigator collecting data from subjects. But because of the in-depth nature of shadowing, it is worth considering how these studies can account for and mitigate bias in observations.

Bias in the character of observations due to the timing or context of a study can occur when a subject is observed during atypical moments in her life. As noted above, shadowing studies should always be designed to observe subjects during the periods that are relevant to the research question(s). In addition, the multiday character of observation in shadowing studies can in itself help to moderate the threat of this type of bias. Where observations occur over an extended period of time, there is a reduced risk that any given moment is atypical, and thus also that all of the moments as a whole are atypical. Even if part of an observation period is unusual for the subject, the sum of the events that are observed should be more likely to give an unbiased overview of the subject’s typical activities than a shorter period of observation. In the VCP study, I test this by looking at changes in the prevalence of topics over the course of the day and across days of the week. Within- and across-subject variation in both of these variables allows me to test which topics are associated with temporal variations. ⁶⁴

Selection bias can also be introduced if shadowees allow only certain activities to be observed. With shadowing, the shadower can often document when this occurs and for what types of activities. I code in the VCP study the circumstances of all events to which the shadower was not invited, most of which were meetings with local bureaucrats. I then use the time-based observations to show that these events accounted for only two percent of overall observation time. ⁶⁵ So, while these types of problems of bias in observations are common to many forms of data collection, shadowing offers multiple tools for evaluating as well as mitigating such risks.

The second form of potential bias in observations comes from observer (or Hawthorne) effects. These effects occur when a subject knows that they are being observed, and so behaves differently than she would when not observed, or when she does not know about the observation. This is a risk and likely occurs in all forms of research that involve a researcher transparently collecting data from or on a subject, including surveys, interviews, and many experiments. Because direct observation is core to shadowing, this challenge must be addressed in considering shadowing’s value as a research tool, both in terms of the risks to valid inferences from the data and in terms of how shadowing-specific techniques can help to mitigate and evaluate these risks.

Four core aspects of shadowing help to mitigate or assess observer bias. First, as with ethnography, while subjects may initially react to the presence of an observer, they can become accustomed to the shadower over time and revert to typical behavior. This may reduce the ongoing risk of observer bias. I discuss below strategies to assess whether this is the case.

Second, the use of a shadower, who is directly documenting the subject’s activities, limits the ability of the subject to control what information is collected, relative to other forms of data collection. For example, in a survey, interview, or diary study, it is the subject who dictates what is said and, thus, the primary information that is collected. With shadowing, the shadower, as guided by the study design, is the person making decisions about what information is documented in the field notes. This can include not only what the subject says and does, but with whom they interact and the character of the environment. These types of details may help the researcher to build a less biased overall impression of the subject’s activities.

Third, placing a shadower in the subject’s environment can give the subject less control than we might initially expect. While a shadowee might plan certain activities for the shadowing time frame—such as meetings with specific individuals or visits to particular areas in their constituency—they often have substantially less control than they would in an alternative setting, such as an interview, over dynamics such as who comes to see them and what is happening in their constituency. For example, in multiple shadowing engagements for the VCP study, constituents suffered injuries or accidental death and the subjects needed to respond while being observed. Also, as discussed below, subjects varied their attention to “showcase” events versus personal activities on the first versus the second shadowing day, respectively. What did not vary were topics related to requests from individual citizens for assistance, phone calls, time in the village, administrative work, and official deadlines. ⁶⁶ These latter activities are those over which we would expect a politician to have relatively less control, and so to be less sensitive to observer effects, and they make up a substantial portion of these politicians’ activities. Thus, subjects are frequently forced to deal with events that they did not plan, or that are core to their jobs, allowing the researcher to evaluate their behavior under these conditions.

Finally, the extended time frame, detailed data collection, and multiple subjects typical to a shadowing study generate opportunities to assess indirectly the possible presence of observer effects, in ways that are difficult with other forms of observational data collection. ⁶⁷ Using a structural topic model, a researcher can evaluate the correlation between observed activities and the stage of the shadowing study, across a sample of subjects, and make this transparent in the discussion of results. While this does not allow the researcher to evaluate the presence of observer effects relative to a counterfactual environment without a shadower present, it does allow her to assess whether shadowees on average alter their behavior with regard to some activities over the course of a shadowing study. If there is no change in the presence of topics, this may indicate either no observer effects or that observer effects are consistent across the time period of shadowing. If, however, there are patterns of change across the sample of shadowees, this may instead indicate changes in the nature of observer effects. For example, in the VCP study, I find that half of the topics do not change in frequency during shadowing. Yet, there are changes in the other topics, with politicians engaging in more public events on the first day, and more personal activities on the second day. ⁶⁸ This suggests that shadowees may become more comfortable with nonprofessional activities later in a shadowing study. Including such observations in a study’s findings offers transparency into potential observer bias and how it may change over time. Thus, the character of shadowing not only potentially reduces opportunities for subjects to shape the content of observations, it also offers a unique possibility for an indirect test to assess, and make transparent, the characteristics of possible observer bias in the data. ⁶⁹

5 Conclusion

In this article, I have provided a set of guidelines for designing and implementing shadowing studies of political elites. In doing so, I have highlighted a number of ways in which shadowing can provide us with rich insights into the behavior of these individuals. In conclusion, I consider how shadowing can serve as a form of, and contributor to, multimethod research. Taking advantage of the complementarities between shadowing and various tools of data collection and analysis can make shadowing an even more powerful methodological resource.

As described herein, shadowing itself is often multimethod. Researchers engage in direct observations of politicians, collecting qualitative field data. Then these data are coded and may be evaluated using multiple methodological tools, including statistical techniques, topic modeling, and qualitative memos to summarize the data and contribute to inferences. Then the complete shadowing study process incorporates multiple analytic techniques.

Shadowing can also contribute to studies that use other methods at different stages. When starting a research project, shadowing can offer insights into day-to-day relations and the environment, which can be vital to the design and implementation of other data collection, such as surveys and field experiments. Later, shadowing can be used to examine the detailed ways in which patterns of behavior documented in survey research or hinted at in interviews play out in the lives of elites, or to test the theoretical mechanisms underlying an experimental intervention.

For example, in a study of political responsiveness, an audit experiment can tell us whether politicians generally respond to cues about a petitioner’s demographic or political characteristics. Yet, it is often less useful for telling us about the character of politician–constituent interactions. Direct observations via shadowing can offer evidence of how citizens approach their officials, which, in addition to politicians’ actual responses, can offer information about constituents’ expectations regarding their representatives’ likely response to requests. ⁷⁰

Shadowing can thus offer answers to specific questions about political behavior while also suggesting fundamental insights to be drawn upon in further research. It is a form of data collection and analysis that, while minimally used to date in political science, has great potential for improving our insights into the behavior of political elites. The tools provided here offer a starting point for scholars to take advantage of these opportunities.