Climate Change? Designing and Implementing Climate Surveys to Promote Inclusivity in Political Science Departments

Abstract

Questions of racial, gendered, and class-based inequality in political science are more salient than ever. This article contends that climate surveys can be a useful tool for developing effective equity and inclusion strategies within academic departments. We offer advice on political, procedural, and messaging issues to consider when undertaking a departmental climate survey.

Questions of racial and gendered inequality in political science are more salient than ever. According to 2019 data, membership in the discipline’s flagship professional organization, the American Political Science Association (APSA), is substantially whiter and more male than the US population at large (Mealy 2019). A now-substantial body of research suggests that ineffective mentoring, leaky pipelines, high service burdens, and exclusionary social networks, as well as racism, sexism, and sexual harassment, all act as barriers for the incorporation of marginalized groups into the academy (Behl 2020; Garcia and Alfaro 2021). ¹ Moreover, political science, like many other disciplines, faces a severe mental health crisis among graduate students, with women and minoritized students reporting higher levels of isolation than their white, male counterparts (Almasri, Read, and Vandeweerdt 2021). In the wake of the #MeToo movement and the resurgence of Black Lives Matter protests, many departments are grappling with their complicity in maintaining patriarchal systems of white privilege.

If diversity work is to be transformative rather than a mere exercise in public relations, it should aim to improve the lived experience of marginalized groups (Ahmed 2012). Despite wide acknowledgment of the need to embrace intersectional approaches in higher-education diversity work, many of the most common diversity practices remain pitched at a relatively general level. Diversity trainings, for example, seek to educate and change norms but often pursue remedies to problems that have not been clearly diagnosed in a particular context (Carter, Onyeador, and Lewis 2020; Devine and Ash 2022, 420). Recruiting more diverse students and faculty members aims to directly change the composition of universities; however, on its own, the existence of “more diversity” is unlikely to address feelings of isolation and/or tokenization (Niemann 2016).

This article contends that climate surveys can be a useful tool for developing effective equity and inclusion strategies within academic departments. Departments are crucial sites where everyday practices—including syllabi construction, classroom and meeting norms, and markers of what constitutes professional behavior—operate, often invisibly, to promote patterns of exclusion. Climate surveys create common knowledge around issues of equity and marginalization, foster conversations about how to improve the learning and work environment for all department members, and enable bespoke interventions that are more likely to improve policy and practices (Harper and Hurtado 2007).

Climate surveys create common knowledge around issues of equity and marginalization, foster conversations about how to improve the learning and work environment for all department members, and enable bespoke interventions that are more likely to improve policy and practices.

By their nature, climate surveys touch on sensitive topics and expose power structures that marginalize underrepresented groups. Although a literature exists on executing climate surveys in university settings, most studies focus on campus-wide surveys initiated by university leadership (Hurtado et al. 1998; Wood et al. 2017). Departments, however, often face a different set of constraints due to their smaller scale, more limited financial resources, and how they are situated within university power hierarchies. Technical questions of survey design and analysis are naturally important elements of any successful survey, but we argue that the effectiveness of a climate survey—in terms of its likelihood of influencing your department’s policies, processes, practices, and programs—depends as much on your ability to navigate these departmental and university politics as on technical expertise. Despite the increasing prevalence of climate surveys in political science departments, ² resources for best practices are in short supply, especially at the departmental level (Stachl et al. 2019).

This article offers advice on key issues to consider when undertaking a departmental climate survey, learned from our collective experience in designing, fielding, and presenting results from a climate survey in an R1 political science department. ³ We organize our advice chronologically, considering three main themes. In the first section, Building Consensus, we discuss the importance of defining survey goals and the opportunities and pitfalls associated with engaging department and university stakeholders—not all of whom may be enthusiastic supporters. In the second section, Procedural and Ethical Issues, we discuss who should analyze the survey, as well as potential Title IX reporting responsibilities. In the third section, From Survey to Action, we highlight strategies for presenting survey results and conducting inclusive deliberations around your findings, as well as strategies for moving forward.

BUILDING CONSENSUS: IDENTIFYING AND ENGAGING STAKEHOLDERS

When initiating a climate survey, it is important to consider who in your department or university is likely to care about or have a vested interest in your project. These individuals, who may have something to gain or lose from a survey, are your potential stakeholders (Alves, Mainardes, and Raposo 2010). Identifying and engaging stakeholders is an opportunity to obtain buy-in from various groups and also to position yourself strategically (Rankin and Reason 2008, 264, 269). Stakeholders will have different priorities, some of which may be tangential or even antithetical to your own. Although it may be tempting to disregard divergent objectives, your survey will benefit from early and direct engagement with a range of perspectives (Ryder and Mitchell 2013, 43). The need to consider which stakeholders to consult is especially relevant for grassroots-led surveys run by students and/or faculty members without explicit institutional backing (Stachl et al. 2019). However, even surveys implemented with the approval of department administrators may encounter resistance from higher-level actors if they perceive that the survey could pose a threat to your institution’s reputation (Brown and Gortmaker 2009, 416).

…even surveys implemented with the approval of department administrators may encounter resistance from higher-level actors if they perceive that the survey could pose a threat to your institution’s reputation.

Stakeholders will differ in their institutional power and their support of a climate survey, both of which you must consider as you begin to engage with them. Based on our experiences, table 1 presents an inductive typology of the relevant actors that can be used as a guide for navigating relationships. ⁴ We categorize stakeholders along two dimensions: institutional power (strong or weak) and support (supportive or hostile). This yields a typology of four types of stakeholders, each of whom can play a role in facilitating or undermining your capacity to implement a high-quality survey covering topics relevant to your department. Following are the three appropriate kinds of engagement that you might cultivate with each type of stakeholder.

Table 1 Types of Stakeholders

Note: *These stakeholders can be in the hostile or supportive category depending on their perceptions of the project.

First, seek out potential allies, even if they lack significant formal power. These stakeholders are especially important for brainstorming the goals and objectives of the survey. Actors such as your department’s diversity committee, graduate student leaders, and supportive faculty members often are allies who can contribute moral and optical support, and they can be influential if they exercise their voice.

Second, we recommend building relationships with institutional anchors. Anchors are groups or individuals who have institutional power and credibility on campus and who can advocate for you in the event of pushback from hostile stakeholders. At a minimum, we recommend meeting with institutional anchors and providing them with an overview of your survey; however, it may be helpful to pursue closer ties. For example, asking institutional anchors to co-sponsor the survey can mitigate pushback from institutional veto players because they also possess institutional power.

Some stakeholders will be either merely tolerant of or hostile toward your climate survey. Skeptics are those individuals who are passively hostile to the survey but who do not possess clear institutional power. Although skeptics may not be able to block your survey, their engagement can influence how your survey results are received—by complaining that their concerns were not incorporated or by focusing, after the fact, on what the survey did not do. We recommend that you keep these individuals informed throughout the process of developing the survey. For instance, you might ensure that everyone in your department—including skeptics—receives a copy of the survey instrument and is asked for feedback.

The third type of stakeholder is the institutional veto player. This stakeholder is not only hostile to the climate survey but also has the potential to block it or to substantially limit its scope. The true threat of an institutional veto player will be moderated by the power of those championing the climate survey. That is, an institutional veto player is likely to pose greater problems for a grassroots campaign led by students than if the survey was requested by a department chair or dean. We advise active rather than passive engagement when interfacing with institutional veto players: request what you need from them rather than ask for their approval. Be prepared to defend your position in the face of resistance from institutional veto players, and enlist your institutional anchors in a supporting role.

PROCEDURAL AND ETHICAL CONSIDERATIONS

The relevant population of interest for your survey depends on the specific issues facing your department community, the goals of the project, and the resources at your disposal (Ryder and Mitchell 2013, 37–39). In some contexts, you might limit the survey to undergraduate or graduate students; in others, you might be interested in collecting information from your entire department community, including faculty members and staff. Consider how a broader population might expand your sample size, potentially allowing for statistical analyses or providing increased anonymity for minority respondents. Conversely, you might limit the population to focus on specific issues or to minimize the amount of data that must be analyzed (particularly if there are many qualitative elements to the survey) (Ryder and Mitchell 2013, 39). Population size also matters: if your department is small, minority respondents may fear the identifiability of (and retaliation based on) their answers (Moreu, Isenberg, and Brauer 2021). For further discussion of considerations in selecting the target population for your survey and strategies for maintaining anonymity in the context of small populations, see online appendix D.

As important as strategic engagement with different types of stakeholders and building consensus around the fundamental goals of your survey is the need to obtain informed consent and to protect respondents’ confidentiality. Issues of privacy and informed consent are somewhat distinct in the context of a climate survey, however, because the “researcher” is embedded in the object of study. This consideration is discussed in much greater detail in online appendix D, including sample text for informed-consent language and an example email to send to potential survey respondents. The following subsections discuss two associated procedural issues—who should analyze the data and legal reporting obligations vis-à-vis Institutional Review Board (IRB) and Title IX—as well as ethical considerations about surveying on sensitive topics.

Choosing a Data Analyst

In addition to requesting consent from participants and taking measures to protect the confidentiality of their identity, another important decision is determining who will handle and analyze the raw survey data (McMahon, Stepleton, and Cusano 2016, 3). This choice, which should be made before fielding the survey, potentially will impact (1) the response rates and, therefore, the quality of data collected; (2) the degree of confidentiality afforded respondents; and (3) the handling of data on sexual misconduct.

In our view, best practice is to contract the data-analysis work to a third-party analyst external to the university. ⁵ Working with a data analyst who has no ties to your department or university allows you to credibly ensure the confidentiality of respondents and collect more detailed demographic data. This also may provide greater legitimacy for your survey if results are critical of your department’s climate because external parties may be perceived as less likely to have a political agenda (Hart and Fellabaum 2008, 229). If securing a third-party data analyst outside of the university is an unrealistic option, the next best option is to contract with a campus office that is not associated with your department to conduct the data analysis. In theory, using a university-affiliated analyst should have many of the same advantages discussed previously—that is, putting respondents at ease and improving their willingness to respond truthfully to the survey. One potential drawback is that respondents still may be concerned that individual responses could be reported to their department leadership. If it is infeasible to contract a third-party analyst—and, therefore, a member of your department will be handling the data—be prepared that the utility of your survey might decrease. Without an impartial analyst, you risk lower response rates and/or reticent respondents. To combat this, select only one or two trusted individuals to handle the raw data and clarify to respondents who will have access to the data. Moreover, carefully consider whether it is appropriate for faculty, administrators, and students to have access to potentially identifying survey data about their peers. For more discussion about selecting a data analyst, see online appendix C.

IRB and Title IX

Climate surveys in university settings have an ambiguous status vis-à-vis academic research (US Department of Justice 2017). Technically, if you do not plan to publish your climate-survey report, intending it only for the internal consumption of your department and/or university, IRB approval is not required. ⁶ If, however, your survey team decides to include questions on sexual misconduct, you may want to consider IRB approval and how it potentially intersects with Title IX reporting requirements (White House Task Force to Protect Students from Sexual Assault 2016, 3).

All universities have policies, supervised by a Title IX office, that require employees to report any disclosed instances of sexual misconduct, a term that we use to encompass both harassment and assault. Reporting requirements depend on several political and institutional factors: whether your data analyst is a university employee; whether your survey received IRB approval; whether your university’s IRB rules contain reporting exemptions for disclosures of sexual misconduct; and whether your departmental and university leadership are supportive of your survey and committed to following reporting obligations (Potter and Edwards 2015; Prevention Innovations Research Center 2015). Online appendix C discusses IRB approval vis-à-vis Title IX reporting requirements in more detail.

Barriers to Disclosure of Sensitive Information

When queried on sensitive topics, survey respondents may feel compelled to provide socially desirable responses, be reticent to answer questions that they perceive as taboo, and have an incentive to lie if they are concerned about the confidentiality of their responses (Tourangeau and Yan 2007). These trends are especially relevant in surveys that ask questions on topics such as experiences of discrimination and sexual misconduct. These concerns can be mitigated by confidentiality assurances (Ong and Weiss 2000); self-administration of the survey (Tourangeau and Yan 2007); and eliciting respondents’ motivation to answer the questions, such as by drawing attention to the socially oriented goals of the survey (Tourangeau, Smith, and Rasinski 1997).

FROM SURVEY TO ACTION

As Livingston (2021) observed, effective organizational change involves three steps: condition (i.e., an understanding of the problem and its root causes), concern (i.e., a willingness to change), and correction (i.e., a concrete action plan for taking steps). Results from climate surveys can identify the condition of a department, as experienced by different groups. However, to generate a shared commitment to change and a plan for moving forward, the next step in the climate-survey process is to share the results with your broader department community (Hart and Fellabaum 2008). To maximize the likelihood of your survey generating useful cultural and institutional reforms, consider how to engage in productive deliberation about your findings, address pushback, and generate an action plan. The next subsection discusses these strategic concerns; strategies for structuring the climate-survey report are in online appendix E.

To maximize the likelihood of your survey generating useful cultural and institutional reforms, consider how to engage in productive deliberation about your findings, address pushback, and generate an action plan.

How to Deliberate?

Although the purpose of a climate survey is to document the range of experiences in your department, it also should open a conversation about how to improve (Livingston 2021). Ideally, within a few weeks of the report’s release, you should convene a meeting of stakeholders. This might include your entire department or separate sessions for key stakeholders, such as faculty, graduate students, and undergraduates—or even administrators.

When your department considers how to deliberate, we encourage you to take an inclusive approach, receiving feedback from all of its members. Because research suggests that the voices of women and minorities often are marginalized—even in small-group deliberative settings—we recommend that careful thought is devoted to creating spaces of supportive communication for these groups (Mendelberg, Karpowitz, and Oliphant 2014). An argument made by a member of a minoritized group is likely to be less influential than the same argument made by those in the majority (Myers 2017). Therefore, consider preparing an ally from your department’s majority group to validate the minority’s points. An online forum for students and staff to submit responses anonymously is another strategy for encouraging inclusive deliberation.

If you anticipate resistance to your findings, consider how common ground might be established with your audience (Hoefer 2019). Is there a shared goal or value that can be used to build bridges? For example, you might emphasize the costs of inaction, such as cementing a reputation of being hostile to minority groups, which then can have negative feedback effects on recruitment. Alternatively, it may be productive to frame discussions positively around the idea that the survey presents an opportunity for your department to forge a new reputation.

Prepare for Pushback

Some stakeholders might be dissatisfied with the findings from your climate survey and may advance a narrative that downplays negative results. Behavioral research on motivated reasoning, for example, reveals that people tend to dismiss new information that conflicts with their priors (Kunda 1990). Motivated cognition can be particularly likely if new evidence is contrary to the standing of an individual’s membership group or that individual’s standing in the group (Liu and Ditto 2013).

Do not hesitate to push back on these narratives. For example, skeptics might ask whether climate problems are widespread or due to the behavior of “a few bad apples.” In this case, it is useful to remind colleagues of research that suggests that the behavior of even one negative team member can have a powerful, negative effect on an entire work environment, leading to a downward spiral in an organization (Felps, Mitchell, and Byington 2006).

Output

Some stakeholders may believe that simply by generating a climate survey, they have fulfilled their diversity obligations. In isolation, however, a climate survey is insufficient to address problems of marginalization. In our view, the ideal long-run response—at least for a first-time climate survey—is the creation of a publicly circulated document akin to a strategic plan or an external review response (Ryder and Mitchell 2013, 42). In this document, which would be shared with your department community and institutional anchors, the department would define its understanding of the problems identified in the survey, the deliberative strategies that it has undertaken, and its plans for moving forward. These plans should include a set of benchmarks and anticipated deliverables, as well as a clear timeline for achieving goals, to hold your department accountable for making concrete changes. Examples of action items from our experience are provided in online appendix E.

CONCLUSION

If diversity work is to be more than a legitimizing performance indicator, it should be responsive to the lived experiences of marginalized groups and the particularities of specific departments. We contend that implementing a climate survey may be a necessary first step for a department that wants to implement anti-racist and anti-sexist practices. That said, a climate survey alone is unlikely to be a sufficient solution to departmental inequalities—rather, it is merely the beginning of this process.

Although climate surveys can provide a useful window into understanding the experiences of individuals outside of dominant social networks, they are not a panacea. If your department is overwhelmingly white, for example, a climate survey is unlikely to provide deep insight into the experiences of minoritized groups. The same might be true in a very small department, where it often is difficult to survey demographic subgroups of interest without compromising their identity.

Similarly, if the majority of work on the climate survey is shouldered by underrepresented groups within your department, this reinforces the same inequalities that the survey ostensibly seeks to address. We therefore encourage those who are considering undertaking a climate survey to ask their departmental leadership for compensation and/or resources. The burden—and costs—of solving diversity and climate issues should not fall on members of underrepresented groups. In the long run, it also may be productive to encourage your university to build capacity for departments to regularly implement climate surveys rather than have each department “reinvent the wheel.” Specific recommendations for avenues of institutional support are listed in online appendix F. This strategy can promote the institutionalization of climate surveys—and the benefits that they provide—without placing undue burden on minoritized groups.