Calibrating Confidence: Civic Education and the Relationship between Objective Political Knowledge and Political Knowledge Confidence

Abstract

The concept of political knowledge is foundational to American politics, but we know little about the extent to which its two dimensions—objective knowledge and knowledge confidence—covary over time as citizens learn about the American political system. We employ a two-wave survey to study whether individuals gain both objective knowledge and knowledge confidence such that they calibrate over time when exposed to civic education coursework. We find students gain both objective knowledge and knowledge confidence over the semester and that, on average, the gap between them shrinks after taking Introduction to American Government. However, we also see evidence that a student’s initial levels of knowledge shape growth in these two concepts and whether they become more closely aligned over the semester. The results shed light on the relationship between what individuals know about politics and what they think they know, and the role of civic education in shaping an active and informed electorate.

Civic education has long been seen as the primary means of boosting political knowledge among citizens and fostering a healthy democracy (e.g., Galston 2001). Knowledgeable citizens tend to have a better understanding of how political decisions are relevant to their lives, yielding more interest and participation in politics (Delli Carpini and Keeter 1993, 1997; Lee and Matsuo 2018). Only recently have scholars begun to identify that political knowledge is multifaceted. Individuals demonstrate knowledge along two dimensions: the ability to remember and recall factual information correctly (i.e., objective knowledge or factual knowledge) and expressing how sure one is in their understanding of factual information (i.e., subjective knowledge, confidence-in-knowledge, or knowledge confidence) (e.g., Lee, Diehl, and Valenzuela 2022; Lee and Matsuo 2018; Leonhard, Karnowski, and Kümpel 2020; Radecki and Jaccard 1995).

Scholars know much less about the extent to which objective political knowledge and political knowledge confidence covary, especially as citizens engage in civic education. The few studies that have examined the relationship show that the two dimensions do not always align (Anson 2018; Rogers and Gooch 2021), and that knowledge confidence is a much stronger predictor of outcomes typically ascribed to objective knowledge, such as increased political participation (Lee, Diehl, and Valenzuela 2022; Lee and Matsuo 2018). More broadly, research shows that what people know (objective knowledge) does not neatly correspond with what they think they know (knowledge confidence); often, low objective knowledge individuals are highly confident in their knowledge (Kruger and Dunning 1999), a phenomenon sometimes referred to as the Dunning-Kruger effect or the “illusion of knowing” (Leonhard, Karnowski, and Kümpel 2020).

A goal of civic education is to produce a more knowledgeable citizenry, but whether gains happen across both dimensions of knowledge, to what degree, and to what effect are open questions. Answering these questions will further illuminate pathways for equipping individuals to be engaged citizens in American democracy. Ideally, all citizens would possess a base level of knowledge about government and have confidence in that knowledge such that they feel equipped to participate in politics. In this sense, their objective knowledge and knowledge confidence should be aligned, or calibrated. By calibrated, we mean there is little to no discrepancy between one’s knowledge confidence and accurately answering objective knowledge questions (Alexander 2013; Hattie 2013). ¹ However, if, as previous research suggests, these two dimensions of political knowledge are not aligned (or are uncalibrated), then knowledgeable citizens may not participate due to a lack of confidence, while others with a false sense of self-assurance may be highly likely to participate. Poor calibration, whether due to overconfidence or underconfidence, is a troublesome outcome for education (Alexander 2013) and for democracy.

We argue that exposure to college-level civics coursework helps students gain both types of knowledge, and that the distance between a students’ knowledge confidence and objective knowledge should shrink over the semester as these dimensions calibrate (i.e., become more closely aligned). To test these expectations, we obtain survey responses from 1,317 students enrolled in 24 sections of Introduction to American Government at the beginning and end of the semester. We assess students’ levels of objective political knowledge and knowledge confidence entering the semester and changes in each over the semester. Importantly, our approach—measuring both concepts at two points in time—allows us to gauge the extent to which objective knowledge and knowledge confidence move together over time, a novel research design in understanding the relationship between the two types of knowledge.

We find that students gained both objective knowledge and knowledge confidence and that, on average, the gap between knowledge confidence and objective knowledge shrinks after taking Introduction to American Government. Importantly, we also see evidence that a student’s initial levels of objective knowledge and knowledge confidence shape growth in these two concepts and whether they become more closely aligned over the semester.

This work makes several important contributions to the bodies of research on political knowledge, civic education, and the Dunning-Kruger effect in political behavior. First, fielding a two-wave survey on a large, diverse student sample reveals that objective knowledge and knowledge confidence are not static but dynamic and can be brought more closely in line with one another through opportunities for civic education. Second, our results speak to political behavior broadly. While previous studies have found that knowledge confidence is a strong predictor of political participation, scholars often lament that those who are most confident tend to be the least informed. Our results suggest that introductory political science courses can help calibrate confidence and knowledge, creating a more confident and informed citizenry.

Third, the analysis improves our understanding of the role of civic education in American democracy. Exposure to college-level civics coursework can help align an individual’s knowledge confidence and objective knowledge. This is important because it implies that civic education is a tool that can strengthen U.S. democracy by encouraging participation among citizens whose lack of confidence may have impeded their engagement in the political process.

Objective Knowledge, Knowledge Confidence, and the Dunning-Kruger Effect

Objective knowledge and knowledge confidence are two dimensions of political knowledge (Lee and Matsuo 2018). Many studies examining political knowledge tend to operationalize it as objective knowledge only, specifically factual recall about the structure of government institutions, the responsibilities of those institutions, and the current set of political actors making decisions in those institutions (Delli Carpini and Keeter 1993). Traditionally, this dimension has been regarded as the best measure of awareness of how politics works (Delli Carpini and Keeter 1993), though it has been criticized recently as reducing a multidimensional, complex concept to a single, simplified dimension (e.g., Cohen and Luttig 2020; Cramer and Toff 2017; Dolan 2011; Kraft and Dolan 2023; Lee and Matsuo 2018; Pérez 2015). Objective knowledge about politics is prized in part because it is thought to increase political participation, political efficacy (i.e., the belief that an individual’s political actions can make a difference), and the ability to make decisions about politics that consistently reflect one’s personal attitudes (e.g., Lee, Diehl, and Valenzuela 2022; Meirick and Wackman 2004; Siegel-Stechler 2019). Indeed, one goal of civic education is to prepare students to meaningfully participate in politics during their lifetime by teaching students about government institutions and actors (Pasek et al. 2008; Weinschenk and Dawes 2022).

Although there is near consensus that objective knowledge is an important trait for understanding individuals’ propensity to participate in politics, it is not without its flaws. First, correctly guessing an answer can inflate a respondent’s objective knowledge score, though it is purporting to measure “retrieval accuracy” (Lee and Matsuo 2018). Indeed, objective knowledge tests may measure the accuracy of factual retrieval and exam taking skills, but not whether the content is actually stored in memory in an actionable manner. Second, the typical operationalization of objective knowledge—a five-question battery focused on institutions and who holds power within them—is not comprehensive of all relevant factual knowledge as it does not cover salient issues or individual rights and liberties (Cohen and Luttig 2020; Weitz-Shapiro and Winters 2023). Third, there are also significant disparities in objective knowledge scores across different demographic groups, such as between men and women, races, ethnicities, socioeconomic class, etc. (Cramer and Toff 2017; Dolan 2011; Kraft and Dolan 2023; Pérez 2015). Yet these apparent knowledge gaps may be driven by the types of questions asked and to which demographics those questions are most relevant in their daily lives (e.g., Kraft and Dolan 2023). Finally, Cramer and Toff (2017) argue that operationalizing political knowledge as holding facts in memory is “misplaced” and political knowledge is better understood by examining how people process and perceive information within the framework of their own experiences (2017, 1).

Understanding knowledge confidence can help capture aspects of the rich and holistic concept of political knowledge that objective knowledge misses. Individuals’ self-assessments of their knowledge, sometimes described as one’s confidence in (Hattie 2013) or perception (Leonhard, Karnowski, and Kümpel 2020) of what one knows or understands (Alexander 2013), may be used alongside objective knowledge to provide a more complete picture of one’s processing of political information. Though described differently in different research contexts, including as one’s ability to access stored information (e.g., Lee and Matsuo 2018), we take the second dimension of political knowledge to mean one’s own judgment of their understanding of politics, which can be thought of as confidence in what one knows (Ortoleva and Snowberg 2015; Hattie 2013) and has been operationalized as such (Lee, Diehl, and Valenzuela 2022; Jansa and Ringsmuth 2022). This is an important, if overlooked (Leonhard, Karnowski, and Kümpel 2020), aspect of political knowledge. In fact, knowledge confidence can often be a stronger indicator of political engagement and participation than measures of factual political knowledge (Lee and Matsuo 2018; Leonhard, Karnowski, and Kümpel 2020; Ortoleva and Snowberg 2015). Lee and Matsuo (2018) find that misinformed citizens (i.e., those demonstrating high knowledge confidence and low objective knowledge) are as politically active as well-informed citizens (i.e., high knowledge confidence and high objective knowledge). This phenomenon is not unique to American politics and citizens. A 2017 survey of German citizens found that 75% of participants considered themselves to be well-informed and confident in their political knowledge, but fewer than half of participants correctly answered factual questions about the nation’s political system (Leonhard, Karnowski, and Kümpel 2020).

Further, people who are overconfident in their political knowledge often make strong political assertions and can be resistant to legitimate counterarguments that conflict with their beliefs (Anson 2018). Indeed, scholars of political misinformation have noted that overconfidence is associated with online news consumption (Leonhard, Karnowski, and Kümpel 2020), as well as being less active in searching for new information about politics and being more likely to rely on cognitive shortcuts (Dancey and Sheagley 2013). Overconfidence has taken on a negative connotation as the “illusion of knowing”—someone thinks they know a lot about politics, but they do not really know much. Put more positively, increases in an individual’s knowledge confidence—regardless of their performance on objective knowledge questions—can increase their likelihood of participating in politics (Lee, Diehl, and Valenzuela 2022). Thus, measuring knowledge confidence is important for assessing the gains one can make when learning about politics (e.g., Jansa and Ringsmuth 2022).

The mismatch between objective knowledge and knowledge confidence is well documented in the literature on the Dunning-Kruger effect. The Dunning-Kruger effect refers to the finding that individuals with low levels of knowledge often have high confidence in their knowledge or skills (Anson 2018; Kruger and Dunning 1999; Motta, Callaghan, and Sylvester 2018; Simons 2013). ² In classroom settings, students with lower performance on objective knowledge batteries are often unaware of their own cognitive limitations—they have more “unknown unknowns” (i.e., knowledge they are not aware they lack because they do not know it exists)—which results in high confidence in their own knowledge and underestimating the knowledge of their peers (Anson 2018). Performance feedback in the form of graded assignments or group discussions moderated by an instructor can help improve objective knowledge and help facilitate more accurate self-assessments by recalibrating knowledge confidence to more accurate levels (Simons 2013). An important part of developing more accurate self-assessment of knowledge confidence is learning not only about one’s own objective performance, but how one’s objective knowledge compares to others (Kruger and Dunning 1999; Schlösser et al. 2013; Simons 2013). This allows people overconfident in their own abilities to learn about their own limitations and make more accurate assessments about the knowledge of the peers to whom they compare themselves.

However, scholars have also shown that in many instances people are aware of their gaps in knowledge, having received negative feedback from testing (i.e., they hold “known unknowns”), yet remain overconfident. In these cases, individuals may report higher levels of confidence because they aspire to know more than they do now (Simons 2013). Further, individuals who are highly knowledgeable tend to slightly underestimate their knowledge, leading to gaps in knowledge confidence and objective knowledge among highly knowledgeable individuals as well. Ideally, civic education would strengthen both dimensions of political knowledge, as needed, and bring objective knowledge and knowledge confidence in line with one another such that individuals have a factual basis they are confident in and ready to deploy in the political process.

Does Civic Education Calibrate Objective Knowledge and Knowledge Confidence?

In this study, we examine how objective knowledge and knowledge confidence change during enrollment in college-level civics coursework, specifically Introduction to American Government. This course is common across college curricula in the United States and is a mandated general education course at many universities. For some students, the course serves as a capstone to their formal civic education before proceeding to major-specific courses. For others, the course could be their first exposure to high-quality information on how politics and government in the United States operates. Taking a college-level civics course like Introduction to American Government has been shown to help students learn about United States politics and government (Pollock and Wilson 2002; Campbell 2008; Botsch and Botsch 2012; McBeth and Robison 2012; Gooch and Rogers 2012; Neundorf, Niemi, and Smets 2016; Bolsen, Evans, and Fleming 2016). College coursework, after all, is a cognitive ³ exercise that asks students to engage with the material in various ways and be tested on their acquisition of knowledge. Therefore, we expect:

Objective Hypothesis Students will have higher levels of objective political knowledge after taking college civics coursework.

However, civic education can also be a metacognitive ⁴ exercise. Exposure to college-level civics courses can lead students to become more familiar with the content, issues, facts, and jargon associated with American politics and government. Coursework challenges students to identify their “unknown unknowns” by exposing them to what political scientists know about American politics. As students identify “unknown unknowns” and move these topics toward “known knowns,” students may become more confident in what they really know about politics and be able to better identify what they really do not know. Relatedly, exposure to information is likely to be processed through and, be connected with, one’s lived experiences (Cramer and Toff 2017), creating a more richly informed student which may manifest not in factual recall but in confidence in understanding of politics as it relates to them. As a result, we expect:

Confidence Hypothesis Students will have higher levels of knowledge confidence after taking college civics coursework.

A good civic education should also ideally bring the dimensions of knowledge closer in line with one another. The process of gaining objective knowledge should provide clarity regarding what students know and how it connects to them, leading to more accurate ratings of their ability to access what they know. Objective knowledge and knowledge confidence, then, should calibrate over the semester such that students’ level of confidence is more in line with their level of retrievable knowledge. Therefore,

Gap Hypothesis The gap between students’ knowledge confidence and objective political knowledge should decrease after taking college-level civics.

We recognize that students can start at different levels of knowledge on either dimension. There may be wide room for growth in objective knowledge for some (i.e., low objective knowledge students) and not as much room for growth in others (i.e., high objective knowledge students). Some students may enter the course overconfident (i.e., knowledge confidence outpaces objective knowledge) while others begin underconfident (i.e., objective knowledge outpaces knowledge confidence). As a result, we expect movement in the gap is likely conditioned by a student’s starting level of knowledge. For low objective knowledge students, who are more likely to be overconfident, we anticipate that decreases in the gap would be driven primarily by increases in objective knowledge. These students are entering their civic education experience without a wide base of knowledge and have much room for growth. For high knowledge students, who are more likely to be underconfident, a change in the gap is likely most attributable to increases in knowledge confidence since there is relatively less room to grow their objective knowledge.

Survey of Students Enrolled in Introduction to American Government

To assess our hypotheses, we fielded a two-wave survey of students enrolled in Introduction to American Government at a large midwestern university. Introduction to American Government is a required general education course for all bachelor’s degree students enrolled at the university. It is also a required course in the political science major and a primary course offering for concurrent and distance learners. As such, a large and diverse group of students takes Introduction to American Government every semester. The two-wave design provides leverage to examine changes in both dimensions of political knowledge over time; other recent examinations of the relationship between objective knowledge and knowledge confidence in political behavior have used cross-sectional analyses (e.g., Anson 2018; Leonhard, Karnowski, and Kümpel 2020; Motta, Callaghan, and Sylvester 2018; Ortoleva and Snowberg 2015), citing the lack of temporal variation as a “notable limitation” (Motta, Callaghan, and Sylvester 2018, 280). We also survey students from two different semesters—Fall 2022 and Spring 2023—yielding a large pool of respondents enrolled in a variety of sections led by different instructors.

In Fall 2022 and Spring 2023, 3,286 students enrolled across twenty-four sections of Introduction to American Government taught by eight different instructors. These students were contacted via an e-mail invitation to complete a Qualtrics-based survey online. Students were informed that they would earn extra credit in the course if they completed both waves of the survey, one near the beginning and one near the end of the semester. Each wave of the survey was open for one week. A total of 2,213 students completed at least one wave of the survey (67% of those contacted) with 1,317 students participating in both waves of the survey (40% of those contacted). ⁵ Of those who completed both waves, 60% identified as female, 32% as racial/ethnic minorities, 31% liberal arts and sciences majors, and 71% freshman. These numbers parallel the demographics of the university—52% women, 35% racial/ethnic minorities, 26% liberal arts and sciences majors—except for freshman, which were only 22% of the student body but made up most students enrolled in Introduction to American Government. ⁶ In the analysis that follows, we limit the data to only students who completed both waves of the survey.

Measurement

The survey included items to measure students’ objective political knowledge and knowledge confidence as well as their demographic background. To measure objective political knowledge, we use a five-item scale with questions capturing knowledge of how government works and people’s social rights and liberties. Table 1 identifies the wording of each question and its correct answer. Each student-respondent is scored as being “correct” (1) or “incorrect” (0) and their score is summed across the five-items. ⁷ We ask the objective knowledge questions in both wave 1 and wave 2. ⁸

Table 1 Objective knowledge and knowledge confidence questions

Like others (e.g., Anson 2018; Lee, Diehl, and Valenzuela 2022; Weitz-Shapiro and Winters 2023; Kraft and Dolan 2023; Perez 2015), we use Delli Carpini and Keeter (1993) as a starting point for selecting and wording questions but make important adjustments. Delli Carpini and Keeter (1993) operationalize political knowledge as a five-item scale meant to capture individuals’ cognitions of facts about politics, defining political knowledge in later work as “factual information about politics and government that individuals retain in their memory” (Keeter 2008, 587). ⁹ Delli Carpini and Keeter’s (1993) five-item scale is meant to tap two core characteristics of political knowledge: 1) knowledge about how government works and 2) awareness of who holds power contemporaneously. Recent research has also identified knowledge of social rights and liberties as an additional core characteristic of objective political knowledge, such as being able to identify the protection of certain rights in state or national constitutions (Weitz-Shapiro and Winters 2023). Importantly, questions about rights and liberties do not exhibit the same levels of gender, race, age, and socioeconomic bias that traditional measures of political knowledge do; tested across several contexts, Weitz-Shapiro and Winters (2023) demonstrate that apparent gaps in objective knowledge shrink when asking about rights and liberties.

We opt to include questions on government institutions, elections, and constitutional rights and liberties, but exclude questions on who is currently in power for a few reasons. First, we are measuring objective knowledge across students enrolled in different sections of Introduction to American Government, which are taught by different instructors who have different tendencies to cover the dynamics of contemporary politics (i.e., who is in charge at any given point in time) but who tend to cover the basics of how the American government functions and the rights and privileges protected by the law and the Constitution. Second, political power could be in flux over time, such as before and after a November general election, ¹⁰ which may affect the difficulty of answering objective knowledge questions from wave 1 to wave 2. Third, including questions on rights, elections, and institutions should provide a more comprehensive measure that is less likely to be biased toward privileged groups (e.g., Weitz-Shapiro and Winters 2023).

We also measure students’ knowledge confidence using a five-item scale following Jansa and Ringsmuth (2022). These five items ask students to assess their confidence in their understanding of five topic areas: political parties, elections, constitutional rights and liberties, checks and balances in lawmaking, and separation of powers. These five topic areas correspond directly to the topics upon which students’ objective knowledge were assessed, as noted in table 1. ¹¹ Students rated their confidence on a five-point scale from “not confident at all” to “extremely confident.” ¹²

These two measures allow us to assess the confidence gap, or the degree to which a student’s level of knowledge confidence is uncalibrated from their level of objective knowledge. To facilitate this comparison, we scale knowledge confidence to run from 0 to 5, matching the scale for objective knowledge. Based on the literature, we then calculate the confidence gap in two ways. First, the directional confidence gap is a student’s level of knowledge confidence minus their objective knowledge. This measure allows us to detect the size and directionality of discrepancy between the two types of knowledge (e.g., Leonhard, Karnowski, and Kümpel 2020; Nederhand, Tabbers, and Rikers 2019; Rawson and Dunlosky 2007). ¹³ It also allows us to track whether respondents shift from being overconfident to underconfident or vice versa from the beginning to the end of the semester. Second, we take the absolute value of the directional confidence gap to create the absolute confidence gap, capturing the distance between one’s confidence and objective knowledge, or the degree to which these differ from each other in either direction (e.g., Nederhand, Tabbers, and Rikers 2019).

Directional and absolute confidence gaps closer to zero indicate better calibration (e.g., Nederhand, Tabbers, and Rikers 2019). We calculate both versions of the confidence gap at wave 1 and wave 2, and we expect each measure to be closer to zero at wave 2, indicating calibration over a semester of civics coursework. By taking a two-wave approach, we deploy well-established measures in a novel way, allowing us to compare the size of the gap for each individual student at two points in time. The absolute confidence gap can range from 0 to 5, while the directional confidence gap may span from -5 to 5.

Our approach builds on previous measurement of political knowledge but makes some important changes to allow us to leverage our two-wave design and assess our research question. Specifically, we ask about one’s confidence across five topics rather than a single perception of knowledge question, providing rich data on the knowledge confidence dimension (e.g., Lee and Matsuo 2018) that can be matched and subtracted from objective knowledge items on the same topic (e.g., Nederhand, Tabbers, and Rikers 2019; see also Rawson and Dunlosky 2007) to create confidence gap measures (e.g., Nederhand, Tabbers, and Rikers 2019; see also Leonhard, Karnowski, and Kümpel 2020). This approach is geared toward calculating changes in the two dimensions together rather than using the dimensions as separate variables (Anson 2018; Lee, Diehl, and Valenzuela 2022).

Analysis and Results

Our data (Jansa, Ringsmuth, and Smith 2024) allow us to track changes in individual students’ knowledge levels from the beginning to the end of the semester using paired sample difference-in-means tests. Figure 1 displays the average objective political knowledge and knowledge confidence at wave 1 and wave 2. Across these measures, we observe movement consistent with what we would expect for students enrolled in an introductory American politics course: both knowledge and confidence increased. As the left panel shows, objective knowledge grew by approximately 0.2 questions correct, from 3.7 out of 5 correct at wave 1 to 3.9 out of 5 correct at wave 2, on average. The mean objective knowledge score at both waves is well above the expected score based on guessing alone (1.33). Gains in objective knowledge were likely constrained by the fact that students entered the semester with a high level of knowledge—the modal score was 4 out of 5 questions correct—leaving less room for growth. ¹⁴

Figure 1 Mean objective political knowledge and knowledge confidence

As the right panel of figure 1 shows, knowledge confidence scores increased, on average, from 2.7 out of 5 at wave 1, to 3.3 out of 5 at wave 2. Paired-sample t-tests demonstrate that the changes in both knowledge confidence and objective knowledge are statistically significant within individuals (p<0.01). The movement along both dimensions is consistent with our predictions in the Objective and Confidence Hypotheses (i.e., students gained both types of political knowledge).

Next, we examine students’ directional confidence gap (i.e., knowledge confidence minus objective knowledge) and the absolute confidence gap (i.e., the absolute value of the confidence gap) over the course of the semester in figure 2. In accordance with the Gap Hypothesis, we observe mean confidence gaps closer to zero at wave 2 compared to wave 1 for both formulations of the gap (p<0.01). The left panel shows that the mean absolute gap decreased from approximately 1.4 to 1.2, indicating decreased distance between the two types of knowledge on average. In the righthand panel, the negative directional gap at wave 1 indicates that students’ objective knowledge exceeded knowledge confidence on average. The directional gap at wave 2 is smaller than at wave 1 and closer to zero. While the directional gap indicates that students on average remain slightly underconfident at the end of the semester, the distance between their knowledge confidence and objective knowledge decreased significantly from approximately -1.0 to -0.6. In sum, figures 1 and 2 show that individuals, on average, gained both objective knowledge and knowledge confidence after taking college-level civics and this helped to calibrate their objective knowledge with their knowledge confidence. ¹⁵

Figure 2 Mean absolute and directional confidence gaps

It is important to note that students enter the semester with different levels of pre-existing political knowledge, which may shape their growth in knowledge and confidence, as well as the relationship between the two. Thus, we also explore whether growth in objective knowledge and knowledge confidence varies based on an individual’s initial level of political knowledge at wave 1.

Figure 3 groups students by their initial level of objective political knowledge and displays how objective knowledge and knowledge confidence change over the semester within each group. The top-left panel shows the averages for those who answered two or fewer out of five objective knowledge questions correctly on wave 1. ¹⁶ This group is the only one in the figure that exhibits significantly greater growth in objective knowledge compared to knowledge confidence (p<0.01). More specifically, those who started the semester with low levels of knowledge increased their objective knowledge score by approximately 1.5 points (from 1.7 to 3.2), a substantial gain. These students had the most to learn and indeed grew significantly more than those at each of the higher levels of objective knowledge in wave 1 (p<0.01).

Figure 3 Mean objective knowledge and knowledge confidence

Note: Panels display respondents by their wave 1 objective knowledge

Interestingly, though, low objective knowledge students (top left panel) were also the students who gained the most in knowledge confidence—approximately 0.3 more points than those with high wave 1 objective knowledge (p<0.01). ¹⁷ This is because, although they began the semester overconfident, their confidence levels were still lower than that of the other students. On average, these students’ confidence grew by about 0.8 points, which is equivalent to moving up one level of knowledge confidence (e.g., from “not at all confident” to “slightly confident”) on three different topics, for example.

Turning to the other panels in figure 3, we continue to see a difference between growth in objective knowledge and knowledge confidence, but it takes a different form. Those with moderate to high levels of objective knowledge at wave 1 exhibit a larger increase in knowledge confidence compared to objective knowledge on average. ¹⁸ Notably, those who answered all five objective knowledge questions correctly at wave 1 experience an average decrease in objective knowledge of 0.6 in wave 2. This highlights the possibility that some students correctly guessed one or more answers in wave 1 and were not so lucky in wave 2. Save for this one group, students across the board exhibited gains along both dimensions, and relatively larger gains in the particular dimension that could help facilitate calibration.

We also find that changes in the confidence gap vary depending on whether students entered the course with low or high levels of objective knowledge. Figure 4 illustrates the absolute confidence gap for students across different starting points of objective knowledge, while figure 5 does the same for the directional confidence gap. The top left panel of both figures displays the average gap for those who answered two or fewer questions correctly at the beginning of the semester.

Figure 4 Mean absolute confidence gap

Note: Panels display respondents by their wave 1 objective knowledge

Figure 5 Mean directional confidence gap

Note: Panels display respondents by their wave 1 objective knowledge

This top left panel in figure 4 shows that the average absolute gap increases for students with the lowest objective knowledge at the beginning of the semester. ¹⁹ The increase in the average absolute gap for these students signals that many students in this group did not close the distance between the two dimensions of knowledge over the semester, which is similar to Nederhand, Tabbers, and Rikers’ finding that low knowledge students were the “worst” at calibration (2019, 1073). But this is likely due to their large growth in objective knowledge (see figure 3) leading many students to flip from being overconfident to underconfident. As the top left panel in figure 5 shows, the directional gap between knowledge confidence and objective knowledge for low knowledge students shifts from being quite positive in wave 1 to slightly negative in wave 2. ²⁰ This change from a positive to a negative average directional gap indicates that a substantial number of students who exhibited the Dunning-Kruger effect shed their overconfidence after taking college-level civics. Indeed, almost half of the students who entered the semester with the lowest level of objective knowledge and were overconfident in their knowledge saw their directional gap move from positive to zero or negative, suggesting that civic education helped them calibrate or increased their awareness that there are aspects of politics that they do not fully understand. As a result, these students ended with knowledge levels that more closely resembled their higher knowledge peers at the beginning of the semester. This is an encouraging finding and raises the possibility that additional civic education could help them calibrate as their higher knowledge counterparts did over the semester.

Moving to the top right panel of figure 4, we see that the average absolute confidence gap again increases slightly for students somewhat low in objective knowledge (answered 3 of 5 questions correctly). However, the corresponding panel in figure 5 indicates these students, on average, began somewhat underconfident on average and remained underconfident at the end of the semester. This difference in absolute and directional gap results reflects that students in this group did not universally shed their under- or overconfidence. Instead, a mix of changes occurred.

The bottom two panels of figure 4 show that the average absolute confidence gap decreased over the semester for those with higher levels of wave 1 objective knowledge. The bottom panels in figure 5 indicate that these students remained underconfident throughout the semester, but that their objective knowledge and knowledge confidence became much more closely aligned after taking college civics coursework. Additionally, those with the highest level of wave 1 knowledge experienced the largest change in the absolute confidence gap across all four groups (p<0.05), suggesting that college level civics is most impactful for these students based on the extent to which they experience calibration. This reinforces the broader finding that a student’s initial level of objective knowledge conditions how their objective knowledge and knowledge confidence respond to taking college-level civics coursework.

Our research design allows us to use difference-in-means tests as straightforward but strong evaluations of our hypotheses because we have paired observations within individuals, all of whom were exposed to civic education. To test the robustness of these results across students, we also modeled the absolute confidence gap at wave 2 as a function of their change in objective political knowledge (wave 2-wave 1) and change in knowledge confidence (wave 2-wave 1) using linear regression, and we similarly modeled the directional confidence gap at wave 2. The models account for students’ starting levels of overconfidence and objective knowledge and control for individual student demographics, specifically self-reported frequency of class attendance, their parents’ education level, their class standing in school (e.g., freshman, etc.), their GPA, identifying as female, and identifying as a racial/ethnic minority. ²¹ Additionally, we control for variation across instructors and semesters by including instructor fixed effects ²² and semester fixed effects.

The models reinforce the conclusions drawn from the earlier t-tests even when controlling for student demographic, instructor, and semester variables. As a matter of presentation, we report the full results in Part F of the online appendix but describe them in brief here. The models of the absolute and directional confidence gaps both highlight how 1) a student’s starting level of objective knowledge and 2) whether they are over- or underconfident conditions the results.

When we look at the absolute confidence gap, gains in knowledge confidence are associated with less distance between the two dimensions of knowledge and gains in objective knowledge are associated with larger absolute confidence gaps (table A7 in the online appendix). Marginal effects plots (figure A6) show that gains in knowledge confidence led to smaller absolute confidence gaps (i.e., more calibration) among underconfident students across levels of objective knowledge. We also see lower absolute confidence gaps (i.e., more calibration) associated with gains in objective knowledge if the student is overconfident to begin with (figure A7). These students’ objective knowledge calibrates with their confidence levels. But we see a divergent effect for underconfident students who gain objective knowledge; these students are learning facts, but their newfound knowledge may lead to larger discrepancies between the two dimensions of knowledge since they need increased confidence to calibrate.

When we examine the models of the directional confidence gap (table A6), the results show that changes in knowledge confidence are associated with larger directional confidence gaps and changes in objective knowledge are associated with smaller directional confidence gaps. The marginal effects plots show that gains in knowledge confidence led to larger directional confidence gaps among both over- and underconfident low objective knowledge students, but this effect flattens as one increases in objective knowledge. For students who are high in objective knowledge and who are universally underconfident, gains in knowledge confidence reduce the directional gap between the two dimensions of knowledge. This is also true for the bulk of students who answered 4 out of 5 objective knowledge questions correctly and who were underconfident. Similarly, students’ directional confidence gap wanes with gains in objective knowledge (figure A5). This effect is most pronounced among students who start low (<=2 out of 5 questions correct) in objective knowledge. The bulk of students in this category experience moderate gains in objective knowledge and are predicted to have low directionality in their confidence gap. For students who start high in objective knowledge (5 out of 5 correct), gains in objective knowledge are not observed since they were already at the top of the objective knowledge scale. Instead, the bulk of students in this category experienced no change or moderate loss in objective knowledge, which are associated with negative but small directional confidence gaps (meaning they remain slightly underconfident).

Discussion and Conclusion

We find consistent evidence that students gain knowledge confidence and objective political knowledge after taking Introduction to American Government. Our study also shows that confidence in one’s knowledge about politics can become more aligned with one’s objective knowledge after taking college civics. The findings speak to the power of civic education to hone individuals’ understanding about politics and to equip them to participate in the democratic process.

Importantly, we also find evidence that a student’s starting level of objective knowledge conditions whether and how the gap between objective knowledge and knowledge confidence closes over the semester. For example, students initially low in objective knowledge were disproportionately overconfident, but these students saw relatively higher gains in objective knowledge compared to knowledge confidence over the semester. Additionally, it is low objective knowledge students who gained the most along both dimensions of knowledge. While some of these students’ calibration did not improve, almost half of those who exhibited the Dunning-Kruger effect at the beginning of the semester decreased their overconfidence after taking college-level civics. This meant that these students, after exhibiting significant growth along both dimensions, ended with knowledge levels that more closely resembled their peers who exhibited higher levels of objective knowledge at the beginning of the semester. This raises the possibility that additional civic education could help overconfident students calibrate like their higher knowledge counterparts did over the semester. In all, the findings suggest that Introduction to American Government courses have the potential to develop those with low levels of objective knowledge in ways heretofore not established in the literature on political knowledge.

For students who were initially high in objective knowledge, the gap between objective knowledge and knowledge confidence decreased over the semester. Closure in their confidence gap was powered by gains in knowledge confidence, along with—at least for those who originally answered all five objective knowledge questions correctly—a decrease, on average, in objective knowledge. For this high objective knowledge group, an important takeaway is that taking college civics bolstered their knowledge confidence—an effect that the literature has shown plays a powerful role in predicting future political participation (e.g., Lee, Diehl, and Valenzuela 2022; Lee and Matsuo 2018) and led to greater absolute and directional calibration of the dimensions of knowledge. Therefore, the results suggest that college-level civic education may increase the likelihood that these ideally informed citizens ultimately engage in the democratic process.

Our two-wave design is unique in the study of the dimensions of political knowledge and the Dunning-Kruger effect in political science, and it permits us to assess how these relationships change after exposure to civics coursework. Despite the advantages of our approach, there is room for improvement in the measurement of political knowledge. For instance, there are relatively few observations of students with the lowest levels of objective knowledge at wave 1. Indeed, a substantial plurality of students answered all but one political knowledge question correctly at the beginning of the semester, leaving little room for growth on the political knowledge scale as it is currently constructed. ²³ Since there was no incentive to look up answers, it is likely our college-student sample entered the course more knowledgeable about basic political facts than the average survey respondent. Though students still have a lot to learn about politics, they may have retained sufficient information in their previous civic education to answer questions that the general public sometimes struggles to answer. Increasing the number of objective knowledge questions would allow future work to assess students’ understanding of a wider range of concepts and would also reduce the proportion of respondents answering all or nearly all questions correctly. One could also aim to ask questions that are more difficult than what is typically asked of the general population, providing a fairer test of knowledge among college students. Yet we do not want to make the test so lengthy and difficult that it undermines participation in the survey.

One might also suggest question learning effects as a possible alternative explanation for students’ growth in objective knowledge, such as if students remembered the sorts of questions asked on the survey and, when encountering the information in class, retained the correct answer because it was salient. While this possibility exists for any multi-wave survey or testing procedure, it is unlikely that question learning effects explains these results since there was a minimum of twelve weeks between completing wave 1 and the opening of wave 2 of the survey. This long span between the surveys, along with the presence of other survey questions (e.g., attitudes about the university and the course, as well as demographics), should mitigate the possibility that students keep the objective knowledge questions at the top of their mind. Additionally, it is possible that respondents looked up the correct answers after completing the first wave, then retained this information over the duration of the semester, using it to improve their score on the second wave. Importantly, students were not given any incentive to do this. We did not tell respondents whether they correctly answered the objective knowledge questions during wave 1, and extra credit was awarded solely for completing the surveys rather than tied to their performance in any way.

In all, our statistical tests on a paired sample suggest a significantly low probability that growth from wave 1 to wave 2 is simply due to random chance and, rather, that students’ common exposure to Introduction to American Government is the simplest explanation for this growth. Indeed, existing work demonstrates student learning in Introduction to American Government (e.g., McBeth and Robison 2012), with learning varying by instructor pedagogical approach (e.g., Jansa and Ringsmuth 2022), or modality (e.g., Bolsen, Evans, and Fleming 2016), suggesting it is the course itself which fuels student learning about politics. Nevertheless, future studies could field the survey in a non-political science course where we would expect not to find any changes in objective knowledge from wave 1 to wave 2. This would increase certainty in the validity of our findings, yet executing such a survey would not fully eliminate the possibility that students might look up answers to objective knowledge questions after completing the first wave. Additionally, such an approach would raise new challenges at a comprehensive university where students in the non-political science course may be simultaneously enrolled in Introduction to American Government or other political science courses, or have already taken these courses, making the issue of question learning effects applicable to that survey as well.

Our results reinforce the notion that objective political knowledge and knowledge confidence are two distinct dimensions of political knowledge (e.g., Lee and Matsuo 2018) and that students gain both during college civics. Knowledge confidence is an important dimension of political knowledge and an important student learning outcome. We know from previous research that knowledge confidence is a strong predictor of future political participation (e.g., Ortoleva and Snowberg 2015), and we now know that it can become more aligned with objective knowledge due to civic education. Future work could directly explore the connection between these two aspects of political knowledge and political participation. We would expect to find that— consistent with the literature—knowledge confidence is more predictive of willingness to participate in politics even as the two dimensions calibrate over the semester. This would show that, although civic education is meant to help students learn facts about politics, knowledge confidence is an important but often overlooked outcome of civic education that encourages young people to engage with democratic processes. Of course, the durability of gains in knowledge confidence, objective knowledge, and willingness to participate post-civic education remains an open question for scholars of political behavior and political science education.

Overall, our results are substantively and normatively important. In particular, the findings inform the ongoing dialogue about the health of American democracy. We know from previous research that those who have “the illusion of knowing” are the most likely to act on their knowledge, despite its existing in confidence only. Our results show that students achieved gains in objective knowledge across the board, including those who exhibited the “illusion of knowing.” While a single civics course is not a cure-all for democracy, our findings suggest that taking college civics can improve individuals’ preparedness to engage in politics. For those whose objective knowledge outpaces their confidence, gains in confidence brought on by civic education may push these informed citizens to become more involved in politics in the future (e.g., Lee, Diehl, and Valenzuela 2022). College-level civics training moves us toward a more informed and confident electorate by increasing political knowledge along two dimensions and, for many students, by bringing these two dimensions into closer alignment with one another. In this way, college-level civic education can be a small part of a multipronged solution to bolster democracy.