Audiences and Outcomes in Online and Traditional American Government Classes Revisited

Abstract

In 1997 we first offered American government classes online as well as face-to-face classes. We administered pre- and posttests to our students to measure their general knowledge of American government, political attitudes, demographics, and some behaviors. Following an initial report in 2001, we continued to gather data for 10 more years; this current study covers nearly 3,200 students during 13 years. We examine the sample as a whole and changes in audiences and outcomes, over time, for the two teaching formats. Although the kinds of students taking online classes have become more similar, a few differences persist. Learning outcome differences continue to be insignificant. Neither format has a clear advantage in students' changes in attitudes, but the online classes increased students' newspaper reading. Class dropout rate and faculty workload both favor face-to-face classes, but flexibility in scheduling and student demand clearly favor online classes.

When future historians study the impact of technology in the late twentieth and early twenty-first centuries, they will doubtless note the sea change that occurred in how courses were taught with the emergence of the Internet. In the late 1990s, we began to explore alternative formats for delivery of our program's “bread and butter” class, American National Government. Our own motivation was more basic than using a new technology. At that time, our university, a regional campus of the state university with about 3,000 full- and part-time students, was an entirely commuter campus in a small town, and drew its students mostly from the surrounding counties. First-generation college students shared parental disdain for government and politics. They saw college mainly as a stepping stone to a good job. Because an American government course was not required for graduation for most students, we struggled to attract students. Thus, our web-delivered courses were born out of desperation as much as technological possibility.

After one of us received release time to develop an online class, we conducted a pilot project in the fall of 1997 with four students. The following spring (1998), we offered the University of South Carolina Aiken's first completely web-based class. We have offered one or more American government online classes every semester since then and developed several web-based upper-level political science courses. Of course, online courses, and even entirely “virtual” universities, are now common around the world.

Distance education has often been viewed with suspicion. Regardless of the method of course delivery, “… classroom instruction has been the standard to match.”¹ Although many of our colleagues initially thought web-based teaching would lack the academic rigor of the face-to-face format, our early research helped put those fears to rest. We began to administer pre- and posttests to our students to determine their general knowledge and political attitudes when they began and completed the class. We also gathered demographic data on the students. After several years, we, like many others who conducted research in the field, concluded that “no significant difference” in the factual knowledge was gained between web-based and face-to-face classes (Anstine and Skidmore 2005; Botsch and Botsch 2000; Botsch and Botsch 2001; Dolan 2008). Web-based courses in a variety of disciplines are no longer unusual at USC Aiken and, during registration, are frequently among the first classes to close. After concluding this 13-year quasi-experimental study² in the spring of 2010 with a sample increased tenfold from the original study, we reexamine our original conclusions and changes that have occurred.

THE TWO TEACHING FORMATS

As long-time colleagues who frequently consult each other, our classes, whether face-to-face or online, have a great deal in common, in large part because we planned the web-based class to be very similar to the face-to-face class format. We used the same standard text until 2009, when we switched to a free, online text that one of us authored.³ All students answer “reading mastery” questions based on the material in the text. To reinforce learning, students in both classes are required to access a web page and/or read a newspaper story on the topic of the week and write a short essay relating the material to concepts in the text. Both classes have opportunities to attend campus events and write reports for extra credit. Both classes have discussions, but discussions in the web classes are totally asynchronous and virtual with no real-time interactions among students and faculty. All interactions are via the Internet, although students contact their teachers individually in their offices, on the phone, or by e-mail. Students must respond within a specified time to a specific discussion question.⁴

Other than the obvious difference that in face-to-face classes discussions are verbal, a significant difference is that meaningful student participation in discussion in the web-based classes is part of the course grade although not generally graded in the face-to-face classes. To compensate for the greater emphasis on formal discussions in the online classes, in face-to-face classes we attempt to stimulate student interest in news reading with occasional extra credit news quizzes and engage the students through in-class exercises and activities to reinforce their readings. Testing procedures are different. Face-to-face classes use mixed-format and closed-book tests: about 70% to 80% are objective questions and one or two are essay questions. All testing for web-based students is online, open-book essays, which is consistent with keeping the schedule highly flexible. Each test, for either class format, generally covers three or four chapters . The maximum enrollment is 20 students for the web-based classes and 25 for the face-to-face classes (plus a few overrides in compelling circumstances). The maximum was 15 in web-based classes a few years ago, but to increase productivity in tight budgetary times, the enrollment was increased.

Despite these differences, the basic format for both web-based and face-to-face classes was well established during our many years of teaching American government before taking it online: The format included close reading of the text with questions to answer, discussions based on short writing assignments, and current events assignments to illustrate concepts. The course has remained in place during the entire 13-year period of this study in both the web-based and face-to-face classes. The only variations were minor fluctuations in the precise numbers of news and web assignments from semester to semester and professor to professor and use of the general knowledge posttest as part of the final examination in some selected classes for a few years.⁵ In sum, both types of classes have much in common despite different delivery formats. The major differences between the web-based and face-to-face classes are testing procedures and the more formalized and graded discussions in the online classes.⁶

COMPARATIVE MEASUREMENTS

Through the spring of 2010, we compiled data on nearly 3,200 students, 2,525 who took the course in the traditional face-to-face format, and 659 who took it online. Having such a large sample allows us to detect with statistical significance much smaller differences than those we reported more than a decade ago, when we had an N of 321 students: 105 online and 215 face-to-face (Botsch and Botsch 2001). The data include a wide range of information, including GPA, major, age, gender, ethnicity, parent's education, and perceived course difficulty. We also gathered information on knowledge, attitudes, and behaviors that might have changed from the beginning of the course to the end of the course (including general political knowledge about American government, political interest, political efficacy, political trust, and daily general newspaper reading). Increases in knowledge,⁷ increasing interest, increasing efficacy, and more frequent exposure to news were goals for a course that purports to improve the civic culture of citizens.

The act of measurement often affects the qualities that are being measured. Because this study was a long-term field experiment, we cannot be certain that we did not change the way we taught because of the less-than-stellar posttest scores. Because we were both experienced professors who were reasonably confident that we were using tried-and-tested teaching techniques before this experiment began, we made no conscious dramatic changes driven by test scores.

AUDIENCES

As we reported a decade ago (Botsch and Botsch 2001), we found and continue to find both similarities and differences (although shrinking) in the audiences for online and traditional classes. Students who enrolled in online classes during the 13 years of this study differ significantly from those who enrolled in traditional lecture classes in terms of age, GPA, gender, major, initial level of information about government and politics, daily newspaper reading, political efficacy, trust, and ethnicity, as shown in table 1.

Table 1 Comparisons of Traditional and Online American National Government Students, All (Spring 1998–Spring 2010) and Initial (Spring 1998–Fall 1999)

^a E-mail [email protected] for a copy of the 59-item test with 63 possible correct answers.

^b “Would you say that you follow what's going on in government and public affairs (4) most of the time, (3) some of the time, (2) only now and then, or (1) hardly at all?”

^c “How much of the time do you think you can trust the government to do what is right? (1) none of the time, (2) only some of the time, (3) most of the time, or (4) just about always.”

^d “Sometimes politics and government seem so complicated that a person like me can't really understand what's going on. Do you (1) strongly agree, (2) agree, (3) have mixed feelings, (4) disagree, or (5) strongly disagree?”

^e “How many days in the past week did you read a daily newspaper?”

^f “How would you compare the difficulty and workload of this course with others you have taken? (1) Easier (2) About the same (3) Or harder?”

On average, web-based students were 2.6 years older than students who enrolled in the face-to-face classes. The difference is down dramatically from the six-year difference we found a decade ago, as shown in table 1. Tracked over time, the average age of students taking the online classes steadily declined during the 13 years of this study (by 10.4 years). Students in the face-to-face classes also declined in age, yet although statistically significant, the decline was small (1.4 years). Some of this overall decline is explained by increased enrollment of younger traditional students at the university. Although online students are typically older than face-to-face students (Tallent-Runnels et al. 2006), clearly the age mix of students in our web-based classes became much closer to those in face-to-face classes. For some time, the scheduling flexibility and convenience of online classes has attracted older, nontraditional students (Anstine and Skidmore 2005; Garson 1998; Harrington and Loffredo 2010; Lei and Gupta 2010). Now, traditional students also seem to place a higher value on scheduling flexibility. This flexibility is especially important with increasing tuition costs when more of the traditional students must work to pay for school. Like others (Kreb 2009), we found that student-athletes like online classes because these courses easily fit around tight practice and game schedules.

Students taking the online version of the class had a slightly but significantly higher GPA than students in the face-to-face class. Part of this difference is related to age, as age is a significant predictor of GPA (p = 0.000). However, it also may be related to the kinds of students who preregister and enroll in high-demand classes that fill quickly and close. Students with high GPAs are probably better planners and less likely to procrastinate during the registration process.

Women have comprised almost exactly two-thirds of the students on our campus throughout the 13 years of this study (Dawe 2011; Herrin 1999). For the first three years of this study, women were significantly less likely to enroll in the web-based classes than in the face-to-face classes (an average of nearly 10 percentage points lower). Yet during 2000–01 the percentages flipped. Since then, women consistently have been more likely to enroll in online classes (an average of nearly 12 percentage points higher).

Although the gender change in 2000–01 may have been a statistical artifact,⁸ we offer an explanation for the years that followed. This explanation has implications beyond gender. In 2001–02, about one-third of the way through our study, a change in general education requirements across campus had a dramatic effect. All students were required to complete an American Political Institutions requirement that could be satisfied by taking either the American government course or an American history course. Prior to this, only education and political science majors were required to take American government. Relatively few students from other majors took the course. We continued to see relatively few nonsocial science or education majors in the face-to-face classes until this requirement changed. However, the online classes were drawing a different mix prior to the 2001–02 change in general education. Online classes especially drew well among science majors who comprised nearly the same percentage as social science majors in web-based classes (11%), more than double what they were in the face-to-face classes (5%). After the requirement change, the mix of majors taking all American government classes began to better reflect the overall mix of majors across campus, and the number of students in American government classes increased dramatically, jumping by more than 65% the first year. Nursing, a major of primarily female students, contributed significantly to this gender shift. Nursing students increased from 6% to 18% of all online students after the requirement change in general education. We note that web-based classes were drawing relatively more nursing students than face-to-face classes even prior to the general education requirement change (6% and 2% respectively). Nursing students face significant scheduling difficulties with their many required nursing classes, so the convenience of online classes has great appeal.

The gender shift was not entirely explained by changes in the composition of majors taking the web-based classes. We examined the percentage of women in web-based classes over time for each major. Although the percentages did fluctuate because of low Ns for some years in some majors, the overall trend for each major was higher percentages of women in the web-based classes during the 13-year period. This trend suggests two other possible explanatory factors: the erasing of a gender digital divide and the increasing importance of convenience for women. In the 1990s, men were more likely than women to be Internet users, but that is no longer true (“The Internet” 1997; Krantz 2000; Stoughton and Walker 1999).⁹ Women have become relatively more comfortable with taking online classes, so any differences between men and women should have been expected to disappear. Yet women did more than erase the difference. Perhaps the increasing pressures on all college students may be disproportionately felt by women, some of whom are single mothers and all of whom live in a traditional culture in which women are expected to help out more at home.

While the composition by academic major of our American government students now better reflects the overall student body in both the web-based and face-to-face classes, academic major still makes a slight, but significant, difference. Students in scientific fields, business, and nursing were more likely, by a few more percentage points, to take the course online. Those students who were undecided in major, in the humanities/fine arts, or in social sciences, were slightly less likely to take the course online. Education majors matched the overall average. As noted earlier, one of our original motivations for creating the online version of the American government course was to attract students from disciplines far removed from political science. Our data suggest that this benefit still exists, although it is smaller than it was a decade ago, largely because students are now more comfortable using the web. The remaining differences are likely to grow smaller as time passes.

Studies conducted in past years have confirmed the existence of a racial “digital divide” in addition to a well-documented income gap (“Digital Divide” 2000; Gladieux and Swail 1999; “Income Gap” 2007; “Survey Shows” 2000). Income differences are positively associated with Internet access (Fairlie 2003; Martin and Robinson 2007). Pew Center researchers found, however, that by 2010, minorities had become as likely as whites to own laptops, although whites were still more likely to have broadband access at home. Changes in technology and the advent of a range of frequently cheaper and portable devices also have given minorities increased Internet access (Washington 2011).¹⁰ Indeed, in recent years we have seen many of our students, including African Americans, answering Blackboard reading mastery questions with their smartphones.

These trends are reflected in our data. Most of the nonwhites on our campus are African Americans. The percentage has remained relatively constant, increasing slightly from 23% in 1999 (Herrin) to 27% in 2010–11 (Dawe). Over the 13 years of the study, significantly fewer minority students enrolled in the online classes than in the traditional classes (19% and 30% respectively), as shown in table 1.

However, if we track the ethnic composition year by year, the differences have dissipated. The biggest break point was 2006–07, when the percentage of African American students increased about eight percentage points. Since then, the percentage of African Americans has remained relatively stable. From 1997–98 through 2005–06, the percentage of African Americans in online classes averaged 16%, and between 2006–07 and 2009–10, the average was 23% (a statistically significant increase, p = .05). Although most of the difference is gone, some remains. Our hopes to attract relatively more minorities to the online classes have been only partially fulfilled. We suspect that this remaining difference is largely due to the same factors national studies have identified. Our nonwhite students come from homes with parents having significantly lower levels of education than white students (p =0.000). Those homes are less likely to have had computers or broadband access.

A 1997 survey of our university's students found that their knowledge of American government was about the same or below that of the general public (Botsch 1998/1999). Similarly, most students entering our classes knew little about American government and politics. In general, Americans know little about the Constitution and their system of government (“Americans' Awareness of First Amendment Freedoms” 2006; Delli Carpini and Keeter 1996; Gallup 2003; “Knowing it by Heart” 2002; “Our Fading Heritage” 2008; “We the People” 1997). Over the years, studies conducted by survey research organizations consistently find that Americans have low levels of political knowledge and that many are unable to identify important political figures and do not closely follow current events.¹¹ In this abyss of ignorance, web-based students, however, proved to be relatively better informed than students in traditional classes. A decade ago web-based students scored about 7 points higher on the general knowledge pretest than face-to-face students, and that difference remains about the same today. Over the entire period, out of a possible score of 63, the average score was 11.6 for the face-to-face classes and 18.9 for the web-based classes, a large and significant difference (p = 0.000). We must add that these are depressingly low scores for students in both kinds of classes. We face great challenges as teachers.

We measured a number of factors together that explained 27% of the total variation in pretest scores for all students (GPA, gender, ethnicity, political interest, political efficacy, daily newspaper reading, and age). Students who chose the web-based classes differed significantly on many of these factors in ways that help explain their relatively higher scores, as shown in table 1. Web-based students had a significantly higher GPA, were more likely to be white, to read newspapers more frequently, and were a little older, as previously discussed. On political interest no difference existed between online and traditional students. Gender and political efficacy played no significant role although both gender¹² and efficacy, by themselves, were strongly associated with low scores.¹³

OUTCOMES

Format did have an impact on changes in attitudes and behaviors. Most of the changes, although significant, were relatively small and did not follow any strong pattern favoring one format more than the other, as shown in table 1. As we explore these changes, we isolate the impact of delivery format by accounting for differences in the students who were taking the two class formats.

We begin with factual knowledge. When online teaching began, educators asked, first, whether it would be comparable to or, second, better than traditional teaching. With more than a decade of experience and collected data measuring factual knowledge gains, our answers are “yes” to the first query and a qualified “no” to the second. Students in both groups improved their scores from the pretest to the posttest in a near statistical tie, as shown in table 1. That students in the online classes had higher GPAs made little difference.¹⁴ We are doing just as well, or just as poorly, with both groups in the transmission of knowledge.¹⁵

Improvements in critical thinking are harder to measure. Online students complete relatively more assignments requiring critical thinking. If we could measure critical thinking improvements, we suspect that high-GPA students in online classes would have more improvement here than high-GPA face-to-face students, because grades in the online classes require more analytical thinking.¹⁶

Many studies chronicle decreases in political interest among Americans¹⁷ (Bennet 1997; Norris 2004). As teachers, increasing political interest is our goal. Political interest increased significantly in both types of classes, but increased slightly more in the web-based classes than in the face-to-face classes as shown in table 1.

Like political interest, political trust has also declined in the United States. Many people mark the decline beginning with the Vietnam War and the events surrounding Watergate (“The Trust in Government” 2008). The long, costly wars in Afghanistan and Iraq, a seemingly never-ending series of scandals involving political office holders behaving very badly, an economic crisis that necessitated a costly and highly unpopular financial bailout, and ballooning deficits and cuts in education and many services have not improved attitudes. All our students began the course with low trust, with the younger and less-knowledgeable face-to-face students having slightly less trust. Students' political trust improved in both classes, with face-to-face students increasing slightly more, so that they caught up with the online students, as shown in table 1. Knowing more about how government operates seems to increase political trust. We note that ultimately trust rests more on policy successes than on anything professors can do.

For the last six decades, between 59% and 71% of American citizens have agreed that politics is “too complicated to understand” (The ANES Guide 2008; “Politics Is Too Complicated” 1999). Although our web-based students scored lower, all our students scored low on political efficacy as they entered the classes. Web-based students improved relatively more, so that both groups ended up with about the same level of confidence, as shown in table 1. We note that while we met our goal of improving efficacy, coming close to only having “mixed feelings” is less than impressive.

For a long time, Americans have paid little attention to current events.¹⁸ Recently, we see little improvement.¹⁹ Of course, one of the best ways to keep up with current events in any detail is to read newspapers every day. We asked students how many days during the past week they had read a newspaper. On average, students in both types of classes read the newspaper about two days a week when they entered the course, as shown in table 1. This number is down from what we observed a decade ago. That is not surprising given the long-term decline in traditional newspapers. Web-based students read a paper slightly more often. Both groups increased by the end of the semester by nearly a day a week, with web-based students increasing significantly more, expanding the difference they enjoyed at the beginning of the semester, as shown in table 1. Most likely, these gains are attributed to the nearly weekly news assignments we give students in both classes. Whether these gains continue is the critical question.

Web-based students' final grades were five points higher than those of face-to-face students, as shown in table 1. Most of this difference is due to online students being somewhat more skilled students who entered the class with higher GPAs. Some of the grade differences were the result of different grading criteria in the two formats. Performance in the web-based classes is measured relatively more on essay questions and completion of written assignments, including discussions. Consequently, effort counts more in the online classes, so that students who make more effort are rewarded relatively more.²⁰

However, despite higher grades, the online class is not easier, and the students perceived a difference. This is no accident. We designed the online class to be rigorous and academically demanding. A question on the posttest asked students to rate the difficulty of the course compared to other classes. Students in both class formats saw their classes as harder than other classes (a rating of 2.0 on the 1 to 3 scale would have been the same level of difficulty, as shown in table 1). But relative to the face-to-face students, web-based students felt their class was even harder.

Yet, despite the fact that students perceive the online class to be relatively more difficult, demand for this class remains high and is growing. For most of the time covered by this study, we offered only a single American government web-based class each semester. Now, we regularly offer two web-based sections. The web-based classes fill faster and regularly have waiting lists. In response to this demand, we now teach an increasing variety of other classes in an asynchronous online format as well as in the usual face-to-face formats. This experience suggests that students may value scheduling convenience more than they fear demanding courses.

When this study began in 1997–98, we saw a class dropout rate of around 25%, about average for this kind of course according to the literature at the time (Merisotis 1999). More current research suggests a dropout rate of 15% to 20% higher than in “traditional” classes (Parry 2010).²¹ Over the next few years, however, dropout rates fell quickly: when we published our first study, we saw no difference, as shown in table 1. For the first few years students who were not willing and able to face a new format seemed to avoid the web-based classes. However, as more traditional students chose the online format and were more comfortable with computers, a better cross section of students chose web-based classes—including many who lacked the necessary self-discipline. Dropout rates for web-based classes are now significantly higher than face-to-face classes, as shown in table 1. What may have seemed like an easy option—not ever having to go to class—turned out to be more difficult.

Another kind of outcome is the impact on workload for professors. Much research indicates that web-based teaching takes more time than face-to-face classes (Berdichevsky 1999; Bradshaw and Weston 1999; Clark-Ibanez and Scott 2008; Maguire 2005; Tallent-Runnels et al. 2006). This outcome certainly has been true for us, even with technological changes that save significant grading time, like the Blackboard reading mastery tests we use now. Although we save time not having to grade these routine objective informational quizzes, we also have to create them and compose comments on right and wrong answers that explain and clarify. Any time we save here easily gets used in making more detailed responses in online class discussions. Reflecting on comments made by 20 to 25 students about the material, commenting on current news articles that presumably illustrate things in the chapters, and grading essay exams are all time-consuming. Of course, much of this also happens in our face-to-face classes, where we also save some time using Blackboard for the reading mastery tests. And that gives us a little extra time for commenting on the assignments students turn in. The biggest differences are (1) in the web-based classes the exams are all essay and open book, consequently longer than the in-class essays that face-to-face students write, and (2) web discussions are all typed and often require individual responses to each student, while in face-to-face classes not all students speak in class and our verbal responses to those who do, take little time. The only way to save significant time would be to reduce the online discussions or move to objective timed tests using Blackboard rather than essay exams, which in our view would reduce the quality of the course.

Although we both enjoy web-based teaching, we would not like to do it exclusively, because we also enjoy the physical and social dynamics and spontaneity that take place in the classroom. Moreover, some exercises and skills, such as simulations or group decision-making or research and statistical exercises that are necessary in research methods classes, do not lend themselves as well to an online format.

CONCLUSIONS

We draw several conclusions about differences in face-to-face and web-based classes from our 13-year experience. First, the audiences for the classes have become far more similar than they were when we started offering classes online. Still, some lingering differences remain. Web-based students are a bit older, have higher GPAs, are more knowledgeable about the subject matter, read newspapers more, are more likely to be women, are slightly more likely to be in majors that rely more on computers, and are slightly less likely to be African American. The driving forces behind these remaining student differences are convenience and flexibility in scheduling and students' comfort in using computers and being future-oriented enough to preregister for these popular classes.

Second, and perhaps most important, is what we did not find. We found no significant difference in factual knowledge gained between the two delivery formats. Moreover, differences in changes in political interest, trust, and efficacy, although statistically significant, were slight and had no clear pattern. Carefully constructed web-based classes in American government can be as effective as traditional lecture/discussion classes in nurturing an interested, trusting, confident, and knowledgeable citizenry.

Third, some differences were larger and seemed more important to us. Newspaper reading gains were significantly greater among web-based students, although we wonder if the gains will be lasting. Web-based students earned significantly higher grades, although that may be explained mostly by their better student skills. Yet despite higher grades, web-based students perceived their classes to be relatively more difficult than face-to-face students because of the greater writing workload in the online format. Dropout rates were also significantly higher for the web-based classes. Although we did not keep hard data on demand, we perceive demand to be higher for web-based classes. These classes almost always close out first and have a much better chance of making the minimum enrollment necessary in the summer semester.²² Finally, we have found our own workload to be greater in web-based classes than face-to-face classes, at least the way we teach them. Given these tradeoffs, our own preferences are to teach one web-based class a semester, although two can work well if we want more flexible schedules and want to reduce pressure to teach at odd hours or in remote locations in an environment of limited classroom space. Given the workload, three web-based classes would be too demanding. Reducing the workload by relying more on standardized testing and less on written essays and give and take in online discussions would, in our view, sacrifice too much quality. Moreover, if we taught most or all of our courses online, we would miss the interpersonal interactions that take place in face-to-face classes, some of which is certainly an important part of learning essential political skills.

Online classes are now rather standard fare on most college campuses and are more accepted and more representative of all students. Here, we have shown that carefully constructed web-based classes can produce similar outcomes to traditional face-to-face classes. Yet some important differences remain in audiences and outcomes. Some of these remaining differences may also dissipate over time as all students become comfortable taking classes online and as face-to-face classes use more online resources to become more like web-based classes. We can still exploit some of these differences to attract more and slightly different students, add flexibility to our own schedules, create a better fit with the busy lives of students, and provide a greater diversity of course experiences that better match a world in which more and more human interaction is online.