Hostname: page-component-745bb68f8f-hvd4g Total loading time: 0 Render date: 2025-01-07T16:31:23.603Z Has data issue: false hasContentIssue false
  • English
  • Français

An Analysis of ANES Items and Their Use in the Construction of Political Knowledge Scales

Published online by Cambridge University Press:  04 January 2017

Matthew T. Pietryka  and
Randall C. MacIntosh
Matthew T. Pietryka*
Affiliation:
Department of Political Science, University of California—Davis, One Shields Avenue, Davis, CA 95616
Randall C. MacIntosh
Affiliation:
Department of Sociology, California State University—Sacramento, 6000 J Street, Sacramento, CA 95819-6005 e-mail: [email protected]
*
e-mail: [email protected] (corresponding author)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Valid comparisons of group scores on additive measures such as political knowledge scales require that the conditional response probabilities for individuals on the observed items be invariant across groups after controlling for their overall level of the latent trait of interest. Using a multi-group confirmatory factor analysis of knowledge items drawn from American National Election Studies, we find that the scales used in recent research are not sufficiently invariant for valid comparisons across a host of theoretically important grouping variables. We demonstrate that it is possible to construct valid invariant scales using a subset of items and show the impact of invariance by comparing results from the valid and invalid scales. We provide an analysis of differential item functioning based on grouping variables commonly used in political science research to explore the utility of each item in the construction of valid knowledge scales. An application of the VTT suggests it is more appropriate to conceive of these items as effects of a latent variable rather than cause or formative indicators. These results suggest that models attempting to explain apparent knowledge gaps between subgroups have been unsuccessful because previously constructed scales were validated by fiat.

Type
Research Article
Information
Political Analysis , Volume 21 , Issue 4 , Autumn 2013 , pp. 407 - 429
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Footnotes

Authors' note: We thank Debra Leiter and Ron Rapoport for helpful comments and Jay Dow for sharing replication materials. Supplementary materials for this article are available on the Political Analysis Web site (Pietryka and Macintosh 2013).

References

Ackerman, Terry A. 1992. A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement 29(1): 67–91.CrossRefGoogle Scholar
Ahn, T. K., Huckfeldt, Robert, and Ryan, John Barry. 2010. Communication, influence, and informational asymmetries among voters. Political Psychology 31: 763–87.Google Scholar
Alba, Joseph W., and Hasher, Lynn. 1983. Is memory schematic? Psychological Bulletin 93: 203–31.Google Scholar
Alvarez, R. Michael, and Nagler, Jonathan. 2004. Party system compactness: Measurement and consequences. Political Analysis 12(1): 46–62.CrossRefGoogle Scholar
American National Election Studies (www.electionstudies.org). 2010. Time Series Cumulative Data File [dataset]. Stanford University and the University of Michigan [producers and distributors].Google Scholar
Arbuckle, James L. 2009. Amos 18 User's Guide [Computer software]. Crawfordville, FL: Amos Development Corporation.Google Scholar
Asparouhov, Tihomir, and Muthen, Bengt. 2006. Robust chi square difference testing with mean and variance adjusted test statistics (Web Notes: No. 10, May 26, 2006). http://www.statmodel.com/download/webnotes/webnote10.pdf (accessed July 12, 2010).Google Scholar
Axelrod, Robert. 1973. Schema theory: An information processing model of perception and cognition. American Political Science Review 67(4): 1248–66.Google Scholar
Bagozzi, Richard P. 2011. Measurement and meaning in information systems and organizational research: Methodological and philosophical foundations. MIS Quarterly 35(2): 261–92.CrossRefGoogle Scholar
Bartels, Larry M. 1996. Uninformed votes: Information effects in presidential elections. American Journal of Political Science 40(1): 194–230.CrossRefGoogle Scholar
Blalock, Hubert M. 1964. Causal inferences in nonexperimental research. Chapel Hill: University of North Carolina Press. http://www.citeulike.org/group/108/article/106824 (accessed January 16, 2013).Google Scholar
Bollen, Kenneth A. 2011. Evaluating effect, composite and causal indicators in structural equation models. MIS Quarterly 35(2): 359–72.Google Scholar
Bollen, Kenneth A., and Ting, Kwok-Fai. 1993. Confirmatory tetrad analysis. Sociological Methodology 23: 147–75.CrossRefGoogle Scholar
Bollen, Kenneth A., and Ting, Kwok-Fai. 2000. A tetrad test for causal indicators. Psychological Methods 5(1): 3–22.CrossRefGoogle ScholarPubMed
Bollen, Kenneth, and Lennox, Richard. 1991. Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin 110(2): 305–14.Google Scholar
Bollen, Kenneth A., and Bauldry, Shawn. 2011. Three cs in measurement models: Causal indicators, composite indicators and covariates. Psychological Methods 16(3): 265–84.Google Scholar
Borsboom, Denny. 2006. The attack of the psychometricians. Psychometrika 71(3): 425–40.Google Scholar
Brown, Timothy. 2006. Confirmatory factor analysis for applied research. New York: Gilford Press.Google Scholar
Byrne, Barbara, Shavelson, Richard, and Muthen, Bengt. 1989. Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin 105: 456–66.CrossRefGoogle Scholar
Clinton, Joshua, Jackman, Simon, and Rivers, Douglas. 2004. The statistical analysis of roll call data. American Political Science Review 98(02): 355–70.Google Scholar
Conover, Pamela Johnston, and Feldman, Stanley. 1984. How people organize the political world: A schematic model. American Journal of Political Science 28(1): 95–126.Google Scholar
Davidov, Eldad. 2009. Measurement equivalence of nationalism and constructive patriotism in the ISSP: 34 countries in a comparative perspective. Political Analysis 17(1): 64–82.Google Scholar
Davis, Darren, and Silver, Brian D. 2003. Stereotype threat and race of interviewer effects in a survey on political knowledge. American Journal of Political Science 47(1): 33–45.Google Scholar
Carpini, Delli, Michael, X., and Keeter, Scott. 1993. Measuring political knowledge: Putting first things first. American Journal of Political Science 37(4): 1179–206.Google Scholar
Carpini, Delli, Michael, X., and Keeter, Scott. 1996. What Americans know about politics and why it matters. New Haven, CT: Yale University Press.Google Scholar
Carpini, Delli, Michael, X., and Keeter, Scott. 2000. Gender and political knowledge. In Gender and American politics: Women, men and the political process, eds. Tolleson-Rinehart, Sue and Josephson, Jyl. J., 21–52. Armonk, NY: M.E. Sharpe.Google Scholar
Diamantopoulos, Adamantios, and Winklhofer, Heidi M. 2001. Index construction with formative indicators: An alternative to scale development. Journal of Marketing Research 38(2): 269–77.Google Scholar
Dolan, Kathleen. 2011. Do women and men know different things? Measuring gender differences in political knowledge. Journal of Politics 73(01): 97–107.CrossRefGoogle Scholar
Dow, Jay K. 2009. Gender differences in political knowledge: Distinguishing characteristics-based and returns-based differences. Political Behavior 31(1): 117–36.CrossRefGoogle Scholar
Downs, Anthony. 1957. An economic theory of democracy. New York: Harper and Row.Google Scholar
Edwards, Jeffrey R. 2001. Multidimensional constructs in organizational behavior research: An integrative analytical framework. Organizational Research Methods 4(14): 144–92.Google Scholar
Edwards, Jeffrey R. 2011. The fallacy of formative measurement. Organizational Research Methods 14(2): 370–88.Google Scholar
Edwards, Jeffrey R., and Bagozzi, Richard P. 2000. On the nature and direction of relationships between constructs and measures. Psychological Methods 5(2): 155–74.CrossRefGoogle ScholarPubMed
Fornell, Claes, and Bookstein, Fred. 1982. Two structural equation models: LISREL and PLS applied to consumer exit-voice theory. Journal of Marketing Research 19(4): 440–52.Google Scholar
Franz, Michael M., Freedman, Paul B., Goldstein, Kenneth M., and Ridout, Travis N. 2007. Campaign Advertising and American Democracy. Philadelphia: Temple University Press.Google Scholar
Frazer, Elizabeth, and Macdonald, Kenneth. 2003. Sex differences in political knowledge in Britain. Political Studies 51(1): 67–83.Google Scholar
Freedman, Paul B., Franz, Michael M., and Goldstein, Kenneth M. 2004. Campaign advertising and democratic citizenship. American Journal of Political Science 48(4): 723–41.Google Scholar
Galston, William A. 2001. Political knowledge, political engagement, and civic education. Annual Review of Political Science 4(1): 217–34.CrossRefGoogle Scholar
Gibson, James L., and Caldeira, Gregory A. 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the High Court. Journal of Politics 71(02): 429–41.Google Scholar
Hauser, Robert M., and Goldberger, Arthur S. 1971. The treatment of unobservable variables in path analysis. Sociological Methodology 3: 81–117.Google Scholar
Highton, Benjamin. 2009. Revisiting the relationship between educational attainment and political sophistication. Journal of Politics 71(04): 1564–76.Google Scholar
Hipp, John R., Bauer, Daniel J., and Bollen, Kenneth A. 2005. Conducting tetrad tests of model fit and contrasts of tetrad-nested models: A new SAS macro. Structural Equation Modeling: A Multidisciplinary Journal 12(1): 76–93.CrossRefGoogle Scholar
Hipp, John R., and Bollen, Kenneth A. 2003. Model fit in structural equation models with censored, ordinal, and dichotomous variables: Testing vanishing tetrads. Sociological Methodology 33(1): 267–305.Google Scholar
Horn, John, and McArdle, John. 1992. A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research 18: 117–44.Google Scholar
Huckfeldt, Robert. 2001. The social communication of political expertise. American Journal of Political Science 45(2): 425–38.Google Scholar
Huckfeldt, R., and Sprague, J. 1995. Citizens, politics, and social communication: Information and influence in an election campaign. New York: Cambridge University Press.Google Scholar
Iacobucci, Dawn. 2010. Structural equations modeling: Fit indices, sample size, and advanced topics. Journal of Consumer Psychology 20(1): 90.CrossRefGoogle Scholar
Jackman, Simon. 2008. Measurement. In The Oxford handbook of political methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David, 119–51. New York: Oxford University Press.Google Scholar
Jöreskog, Karl, and Sörbom, Dag. 2006. LISREL 8.8 for Windows [Computer software]. Lincolnwood, IL: Scientific Software International, Inc.Google Scholar
Kolen, Michael J., and Brennan, Robert L. 2004. Test equating, scaling and linking: Methods and practices, 2nd ed. New York: Springer.Google Scholar
Krosnick, Jon A., Lupia, Arthur, DeBell, Matthew, and Donakowski, Darrell. 2008. Problems with ANES questions measuring political knowledge. http://www.electionstudies.org/announce/newsltr/20080324PoliticalKnowledgeMemo.pdf (accessed June 22, 2010).Google Scholar
Lambert, Ronald D., Curtis, James E., Kay, Barry J., and Brown, Steven D. 1988. The social sources of political knowledge. Canadian Journal of Political Science 21(2): 359–74.Google Scholar
Lau, Richard R., and Redlawsk, David P. 2001. Advantages and disadvantages of cognitive heuristics in political decision making. American Journal of Political Science 45(4): 951–71.Google Scholar
Lau, Richard R., and Redlawsk, David P. 2008. Older but wiser? Effects of age on political cognition. Journal of Politics 70(1): 168–185.Google Scholar
Lizotte, Mary-Kate, and Sidman, Andrew H. 2009. Explaining the gender gap in political knowledge. Politics & Gender 5(2): 127–51.Google Scholar
Lodge, Milton, and Taber, Charles S. 2000. Three steps toward a theory of motivated political reasoning. In Elements of reason: Cognition, choice, and the bounds of rationality, eds. Lupia, Arthur, McCubbins, Matthew D., and Popkin, Samuel L., 183–213. New York, NY: Cambridge University Press.Google Scholar
Lodge, Milton, McGraw, Kathleen, and Stroh, Patrick. 1989. An impression-driven model of candidate evaluation. American Political Science Review 83(2): 399–419.CrossRefGoogle Scholar
Lupia, Arthur. 1994. Shortcuts versus encyclopedias: Information and voting behavior in California insurance reform elections. American Political Science Review 88(1): 63–76.CrossRefGoogle Scholar
Luskin, Robert C. 1987. Measuring political sophistication. American Journal of Political Science 31(4): 856–99.Google Scholar
Luskin, Robert C. 1990. Explaining political sophistication. Political Behavior 12(4): 331–61.Google Scholar
MacCallum, Robert C., and Browne, Michael W. 1993. The use of causal indicators in covariance structure models: Some practical issues. Psychological Bulletin 114(3): 533–41.Google Scholar
Macdonald, Stuart Elaine, Rabinowitz, George, and Listhaug, Ola. 1995. Political sophistication and models of issue voting. British Journal of Political Science 25(4): 453–83.Google Scholar
MacKenzie, Scott B., Podsakoff, Philip M., and Jarvis, Cheryl Burke. 2005. The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology 90(4): 710–30.Google Scholar
McGlone, Matthew S., Aronson, Joshua, and Kobrynowicz, Diane. 2006. Stereotype threat and the gender gap in political knowledge. Psychology of Women Quarterly 30(4): 392–98.Google Scholar
Meredith, William. 1993. Measurement invariance, factor analysis and factorial invariance. Psychometrika 58: 525–43.Google Scholar
Miller, Warren E., Kinder, Donald R., Rosenstone, Steven J., and the National Election Studies. American national election studies, 1992 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor], 1999.Google Scholar
Millsap, Roger, and Yun-Tein, Jenn. 2004. Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research 39: 479–515.Google Scholar
Mondak, Jeffery J. 1999. Reconsidering the measurement of political knowledge. Political Analysis 8(1): 57–82.Google Scholar
Mondak, Jeffery J. 2001. Developing valid knowledge scales. American Journal of Political Science 45(1): 224–38.Google Scholar
Mondak, Jeffery J., and Anderson, Mary R. 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2): 492–512.Google Scholar
Morehouse Mendez, Jeanette, and Tracy Osborn. 2010. Gender and the perception of knowledge in political discussion. Political Research Quarterly 63(2): 270–80.Google Scholar
Muthen, Bengt, and Christoffersson, Anders. 1981. Simultaneous factor analysis of dichotomous variables in several groups. Psychometrika 46: 407–19.Google Scholar
Muthen, Bengt, and Asparouhov, Tihomir. 2002. Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus (Mplus Web Notes: No. 4, Version 5, December 9, 2002). http://www.statmodel.com/examples/webnote.shtml#web4 (accessed May 6, 2010).Google Scholar
Muthen, Linda, and Muthen, Bengt. 2010. Mplus User's Guide. 6th ed. (1998–2010). Los Angeles, CA: Muthen & Muthen.Google Scholar
Nunnally Jum, C., Ira, H., and Bernstein, . 1994. Psychometric theory. 3rd ed. NewYork: McGraw-Hill.Google Scholar
Palfrey, Thomas R., and Poole, Keith T. 1987. The relationship between information, ideology, and voting behavior. American Journal of Political Science 31(3): 511–30.Google Scholar
Pantoja, Adrian D., and Segura, Gary M. 2003. Fear and loathing in California: Contextual threat and political sophistication among Latino voters. Political Behavior 25(3): 265–86.Google Scholar
Pietryka, Matthew T., and MacIntosh, Randall C. 2013. Replication data for: An analysis of ANES items and their use in the construction of political knowledge scales. http://hdl.handle.net/(1902).1/21218UNF:5:mZ5Xd6shurv4qyVRk7M4Fg==IQSS Dataverse Network [Distributor] V1 [Version].Google Scholar
Popkin, Samuel L., and Dimock, Michael A. 1999. Political knowledge and citizen competence. In Citizen competence and democratic institutions, eds. Elkin, Stephen L. and Edward Sołtan, Karol, 117–46. University Park, PA: Penn State Press.Google Scholar
Prior, Markus. 2005. News vs. entertainment: How increasing media choice widens gaps in political knowledge and turnout. American Journal of Political Science 49(3): 577–92.Google Scholar
Prior, Markus, and Lupia, Arthur. 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1): 169–83.Google Scholar
Rapoport, Ronald B. 1979. What they don't know can hurt you. American Journal of Political Science 23(4): 805–15.Google Scholar
Satorra, Albert. 2000. Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In Innovations in multivariate statistical analysis: A Festschrift for Heinz Neudecker, eds. Heijmans, Risto D. H., Pollock, D. S. G., and Satorra, Albert, 233–47. New York: Springer. http://www.econ.upf.edu/docs/papers/downloads/395.pdf (accessed July 14, 2010).Google Scholar
Satorra, Albert, and Bentler, Peter. 1999. A scaled difference chi-square test statistic for moment structure analysis. Technical Report, University of California, Los Angeles. http://www.springerlink.com/content/q4u780616r702384/ (accessed September 15, 2011).Google Scholar
Seeman, Melvin. 1966. Alienation, membership, and political knowledge: A comparative study. Public Opinion Quarterly 30(3): 353–67.Google Scholar
Shadish, W., Cook, T., and Campbell, D. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.Google Scholar
Smiley, M. 1999. Democratic citizenship: A question of competence. In Citizen competence and democratic institutions, eds. Elkin, Stephen L. and Edward Soltan, Karol, 371–83. University Park, PA: Pennsylvania State University Press.Google Scholar
Stegmueller, Daniel. 2011. Apples and oranges? The problem of equivalence in comparative research. Political Analysis 19(4): 471–87.Google Scholar
Stolle, Dietlind, and Gidengil, Elisabeth. 2010. What do women really know? A gendered analysis of varieties of political knowledge. Perspectives on Politics 8(1): 93.Google Scholar
Taber, Charles S., and Lodge, Milton. 2006. Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science 50(3): 755–69.Google Scholar
The American National Election Studies (www.electionstudies.org). The 1996 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor].Google Scholar
The American National Election Studies (www.electionstudies.org). The 2000 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor].Google Scholar
The American National Election Studies (www.electionstudies.org) Time series cumulative data file [dataset]. Stanford University and the University of Michigan [producers and distributors], 2010.Google Scholar
The National Election Studies (www.electionstudies.org). The anes 2004 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor].Google Scholar
Vandenberg, Robert, and Lance, Charles. 2000. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods 2: 4–69.Google Scholar
Verba, Sidney, Schlozman, Kay Lehman, and Brady, Henry E. 1995. Voice and equality: Civic voluntarism in American politics. Cambridge, MA: Harvard University Press.Google Scholar
Verba, Sidney, Burns, Nancy, and Schlozman, Kay Lehman. 1997. Knowing and caring about politics: Gender and political engagement. Journal of Politics 59(4): 1051–72.Google Scholar
Widman, Keith, and Reise, Steven. 1997. Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In The science of prevention: Methodological advances from alcohol and substance abuse research, eds. Bryant, K. J., Windle, M., and West, S. G., 281–324. Washington, DC: American Psychological Association.Google Scholar
Young, Dannagal Goldthwaite. 2004. Late-night comedy in election 2000: Its influence on candidate trait ratings and the moderating effects of political knowledge and partisanship. Journal of Broadcasting & Electronic Media 48(1): 1–22.Google Scholar
Yum, June O., and Kendall, Kathleen E. 1988. Sources of political information in a presidential primary campaign. Journalism Quarterly 65: 148–51.Google Scholar
Zaller, John R. 1992. The nature and origins of mass opinion. New York, NY: Cambridge University Press.Google Scholar
Supplementary material: PDF

Pietryka and MacIntosh supplementary material

Appendix

Download Pietryka and MacIntosh supplementary material(PDF)
PDF 488.1 KB