Hostname: page-component-cd9895bd7-dzt6s Total loading time: 0 Render date: 2024-12-17T09:29:45.011Z Has data issue: false hasContentIssue false
  • English
  • Français

Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures

Published online by Cambridge University Press:  27 September 2021

Luwei Ying Open the ORCID record for Luwei Ying [Opens in a new window] ,
Jacob M. Montgomery Open the ORCID record for Jacob M. Montgomery [Opens in a new window]  and
Brandon M. Stewart Open the ORCID record for Brandon M. Stewart [Opens in a new window]
Luwei Ying*
Affiliation:
Department of Political Science, Washington University in St. Louis, Greater St. Louis, MO, USA. E-mail: [email protected]
Jacob M. Montgomery
Affiliation:
Department of Political Science, Washington University in St. Louis, Greater St. Louis, MO, USA. E-mail: [email protected]
Brandon M. Stewart
Affiliation:
Department of Sociology and the Office of Population Research, Princeton University, Princeton, NJ, USA. E-mail: [email protected]
*
Corresponding author Luwei Ying
Get access Rights & Permissions [Opens in a new window]

Abstract

Topic models, as developed in computer science, are effective tools for exploring and summarizing large document collections. When applied in social science research, however, they are commonly used for measurement, a task that requires careful validation to ensure that the model outputs actually capture the desired concept of interest. In this paper, we review current practices for topic validation in the field and show that extensive model validation is increasingly rare, or at least not systematically reported in papers and appendices. To supplement current practices, we refine an existing crowd-sourcing method by Chang and coauthors for validating topic quality and go on to create new procedures for validating conceptual labels provided by the researcher. We illustrate our method with an analysis of Facebook posts by U.S. Senators and provide software and guidance for researchers wishing to validate their own topic models. While tailored, case-specific validation exercises will always be best, we aim to improve standard practices by providing a general-purpose tool to validate topics as measures.

Keywords

text as datatopic modelvalidationcrowd-sourcingmeasurement
Type
Article
Information
Political Analysis , Volume 30 , Issue 4 , October 2022 , pp. 570 - 589
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Edited by Jeff Gill

References

Adcock, R., and Collier, D.. 2001. “Measurement Validity: A Shared Standard for Qualitative and Quantitative Research.” American Political Science Review 95(3):529–546.CrossRefGoogle Scholar
Armstrong, J. S. 1967. “Derivation of Theory by Means of Factor Analysis or Tom Swift and His Electric Factor Analysis Machine.” The American Statistician 21(5):17–21.Google Scholar
Arora, S., et al. 2013. “A Practical Algorithm for Topic Modeling with Provable Guarantees.” In Proceedings of the International Conference on Machine Learning, 280–288. Atlanta, Georgia, USA. http://proceedings.mlr.press/v28/arora13.html Google Scholar
Bagozzi, B. E., and Berliner, D.. 2018. “The Politics of Scrutiny in Human Rights Monitoring: Evidence from Structural Topic Models of US State Department Human Rights Reports.” Political Science Research and Methods 6(4):661–677.CrossRefGoogle Scholar
Barberá, P., et al. 2019. “Who Leads? Who Follows? Measuring Issue Attention and Agenda Setting by Legislators and the Mass Public Using Social Media Data.” American Political Science Review 113(4):883–901.CrossRefGoogle ScholarPubMed
Barnes, L., and Hicks, T.. 2018. “Making Austerity Popular: The Media and Mass Attitudes toward Fiscal Policy.” American Journal of Political Science 62(2):340–354.CrossRefGoogle Scholar
Benoit, K., Conway, D., Lauderdale, B. E., Laver, M., and Mikhaylov, S.. 2016. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review 110(2):278–295.CrossRefGoogle Scholar
Blaydes, L., Grimmer, J., and McQueen, A.. 2018. “Mirrors for Princes and Sultans: Advice on the Art of Governance in the Medieval Christian and Islamic Worlds.” Journal of Politics 80(4):1150–1167.CrossRefGoogle Scholar
Blei, D. M., Ng, A. Y., and Jordan, M. I.. 2003. “Latent Dirichlet Allocation.” Journal of Machine Learning Research 3:993–1022.Google Scholar
Blumenau, J., and Lauderdale, B. E.. 2018, January. “Never Let a Good Crisis Go to Waste: Agenda Setting and Legislative Voting in Response to the EU Crisis.” The Journal of Politics 80 (2):462–478.CrossRefGoogle Scholar
Brady, H. E. 2010. “Doing Good and Doing Better: How Far Does the Quantitative Template Get Us.” In Rethinking Social Inquiry: Diverse Tools, Shared Standards, edited by Brady, H. E. and Collier, D. (Second Edition). Lanham, MD: Rowman & Littlefield.Google Scholar
Cacioppo, J. T., and Petty, R. E.. 1982. “The Need for Cognition.” Journal of Personality & Social Psychology 42(1):116–131.CrossRefGoogle Scholar
Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J. L., and Blei, D. M.. 2009. “Reading Tea Leaves: How Humans Interpret Topic Models.” In Advances in Neural Information Processing Systems, edited by Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C. K. I., and Culotta, A., 288–296. Cambridge, MA: The MIT Press.Google Scholar
Clinton, J., Jackman, S., and Rivers, D.. 2004. “The Statistical Analysis of Roll Call Data.” American Political Science Review 98(2):355–370.CrossRefGoogle Scholar
Denny, M. J., and Spirling, A.. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do about It.” Political Analysis 26(2):168–189.CrossRefGoogle Scholar
Dietrich, B. J., Hayes, M., and O’Brien, D. Z.. 2019. “Pitch Perfect: Vocal Pitch and the Emotional Intensity of Congressional Speech.” American Political Science Review 113(4):941–962.CrossRefGoogle Scholar
Egami, N., Fong, C. J., Grimmer, J., Roberts, M. E., and Stewart, B. M.. 2018. “How to Make Causal Inferences Using Texts.” arXiv:1802.02163.Google Scholar
Eshima, S., Imai, K., and Sasaki, T.. 2020. “Keyword Assisted Topic Models.” arXiv:2004.05964.Google Scholar
Gelman, A., and Loken, E.. 2013. “The Garden of Forking Paths: Why Multiple Comparisons can be a Problem, Even when there is no ‘Fishing Expedition’ or ‘p-hacking’ and the Research Hypothesis was Posited Ahead of Time.” Department of Statistics, Columbia University 348.Google Scholar
Gibson, J. L., and Bingham, R. D.. 1982. “On the Conceptualization and Measurement of Political Tolerance.” The American Political Science Review 76(3):603–620.CrossRefGoogle Scholar
Gilardi, F., Shipan, C. R., and Wüest, B.. 2021. “Policy Diffusion: The Issue-Definition Stage.” American Journal of Political Science 65(1):21–35.CrossRefGoogle Scholar
Grimmer, J. 2010. “A Bayesian Hierarchical Topic Model for Political Texts: Measuring Expressed Agendas in Senate Press Releases.” Political Analysis 18(1):1–35.CrossRefGoogle Scholar
Grimmer, J. 2013. “Appropriators not Position Takers: The Distorting Effects of Electoral Incentives on Congressional Representation.” American Journal of Political Science 57(3):624–642.CrossRefGoogle Scholar
Grimmer, J., and King, G.. 2011. “General Purpose Computer-Assisted Clustering and Conceptualization.” Proceedings of the National Academy of Sciences 108(7):2643–2650.CrossRefGoogle ScholarPubMed
Grimmer, J., Roberts, M. E., and Stewart, B. M.. 2021. “Machine Learning for Social Science: An Agnostic Approach.” Annual Review of Political Science 24:395–419.CrossRefGoogle Scholar
Grimmer, J., and Stewart, B. M.. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3):267–297.CrossRefGoogle Scholar
Horowitz, M., et al. 2019. “What Makes Foreign Policy Teams Tick: Explaining Variation in Group Performance at Geopolitical Forecasting.” The Journal of Politics 81(4):1388–1404.CrossRefGoogle Scholar
John, L. K., Loewenstein, G., and Prelec, D.. 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science 23(5):524–532.CrossRefGoogle ScholarPubMed
Karell, D., and Freedman, M. R.. 2019. “Rhetorics of Radicalism.” American Sociological Review 84(4):726–753.CrossRefGoogle Scholar
Kim, S. E. 2018. “Media Bias against Foreign Firms as a Veiled Trade Barrier: Evidence from Chinese Newspapers.” American Political Science Review 112(4):954–970.CrossRefGoogle Scholar
King, G. 2009. “The Changing Evidence Base of Social Science Research.” In The Future of Political Science: 100 Perspectives, edited by King, G., Schlozman, K. L., and Nie, N., 91–93. New York: Routledge.CrossRefGoogle Scholar
King, G., Keohane, R. O., and Verba, S.. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton: Princeton University Press.CrossRefGoogle Scholar
Lacombe, M. J. 2019, July. “The Political Weaponization of Gun Owners: The National Rifle Association’s Cultivation, Dissemination, and Use of a Group Social Identity.” The Journal of Politics 81 (4):1342–1356.CrossRefGoogle Scholar
Lau, J. H., Grieser, K., Newman, D., and Baldwin, T. 2011. “Automatic Labelling of Topic Models.” In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 1536–1545. Portland: Association for Computational Linguistics.Google Scholar
Lowe, W., and Benoit, K.. 2013. “Validating Estimates of Latent Traits from Textual Data Using Human Judgment as a Benchmark.” Political Analysis 21(3):298–313.CrossRefGoogle Scholar
Lund, J., et al. 2019, May. “Automatic Evaluation of Local Topic Quality.” arXiv:1905.13126.CrossRefGoogle Scholar
Magaloni, B., and Rodriguez, L.. 2020. “Institutionalized Police Brutality: Torture, the Militarization of Security, and the Reform of Inquisitorial Criminal Justice in Mexico.” American Political Science Review 114(4):1013–1034.CrossRefGoogle Scholar
Martin, G. J., and McCrain, J.. 2019. “Local News and National Politics.” American Political Science Review 113(2):372–384.CrossRefGoogle Scholar
Mimno, D., Wallach, H. M., Talley, E., Leenders, M., and McCallum, A.. 2011. “Optimizing Semantic Coherence in Topic Models.” In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 262–272. Portland: Association for Computational Linguistics.Google Scholar
Motolinia, L. 2021. “Electoral Accountability and Particularistic Legislation: Evidence from an Electoral Reform in Mexico.” American Political Science Review 115(1):97–113.CrossRefGoogle Scholar
Nielsen, R. A. 2020. “Women’s Authority in Patriarchal Social Movements: The Case of Female Salafi Preachers.” American Journal of Political Science 64(1):52–66.CrossRefGoogle Scholar
Pan, J., and Chen, K.. 2018. “Concealing Corruption: How Chinese Officials Distort Upward Reporting of Online Grievances.” American Political Science Review 112(3):602–620.CrossRefGoogle Scholar
Parthasarathy, R., Rao, V., and Palaniswamy, N.. 2019. “Deliberative Democracy in an Unequal World: A Text-As-Data Study of South India’s Village Assemblies.” American Political Science Review 113(3):623–640.CrossRefGoogle Scholar
Pavlick, E., Post, M., Irvine, A., Kachaev, D., and Callison-Burch, C.. 2014. “The Language Demographics of Amazon Mechanical Turk.” Transactions of the Association for Computational Linguistics 2:79–92.CrossRefGoogle Scholar
Poole, K. T., and Rosenthal, H.. 1985. “A Spatial Model for Legislative Roll Call Analysis.” American Journal of Political Science 29(2):357–384.CrossRefGoogle Scholar
Pratto, F., Sidanius, J., Stallworth, L., and Malle, B.. 1994. “Social Dominance Orientation: A Personality Variable Predicting Social and Political Attitudes.” Journal of Personality and Social Psychology 67(4):741–741.CrossRefGoogle Scholar
Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., and Radev, D. R.. 2010. “How to Analyze Political Attention with Minimal Assumptions and Costs.” American Journal of Political Science 54(1):209–228.CrossRefGoogle Scholar
Roberts, M. E., Stewart, B. M., and Airoldi, E. M.. 2016. “A Model of Text for Experimentation in the Social Sciences.” Journal of the American Statistical Association 111(515):988–1003.CrossRefGoogle Scholar
Roberts, M. E., Stewart, B. M., and Nielsen, R. A.. 2020. “Adjusting for Confounding with Text Matching.” American Journal of Political Science 64(4):887–903.CrossRefGoogle Scholar
Roberts, M. E., Stewart, B. M., and Tingley, D.. 2016. “Navigating the Local Modes of Big Data.” In Computational Social Science: Discovery and Prediction, edited by Alvarez, M., 51–97. New York: Cambridge University Press.CrossRefGoogle Scholar
Roberts, M. E., Stewart, B. M., and Tingley, D.. 2019. “stm: An R Package for Structural Topic Models.” Journal of Statistical Software 91(2):1–40.CrossRefGoogle Scholar
Roberts, M. E., Stewart, B. M., Tingley, D., and Airoldi, E. M.. 2013. “The Structural Topic Model and Applied Social Science.” In Advances in Neural Information Processing Systems Workshop on Topic Models: Computation, Application, and Evaluation, edited by Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z. and Weinberger, K.Q., 1–20. Lake Tahoe: Harrahs and Harveys.Google Scholar
Roberts, M. E., et al. 2014. “Structural Topic Models for Open-ended Survey Responses.” American Journal of Political Science 58(4):1064–1082.CrossRefGoogle Scholar
Rozenas, A., and Stukal, D.. 2019, June. “How Autocrats Manipulate Economic News: Evidence from Russia’s State-Controlled Television.” The Journal of Politics 81(3):982–996.CrossRefGoogle Scholar
Wallach, H. M., Murray, I., Salakhutdinov, R., and Mimno, D.. 2009. “Evaluation Methods for Topic Models.” In Proceedings of the 26th Annual International Conference on Machine Learning, 1105–1112. New York: ACM.CrossRefGoogle Scholar
Wickham, H., Hester, J., and Chang, W.. 2020. “devtools: Tools to Make Developing R Packages Easier.” R package version 2.3.1.Google Scholar
Ying, L., Montgomery, J. M., and Stewart, B. M.. 2021. “Replication Data for: Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures.” https://doi.org/10.7910/DVN/S02EBF, Harvard Dataverse, V1, UNF:6:GhlNc6S7kIIif+i0ZT0ung== [fileUNF].CrossRefGoogle Scholar
Supplementary material: Link

Ying et al. Dataset

Dataset

https://doi.org/10.7910/DVN/S02EBF
Link
Supplementary material: PDF

Ying et al. supplementary material

Ying et al. supplementary material

Download Ying et al. supplementary material(PDF)
PDF 2.1 MB