Hostname: page-component-78c5997874-8bhkd Total loading time: 0 Render date: 2024-11-08T09:16:21.853Z Has data issue: false hasContentIssue false
  • English
  • Français

Behavioral Consequences of Probabilistic Precision: Experimental Evidence from National Security Professionals

Published online by Cambridge University Press:  25 September 2017

Jeffrey A. Friedman ,
Jennifer S. Lerner  and
Richard Zeckhauser
Get access

Abstract

National security is one of many fields where experts make vague probability assessments when evaluating high-stakes decisions. This practice has always been controversial, and it is often justified on the grounds that making probability assessments too precise could bias analysts or decision makers. Yet these claims have rarely been submitted to rigorous testing. In this paper, we specify behavioral concerns about probabilistic precision into falsifiable hypotheses which we evaluate through survey experiments involving national security professionals. Contrary to conventional wisdom, we find that decision makers responding to quantitative probability assessments are less willing to support risky actions and more receptive to gathering additional information. Yet we also find that when respondents estimate probabilities themselves, quantification magnifies overconfidence, particularly among low-performing assessors. These results hone wide-ranging concerns about probabilistic precision into a specific and previously undocumented bias that training may be able to correct.

Type
Research Notes
Information
International Organization , Volume 71 , Issue 4 , Fall 2017 , pp. 803 - 826
Copyright
Copyright © The IO Foundation 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Barnes, Alan. 2015. Making Intelligence Analysis More Intelligent. Intelligence and National Security 31 (1):327–44.Google Scholar
Berinsky, Adam J., Huber, Gregory A., and Lenz, Gabriel S.. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk. Political Analysis 20 (3):351–68.Google Scholar
Betts, Richard K. 2006. Enemies of Intelligence: Knowledge and Power in American National Security. New York: Columbia University Press.Google Scholar
Beyerchen, Alan. 1992/93. Clausewitz, Nonlinearity, and the Unpredictability of War. International Security 17 (3):59–90.Google Scholar
Budescu, David V., Por, Han-Hui, Broomell, Stephen B., and Smithson, Michael. 2014. The Interpretation of IPCC Probabilistic Statements Around the World. Nature Climate Change 4: 508–12.Google Scholar
Cokely, Edward T., Galesic, Mirta, Schulz, Eric, Ghazal, Saima, and Garcia-Retamero, Rocio. 2012. Measuring Risk Literacy: The Berlin Numeracy Test. Judgment and Decision Making 7 (1):25–47.CrossRefGoogle Scholar
Dhami, Mandeep K. 2013. Understanding and Communicating Uncertainty in Intelligence Analysis. Report Prepared for Her Majesty's Government. London, UK.Google Scholar
Dhami, Mandeep K., Mandel, David R., Mellers, Barbara A., and Tetlock, Philip E.. 2015. Improving Intelligence Analysis with Decision Science. Perspectives on Psychological Science 10 (6):753–57.CrossRefGoogle ScholarPubMed
Ellsberg, Daniel. 1961. Risk, Ambiguity, and the Savage Axioms. Quarterly Journal of Economics 75 (4):643–69.CrossRefGoogle Scholar
Fingar, Thomas. 2011. Reducing Uncertainty: Intelligence and National Security. Stanford, CA: Stanford Security Studies.Google Scholar
Friedman, Jeffrey A., and Zeckhauser, Richard. 2015. Handling and Mishandling Estimative Probability. Intelligence and National Security 30 (1):77–99.Google Scholar
Gigerenzer, Gerd. 2002. Calculated Risks: How to Know When Numbers Deceive You. New York: Simon and Schuster.Google Scholar
Hafner-Burton, Emilie M., Haggard, Stephan, Lake, David A., and Victor, David G.. 2017. The Behavioral Revolution and the Study of International Relations. International Organization 71 (S1):S1–S31.CrossRefGoogle Scholar
Heuer, Richards J. Jr. 1999. Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence.Google Scholar
Huff, Connor, and Tingley, Dustin. 2015. “Who Are These People?” Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents. Research and Politics 2 (3):1–12.Google Scholar
Hyde, Susan D. 2015. Experiments in International Relations: Lab, Survey, and Field. Annual Review of Political Science 18 (1):403–24.CrossRefGoogle Scholar
Jervis, Robert. 1997. System Effects: Complexity in Political and Social Life. Princeton, NJ: Princeton University Press.Google Scholar
Jervis, Robert. 2010. Why Intelligence Fails: Lessons from the Iranian Revolution and the Iraq War. Ithaca, NY: Cornell University Press.Google Scholar
Johnson, Dominic D.P. 2004. Overconfidence and War: The Havoc and Glory of Positive Illusions. Cambridge, MA: Harvard University Press.Google Scholar
Johnston, Rob. 2005. Analytic Culture in the US Intelligence Community. Washington, DC: Center for the Study of Intelligence.Google Scholar
Kahneman, Daniel, and Renshon, Jonathan. 2007. Why Hawks Win. Foreign Policy 158:34–38.Google Scholar
Kent, Sherman. 1964. Words of Estimative Probability. Studies in Intelligence 8 (4):49–65.Google Scholar
Kertzer, Joshua D., and Brutger, Ryan. 2016. Decomposing Audience Costs: Bringing the Audience Back into Audience Cost Theory. American Journal of Political Science 60 (1):234–49.Google Scholar
Lanir, Zvi, and Kahneman, Daniel. 2006. An Experiment in Decision Analysis in Israel in 1975. Studies in Intelligence 50 (4). Available at <https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol50no4/an-experiment-in-decision-analysis-in-israel-in-1975.html>.Google Scholar
Levy, Jack S. 2013. Psychology and Foreign Policy Decision-Making. In The Oxford Handbook of Political Psychology, 2nd ed., edited by Huddy, Leonie, Sears, David O., and Levy, Jack S., 301–33. Oxford, UK: Oxford University Press.Google Scholar
Lowenthal, Mark M. 2006. Intelligence: From Secrets to Policy, 3rd ed. Washington, DC: CQ Press.Google Scholar
Mandel, David R., and Barnes, Alan. 2014. Accuracy of Forecasts in Strategic Intelligence. Proceedings of the National Academy of Sciences 111 (30):10984–89.Google Scholar
McChrystal, Stanley. 2009. COMISAF Initial Assessment. Kabul, Afghanistan: Headquarters, International Security Assistance Force.Google Scholar
Mellers, Barbara, Ungar, Lyle, Baron, Jonathan, Ramos, Jaime, Gurcay, Burcu, Fincher, Katrina, Scott, Sydney E., Moore, Don, Atanasov, Pavel, Swift, Samuel A., Murray, Terry, Stone, Eric, and Tetlock, Philip E.. 2014. Psychological Strategies for Winning a Geopolitical Forecasting Tournament. Psychological Science 25 (5):1106–15.Google Scholar
Mintz, Alex, and Geva, Nehemia, eds. 1997. Decisionmaking on War and Peace: The Cognitive-Rational Debate. Boulder, CO: Lynne Rienner.Google Scholar
Mosteller, Frederick, and Youtz, Cleo. 1990. Quantifying Probabilistic Expressions. Statistical Science 5 (1):2–12.Google Scholar
Piercey, M. David. 2009. Motivated Reasoning and Verbal vs. Numerical Probability Assessment: Evidence from an Accounting Context. Organizational Behavior and Human Decision Processes 108 (2):330–41.Google Scholar
Press, Daryl G., Sagan, Scott D., and Valentino, Benjamin A.. 2013. Atomic Aversion: Experimental Evidence on Taboos, Traditions, and the Non-Use of Nuclear Weapons. American Political Science Review 107 (1):188–206.Google Scholar
Rapport, Aaron. 2015. Waging War, Planning Peace: US Noncombat Operations and Major Wars. Ithaca, NY: Cornell University Press.Google Scholar
Renshon, Jonathan. 2015. Losing Face and Sinking Costs: Experimental Evidence on the Judgment of Political and Military Leaders. International Organization 69 (3):659–95.Google Scholar
Rovner, Joshua. 2011. Fixing the Facts: National Security and the Politics of Intelligence. Ithaca, NY: Cornell University Press.Google Scholar
Savage, Leonard J. 1954. The Foundations of Statistics. New York: Wiley.Google Scholar
Shapiro, Jacob N., and Cohen, Dara Kay. 2007. Color Blind: Lessons from the Failed Homeland Security Advisory System. International Security 32 (2):121—54.Google Scholar
Sunstein, Cass R. 2014. Valuing Life: Humanizing the Regulatory State. Chicago, IL: University of Chicago Press.Google Scholar
Tetlock, Philip E. 2005. Expert Political Judgment. Princeton, NJ: Princeton University Press.Google Scholar
Tetlock, Philip E., and Gardner, Dan. 2015. Superforecasting: The Art and Science of Prediction. New York: Crown.Google Scholar
Tillers, Peter, and Gottfried, Jonathan. 2006. “Case Comment—United States v. Copeland.” Law, Probability, and Risk 5 (2):135–57.Google Scholar
Tomz, Michael R., and Weeks, Jessica P.. 2013. Public Opinion and the Democratic Peace. American Political Science Review 107 (4):849–65.Google Scholar
US Army. 1997. Field Manual 101-5: Staff Organization and Operations. Washington, DC: Department of the Army.Google Scholar
US Army. 2009. Field Manual 5-0: The Operations Process. Washington, DC: Department of the Army.Google Scholar
Wallsten, Thomas. 1990. Costs and Benefits of Vague Information. In Insights in Decision Making, edited by Hogarth, Robin M., 28–43. Chicago, IL: University of Chicago Press.Google Scholar
Williamson, Vanessa. 2016. On the Ethics of Crowdsourced Research. PS: Political Science and Politics 49 (1):77–81.Google Scholar
Zimmer, Alf C. 1984. A Model for the Interpretation of Verbal Predictions. International Journal of Man-Machine Studies 20 (1):121–34.Google Scholar
Supplementary material: File

Friedman et al supplementary material

Friedman et al supplementary material 1

Download Friedman et al supplementary material(File)
File 938 KB