Hostname: page-component-586b7cd67f-2plfb Total loading time: 0 Render date: 2024-11-24T02:58:11.891Z Has data issue: false hasContentIssue false
  • English
  • Français

Learning in Games: Neural Computations underlying Strategic Learning

Published online by Cambridge University Press:  09 January 2015

Ming Hsu  and
Lusha Zhu
Ming Hsu
Affiliation:
Maas School of Business Helen Wills Neuroscience Program University of California, Berkeley
Lusha Zhu
Affiliation:
Virginia Tech Carilion Research Institute Virginia Polytechnic Institute and State University
Get access

Summary

The past decade has witnessed an unprecedented growth in our understanding of the brain basis of economic decision-making. In particular, research is uncovering not only the location of brain regions where certain processes are taking place, but also the nature of the (economically meaningful) latent variables that are represented, as well as how they relate to behavior. This transition from understanding where to how economic decisions are being made in the brain has been integral to relating neural processes to economic models of behavior. This progress, however, has been notably uneven. Neu-roeconomic studies of individual decision-making, such as those involve risk and time preferences, have the benefit of drawing on decades of work from neuroscientific studies of animal behavior. Critically, many of these findings are based on quantitative, computational approach that lends well to economic experimentation. In contrast, our understanding of the neural systems underlying social behavior is much less specific. A large measure of the current challenge in fact arises from the empirical shortcomings of standard game theoretic predictions of behavior, which are largely equilibrium-based. Using our own study as an example, we show how one can directly search for the latent variables implied by current economic models of strategic learning, and attempt to localize them in the brain. Specifically, we show that the neural systems underlying strategic learning build directly on top of those involved in simple trial-and-error learning, but incorporate additional computations that capture belief-based learning. Finally, we discuss how our approach can be extended to address fundamental problems in economics.

Les dernières décennies ont connu une croissance inédite de notre compréhension des fondements cérébraux de la prise de décision économique. En particulier, la recherche a découvert non seulement la localisation de régions du cerveau où certains processus ont lieu, mais également la nature de variables latentes (économiquement significatives) ainsi que la manière dont elles sont liées au comportement. Cette transition d'une compréhension du lieu de la décision économique vers la manière dont se prend cette décision au niveau cérébral est intégrante à l'identification d'une relation entre processus nerveux et modèles de comportements économiques.

Toutefois, le progrès accompli a été inégal. Les études neuro-économiques sur la prise de décision individuelle, telles que celles impliquant les préférences temporelles ou l'attitude face au risque, ont l'avantage de s'inscrire dans des décennies d'études neuroscientifiques sur les comportements animaliers. La plupart de ces résultats sont basés sur des approches quantitatives et informatiques, qui se prêtent aisément à l'expérimentation économique. En revanche, notre compréhension des systèmes nerveux sous-jacents au comportement social est bien moins spécifique.

Une grande partie du défi actuel résulte des lacunes empiriques des prédictions de comportement issues de la théorie des jeux standard, qui sont largement basées sur l'équilibre. Utilisant notre propre étude comme exemple, nous montrons comment il est possible de chercher directement les variables latentes induites par les modèles actuels d'apprentissage stratégique, et de tenter de les localiser dans le cerveau. Plus précisément, nous montrons que les systèmes nerveux sous-jacents à l'apprentissage stratégique s'ajoutent à ceux impliqués dans l'apprentissage par essais-erreurs, mais incluent également des calculs additionnels qui captent l'apprentissage basé sur la croyance. Finalement, nous discutons la manière dont notre approche peut être élargie pour traiter les problèmes fondamentaux de l'économie.

Keywords

strategic learninggame theoryneuroeconomicsC92D83apprentissage stratégiquethéorie des jeuxneuro-économieC92D83
Type
I) Neurocellular Economics
Information
Recherches Économiques de Louvain/ Louvain Economic Review , Volume 78 , Issue 3-4 , December 2012 , pp. 47 - 72
Copyright
Copyright © Université catholique de Louvain, Institut de recherches économiques et sociales 2012 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Amodio, D. M. and Frith, C. D. (2006). “Meeting of minds: the medial frontal cortex and social cognition.” Nature Reviews Neuroscience 7(16552413): 268–277.Google Scholar
Badre, D. (2008). ‘Cognitive control, hierarchy, and the rostro-caudal organization of the frontal lobes.” Trends in Cognitive Sciences 12(5): 193–200.Google Scholar
Bechara, A., Damasio, H., et al. (2000). “Emotion, decision making and the orbitofrontal cortex.” Cereb Cortex 10(3): 295–307.Google Scholar
Bechara, A., Tranel, D., et al. (1996). “Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex.” Cereb Cortex 6(2): 215–225.Google Scholar
Behrens, T. E. J., Hunt, L., et al. (2008). “Associative learning of social value.” Nature 456(7219): 245.Google Scholar
Bhatt, M. and Camerer, C. F. (2005). “Self-referential thinking and equilibrium as states of mind in games: fMRI evidence.” Games and Economic Behavior 52(2): 424–459.Google Scholar
Botvinick, M. M., Cohen, J. D., et al. (2004). “Conflict monitoring and anterior cingulate cortex: an update.” Trends in Cognitive Sciences 8(12): 539–546.Google Scholar
Brothers, L. (2002). “The social brain: a project for integrating primate behavior and neurophysiology in a new domain.” Foundations in social neuroscience: 367–385.Google Scholar
Camerer, C. (2003). Behavioral game theory: experiments in strategic interaction, Princeton University Press.Google Scholar
Camerer, C. (2003). Behavioral game theory: experiments in strategic interaction. New York, N.Y., Princeton University Press.Google Scholar
Camerer, C. F. and Ho, T. (1999). “Experience-weighted attraction learning in games: A unifying approach.” Econometrica 67(4): 827–874.Google Scholar
Camerer, C. F., Ho, T., et al. (2002). “Sophisticated experience-weighted attraction learning and strategic teaching in repeated games.” Journal of Economic Theory 104(1): 137–188.Google Scholar
Camerer, C. F., Loewenstein, G., et al. (2005). “Neuroeconomics: How neuroscience can inform economics.” Journal of Economic Literature 43(1): 9–64.Google Scholar
Cheung, Y.-W. and Friedman, D. (1997). “Individual Learning in Normal Form Games: Some Laboratory Results.” Games and Economic Behavior 19: 46–76.Google Scholar
Coricelli, G. and Nagel, R. (2009). “Neural correlates of depth of strategic reasoning in medial prefrontal cortex.” PNAS 106(23): 9163–9168.Google Scholar
Crawford, V. (1995). “Adaptive Dynamics in Coordination Games.” Econometrica 63(1): 103–143.Google Scholar
Critchley, H., Mathias, C., et al. (2001). “Neural activity in the human brain relating to uncertainty and arousal during anticipation.” Neuroimage 13(6): S392–S392.Google Scholar
Daw, N. and Doya, K. (2006). “The computational neurobiology of learning and reward.” Current Opinion in Neurobiology 16(2): 199–204.Google Scholar
De Martino, B., Kumaran, D., et al. (2006). “Frames, Biases, and Rational Decision-Making in the Human Brain.” Science 313(5787): 684–687.Google Scholar
de Quervain, D., Fischbacher, U., et al. (2004). “The neural basis of altruistic punishment.” Science 305(5688): 1254–1258.Google Scholar
Dorris, M. and Glimcher, P. (2004). “Activity in Posterior Parietal Cortex Is Correlated with the Relative Subjective Desirability of Action.” Neuron 44(2): 365–378.Google Scholar
Eisenegger, C., Naef, M., et al. (2009). “Prejudice and truth about the effect of testosterone on human bargaining behaviour.” Nature: 1–6.Google Scholar
Erev, I. and Rapoport, A. (1998). “Coordination, “Magic,” and Reinforcement Learning in a Market Entry Game.” Games and Economic Behavior 23(2): 146–175.Google Scholar
Erev, I. and Roth, A. E. (1998). “Predicting how people play games: Reinforcement learning in experimental games with unique, mixed strategy equilibria.” American economic review: 848–881.Google Scholar
Feltovich, N. (2000). “Reinforcement-Based vs. Belief-Based Learning Models in Experimental Asymmetric-Information Games.” Econometrica 68(3): 605–641.Google Scholar
Fiorillo, C. D., Tobler, P. N., et al. (2003). “Discrete coding of reward probability and uncertainty by dopamine neurons.” Science 299(5614): 1898–1902.Google Scholar
Fudenberg, D. and Kreps, D. M. (1993). “Learning mixed equilibria.” Games and Economic Behavior 5: 320–367.Google Scholar
Fudenberg, D. and Levine, D. K. (1998). The theory of learning in games, MIT press.Google Scholar
Fudenberg, D. and Levine, D. K. (2009). “Learning and equilibrium.” Annu. Rev. Econ. 1(1): 385–420.Google Scholar
Gallagher, H. L., Jack, A. I., et al. (2002). “Imaging the intentional stance in a competitive game.” Neuroimage 16(3 Pt 1): 814–821.Google Scholar
Hampton, A. N., Bossaerts, P., et al. (2008). “Neural correlates of mentalizing-related computations during strategic interactions in humans.” PNAS 105(18): 6741–6746.Google Scholar
Ho, T., Camerer, C., et al. (2007). “Self-tuning experience weighted attraction learning in games.” Journal of Economic Theory 133(1): 177–198.Google Scholar
Hofbauer, J. and Sigmund, K. (1998). Evolutionary games and population dynamics, Cambridge Univ Press.Google Scholar
Hsu, M., Anen, C., et al. (2008). “The Right and the Good: Distributive Justice and Neural Encoding of Equity and Efficiency.” Science 320: 1092–1095.Google Scholar
Hsu, M., Bhatt, M., et al. (2005). “Neural systems responding to degrees of uncertainty in human decision-making.” Science 310(5754): 1680–1683.Google Scholar
Hsu, M., Krajbich, I., et al. (2009). “Neural Response to Reward Anticipation under Risk Is Nonlinear in Probabilities.” J Neurosci 29(7): 2231–2237.Google Scholar
Ito, M. and Doya, K. (2011). “Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit.” Current Opinion in Neurobiology 21(21531544): 368–373.Google Scholar
Kuhnen, C. and Knutson, B. (2005). “The Neural Basis of Financial Risk Taking.” Neuron 47(5): 763–770.Google Scholar
Kuo, W.-J., Sjostrom, T., et al. (2009). “Intuition and Deliberation: Two Systems for Strategizing in the Brain.” Science 324(5926): 519–522.Google Scholar
Lee, D. (2008). “Game theory and neural basis of social decision making.” Nat Neuro sci 11(4): 404–409.Google Scholar
Lee, D., McGreevy, B. P., et al. (2005). “Learning and decision making in monkeys during a rock-paper-scissors game.” Cognitive Brain Research 25(2): 416–430.Google Scholar
Lohrenz, T., McCabe, K., et al. (2007). “Neural signature of fictive learning signals in a sequential investment task.” PNAS 104(22): 9493–9498.Google Scholar
Maia, T. V. and Frank, M. J. (2011). “From reinforcement learning models to psychiatric and neurological disorders.” Nature Neuroscience 14(2): 154.Google Scholar
McCabe, K., Houser, D., et al. (2001). “A functional imaging study of cooperation in two-person reciprocal exchange.” PNAS 98(20): 11832–11835.Google Scholar
McClure, S. M., Berns, G. S., et al. (2003). “Temporal prediction errors in a passive learning task activate human striatum.” Neuron 38(2): 339–346.Google Scholar
Mega, M. S. and Cummings, J. L. (1994). “Frontal-subcortical circuits and neuropsychiatrie disorders.” Journal of Neuropsychiatry and Clinical Neurosciences 6(4): 358–370.Google Scholar
O'Doherty, J. P. (2004). “Reward representations and reward-related learning in the human brain: insights from neuroimaging.” Current Opinion in Neurobiology 14(6): 769–776.Google Scholar
O'Doherty, J. P., Dayan, P., et al. (2004). “Dissociable roles of ventral and dorsal striatum in instrumental conditioning.” Science 304(5669): 452–454.Google Scholar
Olds, J. and Milner, P. (1954). “Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain.” J Comp Physiol Psychol 47(6): 419–427.Google Scholar
Padoa-Schioppa, C. and Assad, J. (2006). “Neurons in the orbitofrontal cortex encode economic value.” Nature 441(7090): 223–226.Google Scholar
Preuschoff, K., Bossaerts, P., et al. (2006). “Neural differentiation of expected reward and risk in human subcortical structures.” Neuron 51(3): 381–390.Google Scholar
Rangel, A., Camerer, C. F., et al. (2008). “A framework for studying the neurobiology of value-based decision making.” Nat Rev Neurosci.Google Scholar
Rapoport, A. and Amaldoss, W. (2000). “Mixed strategies and iterative elimination of strongly dominated strategies: An experimental investigation of states of knowledge.” Journal of Economic Behavior & Organization 42: 483–521.Google Scholar
Roth, A. and Erev, I. (1995). “Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term.” Games and Economic Behavior 8(1): 164–212.Google Scholar
Salmon, T. (2001). “An Evaluation of Econometric Models of Adaptive Learning.” Econometrica 69(6): 1597–1628.Google Scholar
Sanfey, A. G., Rilling, J. K., et al. (2003). “The neural basis of economic decisionmaking in the Ultimatum Game.” Science 300(5626): 1755–1758.Google Scholar
Saxe, R. (2006). “Uniquely human social cognition.” Current Opinion in Neurobiology 16(2): 235–239.Google Scholar
Saxe, R. and Powell, L. J. (2006). “It's the thought that counts: specific brain regions for one component of theory of mind.” Psychological science 17(8): 692–699.Google Scholar
Schultz, W., Dayan, P., et al. (1997). “A neural substrate of prediction and reward.” Science 275(5306): 1593–1599.Google Scholar
Schultz, W., Preuschoff, P., et al. (2008). “Explicit neural signals reflecting reward uncertainty.” Philos Trans R Soc Lond, B, Biol Sci 363(1511): 3801–3811.Google Scholar
Seeley, T. D. (1995). The wisdom of the hive: the social physiology of honey bee colonies, Harvard Univ Pr.Google Scholar
Shultz, S. and Dunbar, R. (2007). “The evolution of the social brain: anthropoid primates contrast with other vertebrates.” Proceedings of the Royal Society B: Biological Sciences 274(1624): 2429.Google Scholar
Sutton, R. S. and Barto, A. G. (1998). Reinforcement learning, MIT Press.Google Scholar
Tekin, S. and Cummings, J. L. (2002). “Frontal-subcortical neuronal circuits and clinical neuropsychiatry: an update.” Journal of Psychosomatic Research 53(2): 647–654.Google Scholar
Zhu, L., Mathewson, K. E., et al. (2012). “Dissociable Neural Representations of Reinforcement and Belief Prediction Errors underlying Strategic Learning.” PNAS 109(5): 1419–1424.Google Scholar