Book contents
- Evolution of Learning and Memory Mechanisms
- Evolution of Learning and Memory Mechanisms
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Preface
- Introduction
- Part I Evolution of Learning Processes
- 1 Learning and Memory in the Nematode Caenorhabditis elegans
- 2 Adaptive Evolution of Learning and Memory in a Model Lineage
- 3 Learning in Insects: Perspectives and Possibilities
- 4 Experimental Evolution and Mechanisms for Prepared Learning
- 5 Evolutionary Processes Shaping Learning Ability in Insects
- 6 Brain and Spatial Cognition in Amphibians
- 7 Pavlovian Conditioning, Survival, and Reproductive Success
- 8 Compensatory Responses to Wildlife Control
- 9 Relational Memory Functions of the Hippocampal Pallium in Teleost Fish
- 10 Mechanisms Underlying Absolute and Relative Reward Value in Vertebrates
- 11 The Optimality of “Suboptimal” Choice
- 12 A Behavior Systems Framework
- 13 Dissociable Learning Processes
- 14 Social Learning Strategies
- 15 How Learning Affects Evolution
- Part II Evolution of Memory Processes
- Index
- References
11 - The Optimality of “Suboptimal” Choice
A Psycho-evolutionary Perspective
from Part I - Evolution of Learning Processes
Published online by Cambridge University Press: 26 May 2022
Book contents
- Evolution of Learning and Memory Mechanisms
- Evolution of Learning and Memory Mechanisms
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Preface
- Introduction
- Part I Evolution of Learning Processes
- 1 Learning and Memory in the Nematode Caenorhabditis elegans
- 2 Adaptive Evolution of Learning and Memory in a Model Lineage
- 3 Learning in Insects: Perspectives and Possibilities
- 4 Experimental Evolution and Mechanisms for Prepared Learning
- 5 Evolutionary Processes Shaping Learning Ability in Insects
- 6 Brain and Spatial Cognition in Amphibians
- 7 Pavlovian Conditioning, Survival, and Reproductive Success
- 8 Compensatory Responses to Wildlife Control
- 9 Relational Memory Functions of the Hippocampal Pallium in Teleost Fish
- 10 Mechanisms Underlying Absolute and Relative Reward Value in Vertebrates
- 11 The Optimality of “Suboptimal” Choice
- 12 A Behavior Systems Framework
- 13 Dissociable Learning Processes
- 14 Social Learning Strategies
- 15 How Learning Affects Evolution
- Part II Evolution of Memory Processes
- Index
- References
Summary
Incentive salience denotes the biopsychological process that enables reward-related cues to be approached. Accordingly, organisms often prefer a cue predictive of a food reward delivered with certainty over one predictive of the same reward delivered with uncertainty. However, a closer examination of free-choice behavior in various experimental designs suggests that the principle of reward maximization is an oversimplification. In particular, many studies of suboptimal choice (SOC) reveal that organisms exposed to conditioned stimuli may prefer a food option associated with a lower total amount and a lower probability of food to the more profitable alternative option. In this chapter, I argue that SOC illustrates one important fact: reward maximization can be a correlate of choice behavior but it is not its cause. Instead, organisms track the cues that reliably predict food delivery, independent of the amounts received or the probability of being rewarded. I show how to understand this process in psychological terms and also why this view may make more sense than reward maximization from an evolutionary perspective.
- Type
- Chapter
- Information
- Evolution of Learning and Memory Mechanisms , pp. 193 - 209Publisher: Cambridge University PressPrint publication year: 2022
References
Anselme, P. (2015). Incentive salience attribution under reward uncertainty: A Pavlovian model. Behavioural Processes, 111, 6–18. https://doi.org/10.1016/j.beproc.2014.10.016Google Scholar
Anselme, P. (2018). Uncertainty processing in bees exposed to free choices: Lessons from vertebrates. Psychonomic Bulletin & Review, 25, 2024–2036. https://doi.org/10.3758/s13423-018-1441-xGoogle Scholar
Anselme, P., & Güntürkün, O. (2019). How foraging works: Uncertainty magnifies food-seeking motivation. Behavioral and Brain Sciences, 42(e35), 1–59. https://doi.org/10.1017/S0140525X18000948CrossRefGoogle Scholar
Anselme, P., & Robinson, M. J. F. (2019). Evidence for motivational enhancement of sign-tracking behavior under reward uncertainty. Journal of Experimental Psychology: Animal Learning and Cognition, 45, 350–355. https://doi.org/10.1037/xan0000213Google Scholar
Anselme, P., Robinson, M. J. F., & Berridge, K. C. (2013). Reward uncertainty enhances incentive salience attribution as sign-tracking. Behavioural Brain Research, 238, 53–61. https://doi.org/10.1016/j.bbr.2012.10.006CrossRefGoogle ScholarPubMed
Bateson, M., & Kacelnik, A. (1997). Starlings’ preference for predictable and unpredictable delays to food. Animal Behaviour, 53, 1129–1142. https://doi.org/10.1006/anbe.1996.0388Google Scholar
Belovsky, G. E. (1978). Diet optimization in a generalist herbivore: The moose. Theoretical Population Biology, 14, 105–134. https://doi.org/10.1016/0040-5809(78)90007-2Google Scholar
Berridge, K. C. (2007). The debate over dopamine’s role in reward: The case for incentive salience. Psychopharmacology, 191, 391–431. https://doi.org/10.1007/s00213-006-0578-xCrossRefGoogle ScholarPubMed
Berridge, K. C. (2012). From prediction error to incentive salience: Mesolimbic computation of reward motivation. European Journal of Neuroscience, 35, 1124–1143. https://doi.org/10.1111/j.1460-9568.2012.07990.xCrossRefGoogle ScholarPubMed
Bindra, D. (1978). How adaptive behavior is produced: A perceptual-motivational alternative to response-reinforcement. Behavioural and Brain Sciences, 1, 41–91.Google Scholar
Blaiss, C. A., & Janak, P. H. (2009). The nucleus accumbens core and shell are critical for the expression, but not the consolidation, of Pavlovian conditioned approach. Behavioural Brain Research, 200, 22–32. https://doi.org/10.1016/j.bbr.2008.12.024Google Scholar
Breland, K., & Breland, M. (1961). The misbehavior of organisms. American Psychologist, 16, 681–684. https://psycnet.apa.org/doi/10.1037/h0040090Google Scholar
Brodin, A. (2007). Theoretical models of adaptive energy management in small wintering birds. Philosophical Transactions of the Royal Society B: Biological Sciences, 362, 1857–1871. https://doi.org/10.1098/rstb.2006.1812Google Scholar
Cabanac, M. (1992). Pleasure: The common currency. Journal of Theoretical Biology, 155, 173–200. https://doi.org/10.1016/S0022-5193(05)80594-6CrossRefGoogle ScholarPubMed
Case, J. P., & Zentall, T. R. (2018). Suboptimal choice in pigeons: Does the predictive value of the conditioned reinforcer alone determine choice? Behavioural Processes, 157, 320–326. https://doi.org/10.1016/j.beproc.2018.07.018Google Scholar
Chow, J. J., Smith, A. P., Wilson, A. G., Zentall, T. R., & Beckmann, J. S. (2017). Suboptimal choice in rats: Incentive salience attribution promotes maladative decision-making. Behavioural Brain Research, 320, 244–254. https://doi.org/10.1016/j.bbr.2016.12.013CrossRefGoogle Scholar
Cunningham, P. J., & Shahan, T. A. (2019). Rats engage in suboptimal choice when the delay to food is sufficiently long. Journal of Experimental Psychology: Animal Learning and Cognition, 45, 301–310. https://doi.org/10.1037/xan0000211Google ScholarPubMed
Domjan, M., O’Vary, D., & Greene, P. (1988). Conditioning of appetitive and consummatory sexual behavior in male Japanese quail. Journal of the Experimental Analysis of Behavior, 50, 505–519. https://doi.org/10.1901/jeab.1988.50-505Google Scholar
Dunn, R., & Spetch, M. L. (1990). Choice with uncertain outcomes: Conditioned reinforcement effects. Journal of the Experimental Analysis of Behavior, 53, 201–218. https://doi.org/10.1901/jeab.1990.53-201Google Scholar
Eisenreich, B. R., Hayden, B. Y., & Zimmermann, J. (2019). Macaques are risk-averse in a freely moving foraging task. Scientific Reports, 9, 15091. https://doi.org/10.1038/s41598-019-51442-zCrossRefGoogle Scholar
Fantino, E. (1969). Choice and rate of reinforcement. Journal of the Experimental Analysis of Behavior, 12, 723–730. https://doi.org/10.1901/jeab.1969.12-723CrossRefGoogle ScholarPubMed
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898–1902. https://doi.org/10.1126/science.1077349CrossRefGoogle ScholarPubMed
Flagel, S. B., Clark, J. J., Robinson, T. E., Mayo, L., Czuj, A., Willuhn, I., Akers, C. A., Clinton, S. M., Phillips, P. E. M., & Akil, H. (2011). A selective role for dopamine in stimulus-reward learning. Nature, 469, 53–57. https://doi.org/10.1038/nature09588CrossRefGoogle ScholarPubMed
Fortes, I., Vasconcelos, M., & Machado, A. (2016). Testing the boundaries of “paradoxical” predictions: Pigeons do disregard bad news. Journal of Experimental Psychology: Animal Learning and Cognition, 42, 336–346. https://doi.org/10.1037/xan0000114Google Scholar
Fuentes-Verdugo, E., Pellón, R., Papini, M. R., Torres, C., Fernández-Teruel, A., & Anselme, P. (2020). Effects of partial reinforcement on autoshaping in inbred Roman high- and low-avoidance rats. Physiology and Behavior, 225, 113111. https://doi.org/10.1016/j.physbeh.2020.113111CrossRefGoogle ScholarPubMed
Glueck, A. C., Torres, C., & Papini, M. R. (2018). Transfer between anticipatory and consummatory tasks involving reward loss. Learning and Motivation, 63, 105–125. https://doi.org/10.1016/j.lmot.2018.05.001Google Scholar
Gneezy, U., List, J. A., & Wu, G. (2006). The uncertainty effect: When a risky prospect is valued less than its worst possible outcome. Quarterly Journal of Economics, 121, 1283–1309. https://doi.org/10.1093/qje/121.4.1283Google Scholar
González, V.V., Macías, A., Machado, A., & Vasconcelos, M. (2020). The Δ–Σ hypothesis: How contrast and reinforcement rate combine to generate suboptimal choice. Journal of the Experimental Analysis of Behavior, 113, 591–608. https://doi.org/10.1002/jeab.595CrossRefGoogle Scholar
Gray, J. A., & McNaughton, N. (2000). The neuropsychology of anxiety (2nd ed.). Oxford University Press.Google Scholar
Hart, A. S., Clark, J. J., & Phillips, P. E. M. (2015). Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning. Neurobiology of Learning and Memory, 117, 84–92. https://doi.org/10.1016/j.nlm.2014.07.010Google Scholar
Hayden, B. Y., Heilbronner, S. R., Nair, A. C., & Platt, M. L. (2008). Cognitive influences on risk-seeking by rhesus macaques. Judgment and Decision Making, 3, 389–395.Google Scholar
Hearst, E., & Jenkins, H. M. (1974). Sign-tracking: The stimulus-reinforcer relation and directed action. Austin: Psychonomic Society.Google Scholar
Hinnenkamp, J. E., Shahan, T. A., & Madden, G. J. (2017). How suboptimal is suboptimal choice? Journal of the Experimental Analysis of Behavior, 107, 136–150. https://doi.org/10.1002/jeab.239Google Scholar
Inglis, I. R., Forkman, B., & Lazarus, J. (1997). Free food or earned food? A review and fuzzy model of contrafreeloading. Animal Behaviour, 53, 1171–1191. https://doi.org/10.1006/anbe.1996.0320Google Scholar
Johnson, P. S., Madden, G. J., Brewer, A. T., Pinkston, J. W., & Fowler, S. C. (2011). Effects of acute pramipexole on preference for gambling-like schedules of reinforcement in rats. Psychopharmacology, 213, 11–18. https://doi.org/10.1007/s00213-010-2006-5Google Scholar
de Jonge, F. H., Ooms, M., Kuurman, W. W., Maes, J. H. R., & Spruijt, B. M. (2008). Are pigs sensitive to variability in food rewards? Applied Animal Behaviour Science, 114, 93–104. https://doi.org/10.1016/j.applanim.2008.01.004Google Scholar
Kacelnik, A., & Bateson, M. (1996). Risky theories: The effects of variance on foraging decisions. American Zoologist, 36, 402–434. https://doi.org/10.1093/icb/36.4.402Google Scholar
Kacelnik, A., & Mouden, C. E. (2013). Triumphs and trials of the risk paradigm. Animal Behaviour, 86, 1117–1129. https://doi.org/10.1016/j.anbehav.2013.09.034CrossRefGoogle Scholar
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–292. https://doi.org/10.1142/9789814417358_0006Google Scholar
Kawasaki, K., Annicchiarico, I., Glueck, A. C., Morón, I., & Papini, M. R. (2017). Reward loss and the basolateral amygdala: A function in reward comparisons. Behavioural Brain Research, 331, 205–213. https://doi.org/10.1016/j.bbr.2017.05.036Google Scholar
Kendall, S. B. (1974). Preference for intermittent reinforcement. Journal of the Experimental Analysis of Behavior, 21, 463–473. https://doi.org/10.1901/jeab.1974.21-463Google Scholar
Laude, J. R., Stagner, J. P., & Zentall, T. R. (2014). Suboptimal choice by pigeons may result from the diminishing effect of nonreinforcement. Journal of Experimental Psychology: Animal Learning and Cognition, 40, 12–21. https://doi.org/10.1037/xan0000010Google ScholarPubMed
Lima, S. L. (1986). Predation risk and unpredictable feeding conditions: Determinants of body mass in birds. Ecology, 67, 377–385. https://doi.org/10.2307/1938580Google Scholar
Martinez, M., Alba, R., Rodriguez, W., & Orduña, V. (2017). Incentive salience attribution is not the sole determinant of suboptimal choice in rats: Conditioned inhibition matters. Behavioural Processes, 142, 99–105. https://doi.org/10.1016/j.beproc.2017.06.012Google Scholar
Mascia, P., Neugebauer, N. M., Brown, J., Bubula, N., Nesbitt, K. M., Kennedy, R. T., & Vezina, P. (2019). Exposure to conditions of uncertainty promotes the pursuit of amphetamine. Neuropsychopharmacology 44, 274–280. https://doi.org/10.1038/s41386-018-0099-4Google Scholar
Mazur, J. E. (1987). An adjusting procedure for studying delayed reinforcement. In: Commons, M. L., Mazur, J. E., Nevin, J. A., & Rachlin, H. (Eds.), Quantitative analyses of behavior (Vol. 5). The effect of delay and of intervening events on reinforcement value (pp. 55–73). Erlbaum Associates.Google Scholar
McDevitt, M. A., Dunn, R. M., Spetch, M. L., & Ludvig, E. A. (2016). When good news leads to bad choices. Journal of the Experimental Analysis of Behavior, 105, 23–40. https://doi.org/10.1002/jeab.192Google Scholar
McDevitt, M. A., Spetch, M. L., & Dunn, R. (1997). Contiguity and conditioned reinforcement in probabilistic choice. Journal of the Experimental Analysis of Behavior, 68, 317–327. https://doi.org/10.1901/jeab.1997.68-317Google Scholar
Oinio, V., Sundström, M., Bäckström, P., Uhari-Väänänen, J., Kiianmaa, K., Raasmaja, A., & Piepponen, P. (2018). Amphetamine primes enhanced motivation toward uncertain choices in rats with genetic alcohol preference. Psychopharmacology, 235, 1361–1370. https://doi.org/10.1007/s00213-018-4847-2Google Scholar
Papini, M. R. (2014). Diversity of adjustments to reward downshifts in vertebrates. International Journal of Comparative Psychology, 27, 420–445.CrossRefGoogle Scholar
Pisklak, J. M., McDevitt, M. A., & Dunn, R. M. (2019). Clarifying contrast, acknowledging the past, and expanding the focus. Comparative Cognition and Behavior Reviews, 14, 33–38. https://doi.org/10.3819/CCBR.2019.140004Google Scholar
Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Black, A. H. & Prokasy, W. F. (Eds.), Classical conditioning II: Current theory and research (pp. 64–99). Appleton-Century-Crofts.Google Scholar
Robinson, M. J. F., Anselme, P., Fischer, A. M., & Berridge, K. C. (2014). Initial uncertainty in Pavlovian reward prediction persistently elevates incentive salience and extends sign-tracking to normally unattractive cues. Behavioural Brain Research, 266, 119–130. https://doi.org/10.1016/j.bbr.2014.03.004Google Scholar
Robinson, M. J. F., & Berridge, K. C. (2013). Instant transformation of learned repulsion into motivational “wanting.” Current Biology, 23, 282–289. https://doi.org/10.1016/j.cub.2013.01.016Google Scholar
Robinson, M. J. F., Clibanoff, C., Freeland, C. M., Knes, A. S., Cote, J. R., & Russell, T. I. (2019). Distinguishing between predictive and incentive value of uncertain gambling-like cues in a Pavlovian autoshaping task. Behavioural Brain Research, 371, 111971. https://doi.org/10.1016/j.bbr.2019.111971Google Scholar
Robinson, T. E., & Berridge, K. C. (1993). The neural basis of drug craving: An incentive-sensitization theory of addiction. Brain Research Review, 18, 247–291. https://doi.org/10.1016/0165-0173(93)90013-PCrossRefGoogle ScholarPubMed
Rosse, R. B., Fay-McCarthy, M., Collins, J. P., Jr., Risher-Flowers, D., Alim, T. N., & Deutsch, S. I. (1993). Transient compulsive foraging behavior associated with crack cocaine use. American Journal of Psychiatry, 150, 155–156. http://dx.doi.org/10.1176/ajp.150.1.155Google Scholar
Saunders, B. T., Richard, J. M., Margolis, E. B., & Janak, P. H. (2018). Dopamine neurons create Pavlovian conditioned stimuli with circuit-defined motivational properties. Nature Neuroscience, 21, 1072–1083. https://doi.org/10.1038/s41593-018-0191-4Google Scholar
Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1–27. https://doi.org/10.1152/jn.1998.80.1.1Google Scholar
Simonsohn, U. (2009). Direct risk aversion: Evidence from risky prospects valued below their worst outcome. Psychological Science, 20, 686–692. https://doi.org/10.1111/j.1467-9280.2009.02349.xGoogle Scholar
Smith, A. P., & Zentall, T. R. (2016). Suboptimal choice in pigeons: Choice is primarily based on the value of the conditioned reinforcers rather than overall reinforcement rate. Journal of Experimental Psychology: Animal Learning and Cognition, 42, 212–220. https://doi.org/10.1037/xan0000092Google Scholar
Spetch, M., Belke, T., Barnet, R., Dunn, R., & Pierce, W. (1990). Suboptimal choice in a percentage reinforcement procedure: Effects of signal condition and terminal-link length. Journal of the Experimental Analysis of Behavior, 53, 219–234. https://doi.org/https://doi.org/10.1901/jeab.1990.53-219Google Scholar
Spetch, M., Mondloch, M., Belke, T., & Dunn, R. (1994). Determinants of pigeons’ choice between certain and probabilistic outcomes. Animal Learning & Behavior, 22, 239–251. https://doi.org/10.3758/BF03209832Google Scholar
Stagner, J. P., & Zentall, T. R. (2010). Suboptimal choice behavior by pigeons. Psychonomic Bulletin & Review, 17, 412–416. http://dx.doi.org/10.3758/PBR.17.3.412Google Scholar
Stout, S. C., Boughner, R. L., & Papini, M. R. (2003). Reexamining the frustration effect in rats: Aftereffects of surprising reinforcement and nonreinforcement. Learning and Motivation, 34, 437–456. https://doi.org/10.1016/S0023-9690(03)00038-9Google Scholar
Tindell, A. J., Berridge, K. C., Zhang, J., Peciña, S., & Aldridge, J. W. (2005). Ventral pallidal neurons code incentive motivation: Amplification by mesolimbic sensitization and amphetamine. European Journal of Neuroscience, 22, 2617–2634. https://doi.org/10.1111/j.1460-9568.2005.04411.xCrossRefGoogle ScholarPubMed
Tinklepaugh, O. L. (1928). An experimental study of representative factors in monkeys. Journal of Comparative Psychology, 8, 197–236. https://psycnet.apa.org/doi/10.1037/h0075798Google Scholar
Torres, C., Glueck, A. C., Conrad, S. E., Morón, I., & Papini, M. R. (2016). Dorsomedial striatum lesions affect adjustment to reward uncertainty, but not to reward devaluation or omission. Neuroscience, 332, 13–25. http://dx.doi.org/10.1016/j.neuroscience.2016.06.041CrossRefGoogle ScholarPubMed
Tremblay, M., Silveira, M. M., Kaur, S., Hosking, J. G., Adams, W. K., Baunez, C., & Winstanley, C. A. (2017). Chronic D2/3 agonist ropinirole treatment increases preference for uncertainty in rats regardless of baseline choice patterns. European Journal of Neuroscience, 45, 159–166. https://doi.org/10.1111/ejn.13332Google Scholar
Trujano, R. E., & Orduña, V. (2015). Rats are optimal in a choice task in which pigeons are not. Behavioural Processes, 119, 22–27. https://doi.org/10.1016/j.beproc.2015.07.010CrossRefGoogle Scholar
Vasconcelos, M., Machado, A., & Pandeirada, J. N. S. (2018). Ultimate explanations and suboptimal choice. Behavioural Processes, 152, 63–72. https://doi.org/10.1016/j.beproc.2018.03.023CrossRefGoogle ScholarPubMed
Vasconcelos, M., Monteiro, T., & Kacelnik, A. (2015). Irrational choice and the value of information. Scientific Reports, 5, 13874. https://doi.org/10.1038/srep13874Google Scholar
Witter, M. S., & Cuthill, I. C. (1993). The ecological costs of avian fat storage. Philosophical Transactions of the Royal Society B: Biological Sciences, 340, 73–92. https://doi.org/10.1098/rstb.1993.0050Google Scholar
Zeeb, D. F., Li, Z., Fisher, D. C., Zack, M. H., & Fletcher, P. J. (2017). Uncertainty exposure causes behavioural sensitization and increases risky decision-making in male rats: Toward modelling gambling disorder. Journal of Psychiatry Neuroscience, 42, 404–413. https://doi.org/10.1503%2Fjpn.170003Google Scholar
Zentall, T. R. (2016). Resolving the paradox of suboptimal choice. Journal of Experimental Psychology: Animal Learning and Cognition, 42, 1–14. https://doi.org/10.1037/xan0000085Google Scholar
Zentall, T. R., Andrews, D. M., & Case, J. (2019). Contrast between what is expected and what occurs increases pigeon’s suboptimal choice. Animal Cognition, 22, 81–87. https://doi.org/10.1007/s10071-018-1223-xCrossRefGoogle ScholarPubMed
- 2
- Cited by