Behavioral proxies compete by the time courses of their rewards, including endogenous rewards

Abstract

Natural selection is slow, so behavioral goals must be based on patterns of reward. Addictions are rewarded in the same way as adaptive choice, so they can be distinguished only by their time course. In addition, the reward process is more plastic than is generally recognized, so abstract goals are shaped by the “legibility” of their proxies.

The authors’ distinction between goal and proxy is especially useful in “preference learning” (target article, Table 1, sect. 4.1), where our understanding of selection by reward is in flux: (1) Given the necessarily slow pace of natural selection, the effective basis of behavioral goals must be reward. (2) “Addiction and other maladaptive habits” (target article, sect. 4.1, para. 1) depend on the same neural “behavior prioritization” (target article, sect. 4.1, para. 4) as adaptive choice, and can be distinguished only by their time course. (3) The reward process is more plastic than is generally recognized, so abstract goals are shaped by the “legibility” of their proxies.

1. (1) The authors depict the brain's goal as “the organism's fitness, utility, or wellbeing” (target article, sect. 4.1, para. 2) – evolutionary adaptiveness. But adaptiveness inevitably diverges from rewardingness. The action-selecting reward mechanism has clearly been shaped by natural selection, but the rewards it specifies become outdated when environmental circumstances change, as described in Box 2 in the target article. In the case of drug addictions, evolution is probably “trying” to correct the readiness of the reward mechanism to be hijacked by dopaminergic proxies – for instance, when it selects a gene that reduces the attraction of alcohol in some populations (Agarwal & Goedde, 1989) – but any such correction will take generations to occur. In the meantime, humans perceive welfare as a pattern of reward, and its connection with adaptiveness is purely historic. The connection may even be adversarial, as when birth control increases welfare but reduces numbers of offspring. In the time spans of actual lives, stable goals must be shaped by patterns of reward. As the authors say, “human goals … can be any arbitrary notion of, e.g., utility or wellbeing…” (target article, Box 2).

2. (2) The authors cite recent brain imaging that has confirmed common-currency theories of how motives compete to determine choice (Coddington & Dudman, 2019; Levy & Glimcher, 2012), which clear the way for a reward-based economy of all mental life (Silver, Singh, Precup, & Sutton, 2021). However, the target article describes motives’ operation only for the case of addiction, where a misguided reward signal leads to proxy failure (target article, sect. 4.1): This jibes with a now-standard account of drug addiction (Volkow, Wise, & Baler, 2017), but leaves unclear how the addiction case should be distinguished from routine reward-based goal setting. I propose that addictive proxies should be called hijackers when they lead to preferences that are only temporary; if they were stable they could not be distinguished from normal goals except by an eventual negative effect on evolutionary fitness (a view attributed to Becker & Murphy, 1988, target article, Box 2). A goal in the target article's terms would be an objective that survives temporary preferences.

   Two mechanisms for temporary preference have been widely proposed: (a) The discounting of future outcomes in hyperbolic or similar curve (Green & Myerson, 2018) such that nearby events are overvalued; and/or (b) a temporary burst of rewarding power (Loewenstein, O'Donoghue, & Bhatia, 2015), for instance by emotional arousal or a drug. (a) Hyperbolic discounting is a robust finding both in visceral realms (as with nonhumans and young children) and deliberative activities (as in planning for retirement or dealing with global warming). However, when reported by subjects for the valuation of distant events, discounting raises the unexplored question of how quantitative expectations of such events are formed (Ainslie, 2023, pp. 19–22; Rick & Loewenstein, 2008). (b) Most rewards are enabled by appetite, the presence of which is assumed when people evaluate them at a distance. You may not be willing to pay more for a food if shopping while hungry. But some portion of future appetites seems not to be anticipated, especially when considering options that would be preferred only temporarily (Ariely & Loewenstein, 2006; Badger et al., 2007). Perhaps people avoid considering some appetites so as not to arouse them. The partial neglect of future-aroused appetite, which makes a reward curve spike upward in its presence, is also an effect that needs exploration. In any case, some mechanism of temporary preference is necessary to distinguish addictive proxies from stable goals.

3. (3) “Many abstract goals cannot be observed directly” (target article, sect. 3.2) – likewise distant future goals – so they invite the creation of proxies. Such creation accords with many authors’ suspicion that belief is a reward-seeking activity, and even with behaviorist Howard Rachlin's radical proposal that “mental states (including sensations, perceptions, beliefs, knowledge, even pain) are… patterns of overt behavior” (2012, pp. 3–4). That is, beliefs are incentivized by their capacity to produce reward, which may happen entirely within the agent's imagination. Freed from the need for predicting external rewards, proxies may compete like fiat currencies, needing only to be protected from “inflation” by their unique “legibility” (target article, sects. 3.2 and 3.3), which thus overshadows their “ability to predict the future” as their main claim to selection.

   A proxy that is a good story may turn into a goal in its own right. It may take the form of a proposition: Eating this food – or avoiding it – is morally good; there is a conspiracy to corrupt our children; Matisse produced the painting I bought. But a proxy that does not predict facts must survive through avoiding inflation by not just legibility but also singularity – standing out from alternative legible proxies by such factors as a unique logical argument, a parsimonious explanation, a rare coincidence, or endorsement by an authority (discussed in Ainslie, 2013, 2017, pp. 178–184, 2023, pp. 19–22). This property is important not just to psychology, but also to recent proposals that economics recognize the utility of abstract goals (Bénabou & Tirole, 2016; Loewenstein & Molnar, 2018). The self-selecting potential of proxies in neuroscience sets them apart from the other kinds of proxy the authors describe.