2 - Bounded Rationality

Summary

It is surprising, and perhaps a reflection of a certain provincialism in philosophy, that the problem of induction is so seldom linked to learning. On the face of it, an animal in a changing environment faces problems no different in general principle from those that we as ordinary humans or as specialized scientists face in trying to make predictions about the future.

Patrick Suppes Learning and Projectibility

This chapter applies the ideas developed in the preceding chapter to a class of bounded resource learning procedures known as payoff-based models. Payoff-based models are alternatives to classical Bayesian models that reduce the complexity of a learning situation by disregarding information about states of the world. I am going to focus on one particular payoff-based model, the “basic model of reinforcement learning,” which captures in a precise and mathematically elegant way the idea that acts which are deemed more successful (according to some specific criterion) are more likely to be chosen.

What we are going to see is that the basicmodel can be derived from certain symmetry principles, analogous to the derivation of Carnap's family of inductive methods. Studying the symmetries involved in this derivation leads into a corner of decision theory that is relatively unknown in philosophy. Duncan Luce, in the late 1950s, introduced a thoroughly probabilistic theory of individual choice behavior in which preferences are replaced by choice probabilities. A basic constraint on choice probabilities, known as “Luce's choice axiom,” together with the theory of commutative learning operators, provides us with the fundamental principles governing the basic model of reinforcement learning.

Our exploration of the basic model does not, of course, exhaust the study of payoff-based and other learning models. I indicate some other possible models throughout the chapter and in the appendices. The main conclusion is that learning procedures that stay within a broadly probabilistic framework often arise from symmetry principles in a way that is analogous to Bayesian models.