1 Introduction
The strategy method (SM), an increasingly popular way to estimate preferences, consists of asking participants to indicate their choices at all information sets rather than only those actually reached. One then compares the differences in decisions at different information sets. For example, to identify the effect of a low offer in an ultimatum game, one might compare the changes in decisions for the low-offer information set with the decisions for the high-offer information set. The appeal of SM comes from its simplicity as well as its potential to elucidate the equilibria that are actually played when theoretical models indicate there are multiple equilibria. SM also has the potential to circumvent many of the endogeneity problems that arise in estimating preferences when making comparisons between heterogeneous individuals.
SM is a straightforward yet powerful tool used in economics research that involves requesting participants to make choices at all possible information sets, rather than exclusively at the ones reached. By comparing the variations in decisions across different information sets, researchers can gain valuable insights. For instance, in an ultimatum game, assessing the effects of a low offer can be achieved by contrasting the decisions made in low-offer and high-offer scenarios. SM’s appeal lies in its simplicity and its capacity to reveal the actual equilibria played when theoretical models suggest multiple possibilities. Additionally, SM can help overcome endogeneity issues that emerge when estimating preferences in comparisons between diverse individuals.
However, SM has its limitations and can yield different inferences than data collected using direct elicitation (DE) (Brandts and Charness Reference Brandts and Charness2000, Reference Brandts and Charness2011; García-Pola et al. Reference García-Pola, Iriberri and Kovářík2020; Chen and Schonger Reference Chen and Schonger2023). The open question is why and under which conditions the methods are not unambiguously equivalent. We argue that when the payoffs of the game played with SM are an affine transformation of the payoffs at the induced terminal nodes in the game played with DE, the two games are strategically equivalent, and the game played with SM essentially coincides with the strategic form of the game played with DE. Since this condition might not hold, SM is subject to a possibly severe economic–theoretical bias. A large body of economic theory renders differences in information sets in SM and DE. The information set for a DE decision node is not the same information set for the same decision node in SM. While economic theory of off-equilibrium motivations is frequently modeled, it is implicitly assumed away by researchers using SM. Three factors make off-equilibrium motivations an especially important issue in the SM context. First, SM usually relies on many decisions at different information sets. Second, the most commonly used dependent variables in SM are typically highly related. Third, and this is an intrinsic aspect of SM, the off-equilibrium decisions can affect the utility of decisions at different information sets, even when they do not affect the monetary payoff. These three factors reinforce each other so that, relative to DE, SM for treatment effects could be severely biased.
Motivations that are based on disappointment aversion (Gul, Reference Gul1991), intentions (Battigalli et al., Reference Battigalli, Maggi and Dufwenberg2007; Fehr & Schmidt, Reference Fehr and Schmidt2000), self-image (Bénabou & Tirole, Reference Bénabou and Tirole2011), identity (Bullock, Reference Bullock and Lenz2019), emotions (Elster, Reference Elster1998; Loewenstein, Reference Loewenstein2000), or duty (Chen & Schonger, Reference Chen and Schonger2022), for instance, can cause equilibrium outcomes to differ between SM and DE. We provide a formal, general framework that embeds prior non-formal (psychological).Footnote 1 explanations for differences between SM and DE to show that these explanations only hold under certain conditions.Footnote 2
Another theoretical critique of SM is that the invariance of equilibrium outcomes relies on individuals eliminating weakly dominated strategies. Game theorists may disagree about the actual prevalence in the field of individuals who play weakly dominated strategies, either because eliminating them requires a greater level of cognition or because they may simply be more credible than those eliminated through subgame perfection. Our critique is an independent one. Our theoretical results focus on motivations (preferences) rather than deviations from rationality in decision-making. We provide a model-based elaboration that complements a footnote by Roth (Reference Roth, Kagel and Roth1995, fn 84) that “the notion of subgame perfect equilibrium is lost in the transition from the extensive to the strategic form of the game, since there are no subgames in a game in which players state their strategies simultaneously.” We cite a theorem from Moulin (Reference Moulin1986, pp. 84–86) and Rochet (Reference Rochet1981) that would prima facie invalidate SM.
Dozens of experimental studies have investigated whether SM yields the same responses as DE where participants actually play the extensive form game. Though a recent study concluded in favor of using SM, it reported statistically significant and economically important differences in behavior by elicitation method in a considerable fraction of the studies comparing the two elicitation methods: “We do find, however, that a particular aspect of emotion-related behavior, the use of punishment, is significantly more likely in situations with direct response than with strategy choice.” Brandts and Charness (Reference Brandts and Charness2011, pg. 394) In our reading, the set of games it studied divide into two: in simple games that had moral content,Footnote 3 SM and DE tend to diverge, while in more complex games that were framed as economics games,Footnote 4 SM and DE did not diverge or had mixed results. This difference is consistent with the heightened relevance of off-equilibrium considerations in social preference games. García-Pola et al. (Reference García-Pola, Iriberri and Kovářík2020) test SM vs. DE for four centipede games between two players.Footnote 5 They run two centipede games where the incentives of each player are symmetric and two centipede games where the incentives are asymmetric. In the first two centipede games, they find that SM and DE diverges, in particular, SM seems to yield results that are more cooperative (stopping toward the end of the centipede game) whereas DE yields results that are less cooperative (stopping at the beginning). In the second two centipede games, they find that SM and DE yield similar findings. One interpretation of this difference is that the symmetry in payoffs for the two players allowed them to think more about why the player was moving along the centipede in the strategy method. That is, the off-equilibrium outcomes were more salient for the players because of the symmetry. Schotter et al. (Reference Schotter, Weigelt and Wilson1994) presents games in extensive vs. normal form and finds that differences emerge in the simplest games, where subjects are more likely to use and fear incredible threats. This is consistent with our reading of the previous literature and the interpretation whose formalization we present here.
2 Theoretical background: direct elicitation vs. strategy method
In an experiment, the observable vector is that of monetary payoffs and may not capture what utility players might get from feelings, such as revenge, gratitude, kindness, or warm-glow. But even from a purely theoretical perspective, the Kohlberg–Mertens view is not universally accepted. Harsanyi (Reference Harsanyi1988) disagree and argue that in general the solution of a game with a sequential structure simply has to depend on this sequential structure and cannot be made dependent on the normal form only. We show that even if one accepts the Kohlberg and Mertens (Reference Kohlberg and Mertens1986) view, it cannot be used as a justification for the strategy method of elicitation without further assumptions. The reason, in short, is that researchers neither observe the preferences nor the players’ conception of the game, and there are plausible circumstances where use of SM rather than DE can change players’ conception of the game.
The upshot, in our view, is not to check theoretically which motivations break invariance in every circumstances, since the number of potential motivations is large. For instance, common theoretical motivations like intentions (Battigalli et al., Reference Battigalli, Maggi and Dufwenberg2007; Fehr & Schmidt, Reference Fehr and Schmidt2000), disappointment aversion (Gul, Reference Gul1991), and self-image (Bénabou & Tirole, Reference Bénabou and Tirole2011), to name a few, can cause divergence, but the parameters in the player’s utility function are also unobserved. Rather, off-equilibrium considerations accepted by formal theorists and by experimentalists can intuitively break invariance between SM and DE as we illustrate theoretically in the appendix.
2.1 Linking theory to data
We can visualize the assumption behind experiments that rely on the invariance between SM and DE using the simplified ultimatum game. Under DE:
Under SM:
Even more concretely, the following simplified 0–50 ultimatum game illustrates how non-consequentialist motivations can breakdown the invariance between SM and DE when collecting data. Suppose player 2 has duty motives: If he did not commit or if he did not, in fact, accept the unfair offer, he gets an additional psychic benefit of . In the DE setting, if player 2 is offered 25 and he accepts, the utilities are . If player 2 is offered 10 and he accepts, the utilities are (40, 10).
In the SM setting, the strategy, accept , yields: (p is the subjective belief of the responder on the choice of the proposer), while the strategy, accept , yields: . Then player 2 picks the strategy, accept , if and only if . That is, player 2 only accepts high offers if and only if there is low probability of bearing the adverse consequences of indulging in the psychic benefit of not being a loser. If , then the DE setting yields payoffs (40, 10) while the SM setting yields payoffs (25, 25).Footnote 6
3 Conclusion
Our study suggests that, because of off-equilibrium motivations, conventional SM estimates may be biased, leading to misleading treatment effects relative to DE. We show that when the payoffs of the game played with SM are an affine transformation of the payoffs at the induced terminal nodes in the game played with DE, the two games are strategically equivalent, and the game played with SM essentially coincides with the strategic form of the game played with DE. However, since a large fraction of SM papers rely on many decisions at different information sets that are typically highly related, the off-equilibrium decisions can affect the utility of decisions at different information sets, even when they do not affect the monetary payoff. These factors are mutually reinforcing so that the SM for treatment effects could be biased. Theoretically, SM may be positively or negatively biased away from DE depending on how utility interacts with decisions at other information sets. Chen and Schonger (Reference Chen and Schonger2023) demonstrate that SM is prone to substantial biases, which may be as significant as other observed treatment effects. Additionally, minor manipulations to salience can intensify these discrepancies to a similar extent. Notably, the direction of treatment impacts can vary greatly between SM and direct effects (DE) and may even reverse in sign.
We have illustrated variance of equilibrium in SM vs. DE with simple models of off-equilibrium motivations. Differences between DE and SM can reveal the importance of motivations beyond strong consequentialist ones. An alternative to the view of natural field experiments (a subset of DE) as the gold standard for causal estimates (Harrison & List, Reference Harrison and List2004; Levitt & List, Reference Levitt and List2007) is that differences between SM and DE can be used to understand the general way in which agents’ motivations influence behavior (Camerer, Reference Camerer2011). To be sure, another reason to use SM may be if the situation approximates natural decision-making. However, if DE is the gold standard, one possible solution for experiments is to consider a pilot that first tests whether SM and DE diverges before collecting additional data using SM.
The closest economic analog to our argument in the field may be the drafting of a contract (Battigalli & Maggi, Reference Battigalli and Maggi2002; Tirole, Reference Tirole1999; Schwartz & Watson, Reference Schwartz and Watson2004). Contemplation of all possible contingencies involves SM decision-making, while the actual decision when the information set is revealed involves DE decision-making. Differences in decision-making provide another reason, besides incentive compatibility, why agents might not have incentives aligned with principals. Legal doctrine has neglected this dimension of contractual capacity.
Appendix A: theory
A.1. Background and history
The earliest use of a “strategy method” can be found in Selten (Reference Selten1967), where subjects are asked to give a strategy for the entire game instead of being asked only for decisions and information sets that are actually reached. As Roth (Reference Roth, Kagel and Roth1995) points out, Selten’ s strategy method first lets participants gain practice by playing the game several times, only then asking them for strategies. In addition, Selten uses group discussions and individual advising of participants by the experimenter to help subjects formulate strategies in what are rather complex games. In comparison to Selten’s games, the games used in more recent studies tend to be much simpler, typically two-player games where each player has only one move. In these recent studies, there is no group discussion or individual advising. Thus, the currently used strategy method is the same as Selten’ s except for pre-game practice and the group discussion and advising (for an early example, see Mitzkewitz and Nagel, Reference Mitzkewitz and Nagel1993b). In both Selten’ s SM and the modern SM, subjects are made aware of an extensive game, but instead of actually playing it, they are asked for their (hypothetical) decision at every decision node. Typically the game is not represented in strategic (i.e., matrix) form (for an exception see Schotter et al. (Reference Schotter, Weigelt and Wilson1994)). SM contrasts with DE (also referred to as direct response method) where players are only asked for their decisions at information sets that are actually reached. We follow convention and sometimes refer to SM as the cold, and DE as the hot setting.
Formally, the games played in SM and in DE can both be represented by an extensive form game. The extensive form games differ, but the corresponding normal form is the same for both methods. In that sense, they are theoretically equivalent. There is the view that for rational players the strategic form captures all relevant information, while different corresponding extensive forms differ only in irrelevant representation. Kohlberg and Mertens (Reference Kohlberg and Mertens1986, p. 1011) put this view nicely by writing, “In some sense, the fact that the reduced normal form captures all the relevant information for decision purposes results directly from the (almost tautological) fact that what matters for decision purposes in an outcome is only the corresponding utility vector (and not, e.g., the particular history leading to that outcome).” Osborne and Rubinstein (Reference Osborne and Rubinstein1994, p. 90) echoes Kohlberg and Mertens (Reference Kohlberg and Mertens1986), “As in the case of a strategic game we often specify the players’ preferences over terminal histories by giving payoff functions that represent the preferences.”
A.2. Failure of invariance
Consider the game in Fig. 1. It is a kind of mini-ultimatum game. Player 1, the proposer, divides an endowment of $4 between herself and player 2, the responder. The offer she makes can be either fair (2, 2) or unfair (3, 1). If both players are purely self-interested, the unique subgame perfect equilibrium is resulting in terminal node c. In the strategic form, shown in the left of Fig. 2, that is the unique strategy profile to survive iterated elimination of weakly dominated strategies; thus, we have invariance.
Let us vary the example and show how and when something like duty, for example, can break this invariance. First, let us incorporate duty in a way which does not break invariance. Assume that whenever the responder has accepted an unfair offer, i.e., responded to U, she suffers a psychic loss worth , where . One can interpret this as damage to her honor (Nisbett, Reference Nisbett and Cohen1996). The unique subgame perfect equilibrium is again resulting in terminal node c, which is also the sole surviving profile of iterated elimination of weakly dominated strategies in the strategic form shown in the third matrix of Fig. 2.
Now assume a twist: The responder not only suffers a physic loss when she has responded to U, but also when she has merely bindingly decided to do so. If the game is directly elicited, there is no opportunity to commit, and the unique subgame perfect equilibrium remains resulting in terminal node c. If the game is elicited via SM, what is played is shown in the rightmost matrix in Fig. 2: Four strategy profiles survive iterated elimination of weakly dominated strategies, and the Nash equilibria among those are as before, but in addition . Why does this happen? Note that both and result in node a. But in the strategic form, they now have different utilities. This means that the reason for failure of invariance is that this strategic from cannot represent a game tree of the form shown in Fig. 1.
Thus, non-consequentialist preferences can generate differential predictions in DE vs. SM settings.Footnote 7 One might call these preferences for duty or see them as at least partially rule-based (i.e., to maintain honor). Note that with these parameters, not all concerns that incorporate off-equilibrium information break invariance in this game. For example, a psychic gain when one has committed to accepting an unfair offer (e.g., a turn-the-other-cheek self-image preference) would not break invariance. Such a responder would behave like homo oeconomicus.
A.3. Invariance example
Next, we provide another example (“tribal game”) where emotions affect decision-making, but invariance in DE vs. SM holds.
In the game in Fig. 3, player 1 sends player 2 a message, where L means that she loves ISIS and H means she hates it. Player 2 has an endowment of $2, and in response can either be kind (K, respectively ) and share equally or selfish and keep all to herself (S, respectively ). Thus, the payoff function of is given by and . If both players are purely self-interested, then the game has two subgame-perfect Nash equilibria and , which yield the terminal nodes b, respectively. d, and payoff (0, 2). Elimination of weakly dominated strategies in the strategic form gives the same equilibria.
Consider a social preference, specifically Fehr–Schmidt preferences for player 2. In this game, regardless of the choice of parameters for advantageous and disadvantageous inequality, Fehr–Schmidt preferences imply that . Thus, the ranking of terminal nodes happens to remain unchanged and the analysis of equilibria is as before.
Now let us construct an example where the social preference changes the equilibria. Consider a very altruistic player 2 with preferences represented by . A functional form from payoff vectors into utility that yields such a preference between terminal nodes would be, for example, where . The game has two subgame-perfect Nash equilibria and , which yield the terminal nodes a, respectively c, and payoff (1, 1). Elimination of weakly dominated strategies in the strategic form gives the same equilibria. Again, there is an invariance between the extensive and strategic forms.
Now let us change the game by changing the preference of player 2 only. Assume that she is an avid ISIS fan, and thus prefers to be kind to someone who also claims to love ISIS, and unkind to someone who does not. Specifically, assume that , which, moreover, means that she prefers to encounter people who profess to be fans. Note that these preferences for player 2 are not a function of payoffs only; though she is not indifferent between a and c. Nevertheless, this extensive game is invariant to the method of elicitation: the unique subgame-perfect equilibrium is yielding the terminal node a with payoff (not utility) (1, 1). In the strategic form, iterated elimination of weakly dominated strategies gives the same equilibrium. Note that this invariance holds even though emotions play a role in player 2’s decisions.
A.4. General proof
In standard game theory, one way to describe an extensive form game with perfect information is by means of a tree , a set of players , the set of nodes T, the decision nodes X, and set of terminal nodes Z, who plays at each decision node , and a complete and transitive preference over the terminal nodes represented by Bernoulli utility functions . Thus let describe our extensive form game. Throughout we shall assume rationality and common knowledge.
Whether implemented in a laboratory or field setting, the preferences over terminal nodes are not directly observable by the researcher. One then typically assigns monetary payoffs to each terminal node, thus implementing a “game” , where assigns player i a payoff at every terminal node. is a game in the game-theoretic sense with an additional assumption that all players’ preferences are purely self-interested and this is common knowledge.
We can denote the DE extensive form game as , with extensive form and the corresponding Bernoulli utility functions . We compare the direct elicitation, and with the strategy method, and .
The design choice of experimenter is , . Let (using the natural order of players), where ext. forms ext. forms and : ( associated with several strategy profiles). By definition of SM, . Note that and are neither a design choice nor directly observable. The following chart summarizes the theorem:
|
|
|
|
|
|
) |
|
|
Strategic equivalence: and are strategically equivalent if and only if for all players i, there exist real numbers such that for all .
Conventional wisdom: The strategic forms of and are strategically equivalent.
Outcome-based preferences: If for all players i, there exists a function such that and , then and are strategically equivalent.
These results follow from Axiom Footnote 1, as formulated by Moulin (Reference Moulin1986, pgs. 84–86):
Axiom 1
(one-to-one) A game satisfies the one-to-one condition if for any terminal nodes and any player i:
If for all .
The theorem below follows the formulation of Moulin (Reference Moulin1986, pgs. 84–86) and Rochet (Reference Rochet1981):
Theorem 1
Let be an N-player game in extensive form with perfect information satisfying the one-to-one assumption. Then the associated normal form of G is solvable by iterated elimination of weakly dominated strategies, and the equilibrium payoffs are the same as obtained in the extensive form by Kuhn’s algorithm.
Theorem Footnote 1 is only applicable if the payoffs given in the game are indeed the Bernoulli utility of the players. But researchers observe the monetary payoffs, but not the Bernoulli utility numbers of the players. Put simply, many motivations commonly modeled and tested in economics research will break invariance and the number of potential motivations is large. We present a few applications to illustrate.
The following two observations extend the applicability of the original theorem. First note that the risk attitude of a player need not be neutral, but can be anything:
Corollary 1
(Risk attitude) Let be an N-player game in extensive form satisfying the one-to-one assumption. Let the domain of preferences be the agent’s payoffs. Let preferences be a strict ordering. Then the normal form of G is solvable by iterated elimination of weakly dominated strategies, and the equilibrium payoffs are the same as obtained in the extensive form by Kuhn’s algorithm.
Corollary Footnote 2 (Social preferences) extends this result to social preferences:
Corollary 2
(Social preferences) Let be an N-player game in extensive form satisfying the one-to-one assumption. Let the domain of preferences be the vector of payoffs. Let preferences be locally non-satiated. Then, the normal form of G is solvable by iterated elimination of weakly dominated strategies, and the equilibrium payoffs are the same as obtained in the extensive form by Kuhn’s algorithm.
A standard response in behavioral economics to inaccurate predictions of the homo oeconomicus model is to assume richer preferences, particularly those that depend not only on the agent’s own monetary payoff but also on the payoffs of others. We say that a player is purely self-interested (homo oeconomicus) if for all terminal nodes a, b, if and only if . Social preferences is where a player’s preference between two nodes is a function of their monetary payoffs only.Footnote 8 Thus, we say that a player has social preferences if for all terminal nodes a, b, if then .
This subsection discusses whether such preferences can generate differential predictions for DE vs. SM when the standard ones fail to do so. The answer is negative, and it is negative for all strongly consequentialist preferences, which we define as follows:
Definition 1
A preference is strongly consequentialist if it depends on payoffs (agent’s own and others’) only.
Fact
If the equilibrium concept depends on the reduced normal form only, then for all strongly consequentialist preferences the set of equilibria under direct elicitation is identical to the set of equilibria under the strategy method.
Thus, while social preferences can generate a different prediction than standard preferences about what the equilibrium will be, each social preference creates the same equilibrium prediction for DE and SM as long as one follows the Kohlberg–Mertens view.