The themes of this commentary are tensions between subjective and objective judgment, between quality and quantity, between intuition and justified choice, between trust and accountability. Scalar proxies are increasingly employed to measure our performance. We are subject to continuous assessment and asked to justify our decisions when we evaluate others. Our choices are not taken on trust. At the same time, we rely more and more on artificial intelligences that make decisions for reasons that are opaque, both to human users and to the algorithms themselves. In order to gain our trust, these intelligences are asked to justify their intuitions (so-called “explainable AI”). A need to explain the reasons for their choices to others is the beginning of a kind of self-awareness in which artificial intelligences attempt to understand their own decisions based on proxies of their own internal processes.
Partisans of intuitive decision making reason that qualitative choices are best based on subjective feeling because quantitative measures mislead and distort. We should trust our intuition. Defenders of objective metrics feel that subjective evaluations are discriminatory, easily corrupted by criteria of choice that should be irrelevant to the decision at hand. We cannot trust first impressions.
These two modes of decision making sometimes result in different choices. Does one of them reliably make better decisions than the other? The question implies the existence of some metric on which one set of choices can be judged “better” than the other. The conceptual difference between subjective and objective choice may be less than partisans think. John et al. argue that the brain uses proxies to determine its own priorities. The opacity of an intuitive decision is a reflection of the limited insight we have into our internal use of metrics in choice.
Scalar metrics project multidimensional spaces onto lower-dimensional criteria of choice and promote uniformity of regulated behaviors. Legibility of the criteria of judgment favors working to rule, but there may be more striving for excellence when an agent does not know how their work will be judged. Monet and Picasso did not paint by numbers. Quantitative measures seem unsuited to evaluation of creative work where goals are under-determined. A connoisseur may not know what they are looking for until they have seen it. Judgments of taste are typically intuitive rather than objectively justified.
A conceptual distinction can be made between evaluations of agents for bonuses and jobs. The problem to be solved in the design of bonuses is what incentives will motivate the best work before evaluation. Agents need to know the general goal but too much guidance from regulators about what proxies will be used in evaluating performance may undermine achievement of the goal. The problem to be solved at a job interview is what proxies will best predict the quality of work after evaluation. A less-capable candidate may succeed if they perform a more convincing impersonation of the qualities being sought.
A subject's power to make decisions that others must accept on trust is a measure of the subject's sovereign agency. Being “called to account” situates one in the relation of an untrusted subject of a suspicious sovereign. (The etymological associations of subject and agent, connoting as they do both independent and dependent action, are complex. An agent works for another, and a subject is ruled by a sovereign, but I am a grammatical subject who possesses more free agency the less I need to justify my choices to others.)
In my role as an evaluator of student work, I am locked in a power struggle with students over the “subjectiveness” of my evaluations. Less-able students want to limit, I to defend, my freedom of judgment. They want me to justify their grades by showing how they conform to detailed rubrics that provide predefined metrics on which grades will be based. Administrators judge the performance of faculty by student evaluations that are contingent on the grades that we give. Faculty collude with students to obtain good evaluations for good grades and, by this process, grades are degraded.
The tension between objective metrics and subjective impressions reflects a tension between different risks of corruption. Greater legibility of the criteria of choice increases opportunities for corrupt action by agents but reduces the opportunities for corrupt choices by regulators. If our goal is to make better and fairer decisions – an abstract desideratum that is difficult to quantify – then we need to understand the interplay between these two risks of corruption. Mitigation of corrupt behavior is likely to be a constantly moving target as agents and regulators find new ways that “fair procedures” can be gamed for personal benefit.
John et al. suggest that market selection imposes a “hard constraint on proxy failure” within corporations. The exigencies of survival (natural selection) impose similar constraints on rampant proxy failure within organisms. Markets and natural selection do not play games. They are not themselves intentional agents. It is notable that the criteria for success before their tribunals have low legibility before judgment is pronounced. Revealed preference in economics and fitness in evolutionary biology are recognized after the fact and what has worked in the past is imperfectly predictive of what will work in the future. Preference and fitness are not proxies for anything else. Natural selection favors short-term genetic advantages but is not guaranteed to promote the long-term survival of an evolutionary lineage. Regulated markets possess features of intentional design by market engineers. The tragedy of the commons is not “market failure” but a predictable outcome of unregulated reliance on the “invisible hand.” Self-interest is not a reliable proxy for the collective good.
You have
Access
The themes of this commentary are tensions between subjective and objective judgment, between quality and quantity, between intuition and justified choice, between trust and accountability. Scalar proxies are increasingly employed to measure our performance. We are subject to continuous assessment and asked to justify our decisions when we evaluate others. Our choices are not taken on trust. At the same time, we rely more and more on artificial intelligences that make decisions for reasons that are opaque, both to human users and to the algorithms themselves. In order to gain our trust, these intelligences are asked to justify their intuitions (so-called “explainable AI”). A need to explain the reasons for their choices to others is the beginning of a kind of self-awareness in which artificial intelligences attempt to understand their own decisions based on proxies of their own internal processes.
Partisans of intuitive decision making reason that qualitative choices are best based on subjective feeling because quantitative measures mislead and distort. We should trust our intuition. Defenders of objective metrics feel that subjective evaluations are discriminatory, easily corrupted by criteria of choice that should be irrelevant to the decision at hand. We cannot trust first impressions.
These two modes of decision making sometimes result in different choices. Does one of them reliably make better decisions than the other? The question implies the existence of some metric on which one set of choices can be judged “better” than the other. The conceptual difference between subjective and objective choice may be less than partisans think. John et al. argue that the brain uses proxies to determine its own priorities. The opacity of an intuitive decision is a reflection of the limited insight we have into our internal use of metrics in choice.
Scalar metrics project multidimensional spaces onto lower-dimensional criteria of choice and promote uniformity of regulated behaviors. Legibility of the criteria of judgment favors working to rule, but there may be more striving for excellence when an agent does not know how their work will be judged. Monet and Picasso did not paint by numbers. Quantitative measures seem unsuited to evaluation of creative work where goals are under-determined. A connoisseur may not know what they are looking for until they have seen it. Judgments of taste are typically intuitive rather than objectively justified.
A conceptual distinction can be made between evaluations of agents for bonuses and jobs. The problem to be solved in the design of bonuses is what incentives will motivate the best work before evaluation. Agents need to know the general goal but too much guidance from regulators about what proxies will be used in evaluating performance may undermine achievement of the goal. The problem to be solved at a job interview is what proxies will best predict the quality of work after evaluation. A less-capable candidate may succeed if they perform a more convincing impersonation of the qualities being sought.
A subject's power to make decisions that others must accept on trust is a measure of the subject's sovereign agency. Being “called to account” situates one in the relation of an untrusted subject of a suspicious sovereign. (The etymological associations of subject and agent, connoting as they do both independent and dependent action, are complex. An agent works for another, and a subject is ruled by a sovereign, but I am a grammatical subject who possesses more free agency the less I need to justify my choices to others.)
In my role as an evaluator of student work, I am locked in a power struggle with students over the “subjectiveness” of my evaluations. Less-able students want to limit, I to defend, my freedom of judgment. They want me to justify their grades by showing how they conform to detailed rubrics that provide predefined metrics on which grades will be based. Administrators judge the performance of faculty by student evaluations that are contingent on the grades that we give. Faculty collude with students to obtain good evaluations for good grades and, by this process, grades are degraded.
The tension between objective metrics and subjective impressions reflects a tension between different risks of corruption. Greater legibility of the criteria of choice increases opportunities for corrupt action by agents but reduces the opportunities for corrupt choices by regulators. If our goal is to make better and fairer decisions – an abstract desideratum that is difficult to quantify – then we need to understand the interplay between these two risks of corruption. Mitigation of corrupt behavior is likely to be a constantly moving target as agents and regulators find new ways that “fair procedures” can be gamed for personal benefit.
John et al. suggest that market selection imposes a “hard constraint on proxy failure” within corporations. The exigencies of survival (natural selection) impose similar constraints on rampant proxy failure within organisms. Markets and natural selection do not play games. They are not themselves intentional agents. It is notable that the criteria for success before their tribunals have low legibility before judgment is pronounced. Revealed preference in economics and fitness in evolutionary biology are recognized after the fact and what has worked in the past is imperfectly predictive of what will work in the future. Preference and fitness are not proxies for anything else. Natural selection favors short-term genetic advantages but is not guaranteed to promote the long-term survival of an evolutionary lineage. Regulated markets possess features of intentional design by market engineers. The tragedy of the commons is not “market failure” but a predictable outcome of unregulated reliance on the “invisible hand.” Self-interest is not a reliable proxy for the collective good.
Financial support
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interest
None.