The world suffers under a dictatorship of no alternatives. Although ideas all by themselves are powerless to overthrow this dictatorship we cannot overthrow it without ideas.
Roberto Unger, The Dictatorship of No Alternatives
I. Who Knows?
On August 9, 2011 the New York Times reported that the Spanish Data Protection Agency had chosen to champion the claims of ninety ordinary citizens who were determined to preserve inherited meaning for a world bent on change at the speed of light.Footnote 1 In the name of these citizens’ “right to be forgotten,” the Agency ordered Google to stop indexing contested links pertaining to their pasts.
Each had a unique complaint. One had been terrorized by her former husband and didn’t want him to find her address. A middle-aged woman was embarrassed by an old arrest from her days as a university student. An attorney, Mario Costejo González, had suffered the foreclosure of his home years earlier. Although the matter had long been resolved, a Google search of his name continued to deliver links to the foreclosure notice, which, he argued, damaged his reputation.
The Agency concluded that citizens had the right to request the removal of links, and ordered Google to stop indexing the information and to remove existing links to its original sources. Google had unilaterally undertaken to change the rules of the information life cycle, when it decided to crawl, index, and make accessible personal details across the World Wide Web without asking anyone’s permission. As a result, information that would normally age had been transformed into a state of perpetual youth and highlighted in the foreground of each person’s digital identity. After all, the Spanish Data Protection Agency reasoned, not all information is worthy of immortality. Some information should be forgotten, because that is only human.
Unsurprisingly, Google challenged the Agency’s order before the Spanish High Court, which selected Mario Costeja González’s case for referral to the Court of Justice of the European Union. On May 13, 2014, after lengthy and dramatic deliberations, the Court of Justice announced its decision to assert the “right to be forgotten” as a fundamental principle of European law.Footnote 2
From the beginning, the case was framed in a peculiar way, pitting “privacy” against an indeterminate open-ended “right to know.” As one expert told the Times: “Europe sees the need to balance freedom of speech and the right to know against a person’s right to privacy or dignity.”Footnote 3 Three years later when the Court of Justice ruled in favor of Costejo González and his right to be forgotten, Google’s then CEO Eric Schmidt repeated that odd juxtaposition. Speaking to his company’s shareholders, he characterized the Court’s decision as a “balance that was struck wrong” in the “collision between a right to be forgotten and a right to know.”Footnote 4
In fact, there was no “balance” that was “struck wrong.” The Court’s decision was not one of “balancing” two conflicting goods, but rather one of redistributing a single good. The conceptual problem here is that the “right to be forgotten” does not stand in opposition to a “right to know.” Rather, it is a “right to know.” The distinction is critical, because it lifts the veil on a political contest over a new domain of fundamental rights: epistemic rights. Such rights confer inalienable entitlements to learning and to knowing. Epistemic rights are the cause of which privacy is the effect. This political contest has been obfuscated to the point of invisibility, despite the fact that its outcome will define the moral and political milieu of our information civilization.
The distribution of epistemic rights determines the degree of epistemic inequality, defined as unequal access to learning imposed by hidden mechanisms of information capture, production, analysis, and control. It is best exemplified in the fast growing abyss between what people can know and what can be known about them. The new axis of epistemic equality/inequality does not reflect what we can earn but rather what we can learn. It represents a focal shift from ownership of “the means of production” to ownership of “the production of meaning.”
Epistemic equality depends on epistemic justice, the scope of which is delineated by three essential questions that reflect the nested dilemmas of knowledge, authority, and power: What is the distribution of knowledge? What are the sources of authority that legitimate the distribution of knowledge? What is the power that sustains that authority? Put simply, “Who knows?” “Who decides who knows?” “Who decides who decides who knows?” The answers to these questions determine a society’s progress toward epistemic equality.
From this vantage point, one may observe that the real decision faced by the Court of Justice was not one of balancing “privacy” against a “right to know,” but rather about the just distribution of epistemic rights. It asked, “Who has the right to know about one’s past and in what degree?” The Court judged whether an inalienable right to learn and to know about an individual’s past adhered primarily to the individual or to Google. What dismayed Schmidt was that the Court rejected Google’s self-authorized claim to a totality of epistemic rights and instead distributed this “right to know about one’s past” in a new pattern.
The Court’s decision answered the three essential questions. First, it privileged the individual with the right to learning and knowledge about one’s own past. Second, it created a juridical “right to be forgotten” that stands as the source of legitimate authority. Third, it was to be Europe’s democratic institutions and their power to govern through the rule of law that sustain this authority – not the private power of a corporation.
The primacy of epistemic rights as the cause of privacy was implied in Justice William O. Douglas’s 1967 dissenting opinion in the Fourth Amendment case, Warden v. Hayden:
Privacy involves the choice of the individual to disclose or to reveal what he believes, what he thinks, what he possesses …. the individual should have the freedom to select for himself the time and circumstances when he will share his secrets with others and decide the extent of that sharing.Footnote 5
In Douglas’s formulation, privacy is contingent on the individual’s sovereign right to self/knowledge, a right that confers the ability to choose whether to disclose or withhold such knowledge, to what degree, and for what purpose. This choice has been an elemental right throughout most of human history. By “elemental,” I mean to mark a distinction between, on the one hand, tacit rights that are given under the conditions of human existence, and, on the other, juridical rights codified in law.
Others have addressed this distinction, and linguistic philosopher John Searle’s “pragmatic considerations of the formulation of rights” are useful here.Footnote 6 Searle argues that elemental conditions of existence are crystallized as formal human rights only at that moment in history when they come under systematic threat. For example, the ability to speak is an elemental right born of an elemental condition. The right to “freedom of expression” is a juridical right, which only emerged when society evolved to a degree of political complexity that the freedom to express oneself came under threat. Searle observes that speech is not more central to human life than breathing or being able to move one’s body. No one has declared a “right to breathe” or a “right to bodily movement” because these elemental rights have not come under attack and therefore do not require legal codification. What counts as a fundamental human right, Searle argues, is both “historically contingent” and “pragmatic.”
As is the case with all elemental rights, many epistemic rights have not yet been codified into law for the simple reason that it has not been necessary to do so. The epistemic “right to be forgotten,” for example, has always existed as an irreducible element of human experience. In a preliterate world no one needed a legal right to be forgotten when the primary record of the past was memory. One lives, one ages, and memories age too. The past is hazy, fragmented, dispersed, prismatic, and then drifts into some combination of oblivion, stories, and myth.
In traditional society, the past was preserved in ritual, and in the modern era it is technology. The printing press and widespread literacy, photography, voice recording – each made it easier to convey one’s past to the next generation, chipping away at oblivion. With these inventions, elemental epistemic rights began their migration toward formal codification. For example, Justice Brandeis was moved to formalize the right to privacy, motivated in part by the invasiveness of the newspaperman’s camera, as it bestowed an independent life on anyone’s face, far from the elemental rights of the subject before she was framed in the camera’s lens. As if in anticipation of Searle, Brandeis wrote:
That the individual shall have full protection in person and in property is a principle as old as the common law; but it has been found necessary from time to time to define anew the exact nature and extent of such protection. Political, social, and economic changes entail the recognition of new rights, and the common law, in its eternal youth, grows to meet the demands of society.Footnote 7
In retrospect, Brandeis’s indignation aimed at the journalist’s camera recalls the last stages of an infinitely long age of epistemic innocence that ended decisively over the last three decades, as the “digital tornado” abruptly transformed the conditions of existence for many twenty-first century humans. In 1986, just 1 percent of the world’s capacity to store information was in digital format; 25 percent in 2000. The year 2002 was the tipping point, when more information was stored on digital than on analogue storage devices. By 2007 digitalization had exploded to 97 percent and by 2020 the shift was largely complete.Footnote 8 Information scholar Martin Hilbert and his colleagues observe that even the foundational elements of civilization, including “language, cultural assets, traditions, institutions, rules, and laws … are currently being digitized, and for the first time, explicitly put into visible code,” then returned to society through the filter of “intelligent algorithms” deployed to govern a rapidly multiplying range of commercial, governmental, and social functions.Footnote 9
Google Inc. was a product of, driving force in, and beneficiary of this sudden sweeping change. The digitalization of everything produced vast new information territories, planetary in scope, and once celebrated by Eric Schmidt as “the world’s largest ungoverned space.”Footnote 10 By 2002, Google was well on its way to elaborating a new economic logic that I have called surveillance capitalism, whose novel imperatives compelled it to hunt and capture ever more dimensions of once private experience as raw material for newly invented processes of datafication, production, and sales. Google was not alone. The vast lawless regions of the digital became the landscape in which companies and governments ruthlessly battle for information dominance, reenacting earlier epochs of invasion, conquest, and empire building in the physical world.
In order to achieve their objectives, the leading surveillance capitalists sought to establish unrivaled dominance over the totality of the world’s information now rendered in digital format.Footnote 11 Their complex supply chains require hyperscale operations capable of storing and processing vast data flows. Surveillance capital has built most of the world’s largest computer networks, data centers, populations of servers, undersea transmission cables, advanced microchips, and frontier machine intelligence, igniting an arms race for the 10,000 or so specialists on the planet who know how to coax knowledge from these vast new data continents.Footnote 12
With Google in the lead, the top surveillance capitalists seek to control labor markets in critical expertise including data science and animal research, elbowing out competitors such as start-ups, universities, high schools, municipalities, state and federal government agencies, established corporations in other industries and less wealthy countries. In 2016, 57 percent of American computer science Ph.D. graduates took jobs in industry, while only 11 percent became tenure-track faculty. With so few teaching faculty, colleges and universities have had to ration computer science enrollments, which has significantly disrupted the knowledge transfer between generations. It’s not just an American problem. In Britain, university administrators contemplate a “missing generation” of data scientists. A Canadian scientist laments, “the power, the expertise, the data are all concentrated in the hands of a few companies.”Footnote 13
Under these unprecedented conditions, elemental epistemic rights can no longer be taken for granted. It’s not that such rights are eliminated. Rather, they fall to hidden powers – commandeered, stolen, redistributed, cornered, and hoarded.
More than 600 years ago the printing press put the written word into the hands of ordinary people, bypassing the priesthood, rescuing the prayers, and delivering spiritual communion directly into the hands of the prayerful. It was perhaps the first great event in the annals of technological disintermediation, removing the “middleman” in favor of a direct line to the end consumer.
The Internet was welcomed as an equally fearsome force of empowerment: the ultimate disintermediator, amplifying Gutenberg’s revolution as it liberates information from the old institutions and distributes it directly to the people. Thanks to the mighty powers of the digital, corporations would no longer decide the music people buy, the news they read, the knowledge they access, or the goods and services they enjoy.
Celebration distracted from a parallel development that moved in stealth just beyond the sightlines as the Internet became a Trojan horse for a novel economics that would eventually infiltrate every aspect of peoples’ lives. The shooting star of disintermediation quickly faded, leaving in its wake the secretive new middleman, that is, surveillance capitalism, which quietly remediated the relationship to all things digital. The result has been that the Internet is not ungoverned. Rather, it is owned and operated by this dark economic logic and wholly subject to its iron laws.
The digital century was to have been democracy’s Golden Age. Instead, many societies enter the third decade of the twenty-first century marked by an extremely new form of social inequality that threatens to remake the social order as it unmakes democracy. A new age of epistemic inequality has dawned in which individuals’ inalienable rights to learning and knowing about one’s own life must be codified in law if they are to exist at all.
Unequal knowledge about us produces unequal power over us, turning epistemic inequality into a critical zone of social contest in our time. Twentieth-century industrial society was based on a division of labor, and it followed that the struggle for economic equality would shape the politics of that age. Our digital century shifts society’s coordinates from a division of labor to a division of learning, and it follows that the struggle for epistemic equality will shape the politics of this age.
In redistributing the “right to be forgotten” to individuals, the European Court of Justice declared that Google’s was not to be the last word on the human or the digital future. It asserted that decisive authority must rest with the people, their laws, and their democratic institutions, even in the face of a great private power. It was to be the beginning, not the end, of a bitter struggle over the fundamental rights that will define the digital future.
The remainder of this chapter explores the iron laws of this private power, their consequences for people and the democratic polis, the historical significance of this struggle, and its remedies.
II. What Is Surveillance Capitalism?
It has long been understood that capitalism evolves by claiming things that exist outside of the market dynamic and turning them into market commodities for sale and purchase. In historian Karl Polanyi’s 1944 grand narrative of the “great transformation” to a self-regulating market economy, he described the origins of this translation process in three astonishing and crucial mental inventions that he called “commodity fictions.” The first was that human life could be subordinated to market dynamics and reborn as “labor” to be bought and sold. The second was that nature could be translated into the market and reborn as “land” or “real estate.” The third was that exchange could be reborn as “money.”Footnote 14
Surveillance capitalism originates in an even more startling mental invention, declaring private human experience as free raw material for translation into production and sales. Once private human experience is claimed for the market, it is translated into behavioral data for computational production.
Early on, it was discovered that, unknown to users, even data freely given harbor rich predictive signals, a surplus that is more than what is required for service improvement. It isn’t only what you post online, but whether you use exclamation points or the color saturation of your photos; not just where you walk but the stoop of your shoulders; not just the identity of your face but the emotional states conveyed by your “micro-expressions”; not just what you like but the pattern of likes across engagements. Soon this behavioral surplus was covertly hunted and captured across virtual and real worlds, accessible to the always-on ubiquitous digital architecture that I call “Big Other” and claimed as proprietary data.
Behavioral surplus is ultimately conveyed through complex supply chains of devices, tracking and monitoring software, and ecosystems of apps and companies that specialize in niche data flows captured in secret.Footnote 15 Data flows empty into surveillance capitalists’ computational factories, called “artificial intelligence,” where they are manufactured into behavioral predictions. A leaked 2018 Facebook document provides some insight into factory operations.Footnote 16 Facebook’s “prediction engine” is built on a machine intelligence platform called “FB Learner Flow,” which the company describes as its “AI backbone” and the key to “personalized experiences” that deliver “the most relevant content.” The machine learning system “ingests trillions of data points every day, trains thousands of models – either offline or in real time – and then deploys them to the server fleet for live predictions.” The company explains that, “since its inception, more than a million models have been trained, and our prediction service has grown to make more than 6 million predictions per second.”Footnote 17
Finally, these prediction products are rapidly swept up into the life of the market, traded in newly constituted marketplaces for behavioral predictions: human futures markets. These markets link surveillance capitalists to business customers with a keen interest in the future behavior of current and potential “users” or consumers. Certainty in human affairs is the lifeblood of these markets, where surveillance capitalists compete on the quality of their predictions, which are about individuals but are not for individuals. Surveillance capitalists have grown immensely wealthy from these trading operations, as many companies are eager to lay bets on future human behavior.
Surveillance capitalism was invented as the solution to financial emergency in the teeth of the dot.com bust when the fledgling company called Google faced the loss of investor confidence. As pressure mounted, Google’s leaders decided to boost ad revenue by using their exclusive access to user data logs, in combination with their already substantial analytical capabilities and computational power, to fabricate predictions of user click-through rates, regarded as a signal of an ad’s relevance.
Operationally, this meant that Google would both repurpose its growing cache of behavioral data, now put to work as a behavioral surplus, and develop methods to aggressively seek new sources of behavioral surplus. According to its own scientists’ accounts, the company developed new methods of surplus hunt and capture that were prized for their ability to find data that users intentionally opted to keep private and to infer extensive personal information that users did not or would not provide. This surplus would be analyzed for predictive patterns that could match a specific ad with a specific user. These new operations institutionalized a new logic of accumulation derived from the social relations of the one-way mirror: surveillance. Its mechanisms and methods are carefully engineered to produce user ignorance through obfuscation, undetectability, indecipherability, and misdirection. Success relies on bypassing individual awareness and thus overriding the individual’s rights to decide the privacy of one’s experience and one’s future course of action. Surveillance is essential to the DNA of this market form.
The elements of this economic logic were discovered, invented, elaborated, and deployed at Google from 2000 to 2004, while held in the strictest secrecy. Only when Google went public in 2004 did the world learn that during that period its revenues increased by 3,590%.Footnote 18 This increase represents the “surveillance dividend,” which raised the bar for attracting investors to the new Internet domain. The shift in the use of behavioral surplus was an historic turning point: the game-changing zero-cost asset that could be diverted from service improvement toward a genuine market exchange. This was not an exchange with “users” but rather with other companies that learned how to profit from low risk bets on users’ future behavior.
Surveillance capitalism migrated to Facebook with Google-turned-Facebook executive Sheryl Sandberg and quickly become the default model of information capitalism, attached to nearly every Internet company, start-up, and app. Like an invasive species with no natural predators, its financial prowess quickly overwhelmed the networked sphere, grossly disfiguring the earlier dream of digital technology as an empowering and emancipatory force.
While online advertisers were the dominant players in the early history of the new futures markets, surveillance capitalism is no more restricted to online ad targeting than mass production was restricted to the fabrication of the Model T. Today any actor with an interest in monetizing probabilistic information about behavior can pay to play in a range of human futures markets where behavioral predictions are told and sold.
In a world of highly commoditized products and services, companies now turn to the surveillance dividend as the source of higher margins. The result is whole new ecosystems of behavioral surplus suppliers, as companies from every sector seek ways to participate in the unilateral dispossession of private experience. Surveillance capitalism now spreads across the “normal” economy in traditionally information intensive sectors such as insurance and finance, but also in healthcare, retail, education, real estate development, and automobiles, to name but a few.
One poignant illustration of these new facts is found in the birthplace of mass production, the Ford Motor Company. One hundred years ago, pioneer capitalists like Ford bent to the revolutionary task of making things at a price that people could afford and shaping a new century of mass consumption. Henry Ford was proud to author the definition of “mass production” for the Encyclopedia Britannica, the Google of his day, describing it as “a productive organization that delivers in quantities a useful commodity of standard material, workmanship, and design, at minimum cost.”Footnote 19 Ford understood that the mass production revolution was the result, not the cause, of new era of demand in US society – farmers and shopkeepers who wanted automobiles too, but at a price they could afford. “The necessary, precedent condition of mass production is a capacity, latent or developed, of mass consumption,” he wrote. “The two go together and in the latter may be traced the reasons for the former.”Footnote 20 In Ford’s cosmos, demand and supply were twinborn, with customers and workers forever linked in a cycle of production and sales that combined low cost goods with consumption-worthy wages immortalized by the five-dollar day.
Had Henry Ford been listening to the Freakonomics Radio podcast in November 2018, he would have learned that his authoritative rendition of supply and demand had been relegated to the dustbin of history by his own distant successor, Jim Hackett. In response to the sustained slump in global auto sales, the company is already cutting thousands of jobs and eliminating models in pursuit of a price-earnings ratio befitting a high tech data company: Hackett wants Ford to become more like Facebook and Google. In this vision, the work of making and selling cars gives way to proprietary monetizable data flows – a “transportation operating system.”Footnote 21 Hackett wants Henry’s company to collect data from the “100 million people … that are sitting in Ford blue-oval vehicles … . We have as much data in the future coming from vehicles, or from users in those vehicles, or from cities talking to those vehicles, as the other competitors that you and I would be talking about [like Facebook and Google] that have monetizable attraction.”
Once customers are reinvented as data sources, it’s easy for Hackett to imagine the next step in which the data that stream from vehicles in real time are combined with Ford’s financing data, where, he says, “we already know … what people make … we know where they work; we know if they’re married. We know how long they’ve lived in their house.” Hackett concludes, “And that’s the leverage we’ve got here with the data … I think our higher purpose is that the smart vehicle and the smart world have an interaction in the future that’s much bigger than it was in the past.”Footnote 22 As one industry analyst put it, Ford “could make a fortune monetizing data. They won’t need engineers, factories, or dealers to do it. It’s almost pure profit.”Footnote 23
This is where we live now: a world in which nearly every product or service that begins with the word “smart,” “personalized,” or “connected,” from cars to “digital assistants,” to devices, appliances, and more, is a supply-chain interface for the unobstructed flow of behavioral surplus. The growth of these connections continuously increases what military strategists call the digital “attack surface,” where behavioral surplus is relentlessly tracked, hunted, coaxed, and captured. What began as a solution to financial emergency is now a burgeoning surveillance-based economic order: a surveillance economy. The dominant market form shifts under our gaze: once profits from products, then services, then profits from speculation, and now profits from surveillance.
Surveillance capitalism can no longer be defined as a specific group of corporations, neither can it be conflated with the digital technologies on which it depends. While it is impossible to imagine surveillance capitalism without the digital, it is easy to imagine the digital without surveillance capitalism. The point cannot be emphasized enough: Surveillance capitalism is not technology. Digital technologies can take many forms and have many effects, depending on the social and economic logics that bring them to life. Surveillance capitalism relies on data-gathering devices like computers, phones, sensors, microphones, and cameras. It deploys machine intelligence and platforms. It expresses itself in algorithms. But it is not the same as any of those. Just as an X-ray reveals bone and muscle, but not the soft tissue that binds them, technology is the bone and muscle here, while surveillance capitalism is the soft tissue that binds the elements and directs them into action. It is the shadow that falls over the digital, the hidden pattern that explains how this once emancipatory project transformed people and society into raw material for others’ economic gain, as the Internet itself falls to the ownership and operations of surveillance capital.
The phrase “surveillance capitalism” is not arbitrary. Surveillance capitalism’s operations are designed for the social relations of the one-way mirror. They know everything about users, but users know nothing about them. Surveillance is baked into the DNA of this logic of accumulation, because without it the surveillance dividend as expressed in the revenues, profits, and market capitalization that mark the first two decades of the twenty-first century would have been impossible to achieve.
III. Surveillance Capitalism’s Economic Imperatives
Surveillance capitalists sell certainty. They compete in human futures markets on the quality of their prediction products, which aim to guarantee outcomes or at least the ever-improving approximation to such guarantees. These guarantees have value, but in the novel logic of surveillance capitalism, their value is a function of markets that bear no organic reciprocities with their populations, now repurposed as the source of unlimited raw material supplies. The competitive dynamics thus set into motion reveal key economic imperatives, and it is these imperatives that compel epistemic inequality, setting surveillance capitalism on a collision course with democracy itself.
First, because predictions must improve in the direction of something like certainty, surplus extraction must move in the direction of something like totality. Machine learning for behavioral prediction wants data in volume and thus economies of scale in data production lay the foundation for all operations.
Once competition for prediction products intensifies, volume is not enough. Surveillance capitalists are compelled to search out ever more predictive sources of behavioral surplus. Machine learning needs volume but also variety, economies of scale but also economies of scope. This realization helped drive the “mobile revolution” sending users into the real world armed with cameras, gyroscopes, and microphones packed inside their smart new pocket-size computers, the ubiquitous interface that conveys surplus supplies to the AI hub. In the competition for scope, surveillance capitalists want your home and what you say and do within its walls.Footnote 24 They want your car, your medical conditions, and the shows you stream; your location as well as all the streets and buildings in your path and all the behavior of all the people in your city.Footnote 25 They want your voice and what you eat and what you buy; your children’s play time and their schooling; your brainwaves and your bloodstream.Footnote 26 Nothing is exempt.Footnote 27
With continued competitive intensification, surveillance capitalists discovered that the most predictive data come from intervening in behavior to tune, herd, and modify action in the direction of commercial objectives. Data scientists describe this as the shift from monitoring to actuation, in which a critical mass of knowledge about a machine system enables the remote control of that system. Now people become targets for remote control, as a third imperative, economies of action emerges as an arena of intense experimentation. Here are the practical grounds on which unequal knowledge morphs into unequal power. Epistemic inequality widens to include the distance between what people can do and what can be done to them. “We are learning how to write the music,” one data scientist explained, “and then we let the music make them dance.”Footnote 28
How shall we understand this new power “to make them dance?” Unlike twentieth-century totalitarianism, it does not employ soldiers and henchmen to threaten terror and murder. It arrives with a cappuccino, not a gun. It is a new instrumentarian power that works its will through Big Other’s medium of ubiquitous digital instrumentation to manipulate subliminal cues, psychologically target communications, impose choice architectures, trigger social comparison dynamics, and levy rewards and punishments – all of it aimed at remotely tuning, herding, and modifying human behavior in the direction of profitable outcomes and always engineered to preserve users’ ignorance.
Although he did not name it, the visionary of ubiquitous computing, Mark Weiser, foresaw the immensity of instrumentarian power as a totalizing social project. He did so in a way that suggests both its utter lack of precedent and the danger of confounding it with what has gone before: “[H]undreds of computers in every room, all capable of sensing people near them and linked by high-speed networks, have the potential to make totalitarianism up to now seem like sheerest anarchy.”Footnote 29 In fact, all those computers are not the means to a digital totalitarianism. They are, as I think Weiser sensed, the foundation of an unprecedented power that can reshape society in unprecedented ways. If instrumentarian power can make totalitarianism look like anarchy, then what might it have in store for this century?
While all power yearns toward totality, instrumentarian power’s specific purposes and methods are not only distinct from totalitarianism but they are in many ways its precise opposite. Instrumentarian power has no principle to instruct, no interest the reformation of the soul. There is no aim toward spiritual salvation, no ideology against which to judge human action. Totalitarianism was a political project that converged with economics to overwhelm society. Instrumentarianism is a market project that converges with the digital to achieve its own unique brand of social domination. Totalitarianism operated through the means of violence, but instrumentarian power operates through the means of behavioral modification.
Instrumentarianism’s specific “viewpoint of observation” was forged in the controversial intellectual domain of “radical behaviorism.” Thanks to Big Other’s capabilities, instrumentarian power reduces human experience to measurable, observable behavior, while remaining profoundly, infinitely, and radically indifferent to the meanings and motives of its targets. Radical indifference produces observation without witness. Instead of an intimate violent political religion, Big Other’s way of knowing yields the remote but inescapable presence of impenetrably complex machine systems and the interests that author them, carrying individuals on a fast-moving current to the fulfilment of others’ ends. Trained on measurable action, Big Other cares only about observing behavior and ensuring that it is continuously accessible to its ever-evolving operations of monitoring, datafication, calculation, actuation, and monetization.
Instrumentarianism’s radical indifference is operationalized in Big Other’s dehumanized methods of evaluation that produce equivalence without equality by reducing individuals to the lowest common denominator of sameness – organisms among organisms. There is no need for mass submission to social norms, no loss of self to the collective induced by terror and compulsion, no inducements of acceptance and belonging as a reward for bending to the group. All of that is superseded by a market-based digital order that thrives within things and bodies, transforming volition into reinforcement and action into conditioned response. Thanks to Big Other’s capabilities to know and to do, instrumentarian power aims for a condition of certainty without terror in the form of guaranteed outcomes. In the execution of economies of action, Big Other transforms “natural selection” into the “unnatural selection” of variation and reinforcement authored by market players and the competition for surveillance revenues.
The paradox is that because instrumentarianism does not claim bodies for some grotesque regime of pain and murder, many are prone to undervalue its effects and lower their guard. Under the regime of instrumentarian power, the mental agency and self-possession of autonomous human action are gradually submerged beneath a new kind of automaticity: a lived routine of stimulus-response-reinforcement that operates outside of awareness and is aggregated as statistical phenomena: the comings and goings of mere organisms.
The challenges associated with successful economies of action have become a critical experimental zone for the elaboration of instrumentarian power. It is likely that much of this experimentation is invisible to the public, but at least some hides in plain sight, where one can observe how knowledge tips into power, epistemic inequality into epistemic injustice.
For example, Facebook conducted “massive-scale contagion experiments,” the results of which were published in 2012 and 2014. The first aimed to subliminally induce voting in the lead-up to the 2010 US mid-term elections. The second aimed to influence users to feel “happy” or “sad.” As a result of these experiments researchers concluded that, (1) it is possible to deploy subliminal cues on Facebook pages to alter real-world behaviour and emotions and (2), it is possible to accomplish such remote behavioral and affective modification with undetectable methods designed to bypass human awareness. Indeed, the very first paragraph of the 2014 research article on emotional contagion celebrates these findings: “Emotional states can be transferred to others via emotional contagion, leading people to experience the same emotions without their awareness.”Footnote 30
Facebook provides yet another observation point for the development of economies of action. In May 2017, three years after the publication of the contagion studies, The Australian broke the story on a confidential twenty-three-page Facebook document written by two Facebook executives and aimed at the company’s Australian and New Zealand advertisers. The report depicted the corporation’s systems for gathering “psychological insights” on 6.4 million high school and tertiary students as well as young Australians and New Zealanders already in the workforce. The Facebook document detailed the many ways in which the corporation uses its stores of behavioral surplus to simulate and predict individual and group affective patterns in order to pinpoint the exact moment at which a young person needs a “confidence boost” and is therefore most vulnerable to a specific configuration of advertising cues and nudges: “By monitoring posts, pictures, interactions, and Internet activity, Facebook can work out when young people feel ‘stressed,’ ‘defeated,’ ‘overwhelmed,’ ‘anxious,’ ‘nervous,’ ‘stupid,’ ‘silly,’ ‘useless,’ and a ‘failure.’”Footnote 31
The report reveals Facebook’s interest in leveraging this affective surplus for the pivot from monitoring to actuation. It boasts detailed information on “mood shifts” among young people based on “internal Facebook data,” and it claims that not only can Facebook’s prediction products “detect sentiment,” they can also predict how emotions are communicated at different points during the week. These data are then used to match each emotional phase with appropriate ad messaging for the maximum probability of guaranteed sales. “Anticipatory emotions are more likely to be expressed early in the week,” the analysis counsels, “while reflective emotions increase on the weekend. Monday–Thursday is about building confidence; the weekend is for broadcasting achievements.” The young adults of Australia’s and New Zealand’s cities and towns had no reason to suspect that their fears and fantasies were being routinely exploited for commercial result at the precise moment of their greatest vulnerability. (“NEED A CONFIDENCE BOOST? CLICK HERE! BUY THIS BLACK LEATHER JACKET NOW! FREE OVERNIGHT DELIVERY!”)
Facebook publicly denied these practices, but former Facebook product manager Antonio Garcia-Martinez, the author of Chaos Monkeys, a useful account of Silicon Valley, described in the Guardian the routine application of such practices and accused the corporation of “lying through their teeth.” He concluded: “The hard reality is that Facebook will never try to limit such use of their data unless the public uproar reaches such a crescendo as to be un-mutable.”Footnote 32 It is Facebook that knows. It decides who knows. It decides who decides.
The public’s intolerable knowledge disadvantage is deepened by surveillance capitalists’ perfection of mass communications as gaslighting. Indeed, these firms have long mastered the tactical arts of disinformation and fake news, paving the way for social complacency toward the crisis of truthfulness and social trust that has engulfed public communications. A few examples are illustrative. On April 30, 2019 Mark Zuckerberg made a dramatic announcement at the company’s annual developer conference, declaring: “The future is private.”Footnote 33 A few weeks later, a Facebook litigator appeared before a federal district judge in California to thwart a user lawsuit over privacy invasion, arguing that the very act of using Facebook negates any reasonable expectation of privacy “as a matter of law.”Footnote 34 While leaked internal documents describe the firm’s sophisticated methods for accruing granular psychological insights for targeting and triggering individuals, in early 2020 its Vice President of Public Policy told a public forum: “We don’t do surveillance capitalism, that by definition is surreptitious; we work hard to be transparent.”Footnote 35 In May 2019, Google CEO Sundar Pichai wrote in the New York Times of his corporation’s commitment to the principle that “privacy cannot be a luxury good.”Footnote 36 Five months later Google contractors were observed offering $5 gift cards to homeless people of color in an Atlanta park in return for a facial scan.Footnote 37 While Amazon cracked down on employees for violating the company’s privacy by publicly discussing its policies and practices, the corporation aggressively strengthens the one-way mirror, marketing its collection of surveillance-as-a-service connected devices and appliances based on the Alexa voice recognition system. Its latest suite of Internet-enabled devices was characterized by the Seattle Times as “a sweeping vision of automation, entertainment, ubiquitous surveillance and commerce permeating nearly every aspect of life.”Footnote 38
Facebook’s denial of psychological targeting practices invites even more scrutiny in light of the leaked 2018 company document, which described its “AI Hub.”Footnote 39 That report also indicated that the company’s extraordinary data flows and computational production are dedicated to meeting its corporate customers’ “core business challenges” with procedures that link prediction, microtargeting, intervention, and behavior modification. For example, a Facebook service called “loyalty prediction” was touted for its ability to plumb proprietary behavioral surplus in order to predict which individuals are “at risk” of shifting their brand allegiance. This knowledge alerts advertisers to intervene promptly with targeted messages designed to stabilize loyalty just in time to alter the course of the future.
Google’s experimentation with economies of action moved boldly into the real world with the augmented reality game Pokémon Go. The project had been incubated at Google for many years, led by John Hanke, an early inventor of satellite mapping and leader of Google’s mapping operations including Google Earth and Street View, both critical sources of surplus data supplies. Later Hanke headed up his own augmented reality shop inside Google, Niantic Labs, where Pokémon Go was developed and spun off from the company just in time to go to market with Hanke as its head and Goggle its principal investor.Footnote 40
Pokémon Go brought the emerging science of remote population tuning and herding to the real world: real streets, real towns, real cities. It added the rewards and punishments of gamification to the methods of subliminal cueing and the manipulation of social comparison dynamics in order to bypass users’ awareness of the situational facts: families and friends were engaged in a search game without knowing that it was they who were being searched and gamed. Players were, in fact, unwitting pawns in a hidden higher-order game that aimed to provide “footfall” to fee-paying establishments as they vied for real-world consumer visits in exactly the same way that online advertisers pay for the virtual visits of clicks and engagement. Niantic used immense caches of data collected from game players’ devices in order to apply the incentives and reinforcements of gamification for the sake of herding players to the real-world business customers in its futures markets, from McDonald’s and Starbucks to Joe’s Pizza.Footnote 41
These escalating zones of experimentation and their practical success suggest a disturbing conclusion: the competitive necessity of economies of action means that surveillance capitalists must use all means available to supplant autonomous action with heteronomous behavior. Human awareness is a threat to surveillance revenues because the mobilization of awareness endangers the larger project of behavior modification. Philosophers recognize “self-regulation,” “self-determination,” and “autonomy” as expressions of “freedom of will,” and a flourishing research literature illuminates the antecedents, conditions, consequences, and challenges of human self-regulation as a universal need. The capacity for self-determination is understood as an essential foundation for behaviors associated with critical human capabilities such as empathy, volition, reflection, personal development, authenticity, integrity, learning, goal accomplishment, impulse control, creativity, and the sustenance of intimate relationships. “Implicit in this process is a self that sets goals and standards, is aware of its own thoughts and behaviors, and has the capacity to change them. Indeed, some theorists have suggested that the primary purpose of self-awareness is to enable self-regulation.” Every threat to human autonomy begins with an assault on awareness, “tearing down our capacity to regulate our thoughts, emotions, and desires.”Footnote 42
The salience of self-awareness as a bulwark against self-regulatory failure is also highlighted in recent research on “susceptibility to persuasion,” which concludes that “the ability to premeditate” is the single-most-important determinant of one’s ability to resist persuasion.Footnote 43 People who harness self-awareness to think through the consequences of their actions are more disposed to chart their own course and are thus significantly less vulnerable to persuasion techniques. Self-awareness also figures in the second highest-ranking protection from susceptibility to persuasion: commitment. Those who consciously commit to a course of action or set of principles are less likely to be persuaded to do something that violates their commitment.
In one sense, there is nothing remarkable in observing that capitalists would prefer individuals who submit to arrangements that advantage capital. It would be incorrect, however, to conclude that today’s surveillance capitalists simply represent more of the same. The structural requirements of economies of action turn the means of behavioral modification into an engine of growth. At no other time in history have the wealthiest private corporations had at their disposal a pervasive global architecture of ubiquitous computation able unilaterally to amass unparalleled concentrations of information about individuals, groups, and populations sufficient to mobilize the pivot from knowledge about behavior to the actuation of commercially desirable behavior. In other words, when we climb the mountain of the division of learning and peek into the fortress, we see a frontier operation run by geniuses and funded by vast capital outlays that is furiously dedicated to knowing everything about us and pivoting that knowledge to the remote control of people. These are unprecedented conditions that bestow an unprecedented instrumentarian power on private capital.
IV. Information Warfare
While democracy slept, epistemic inequality was produced, institutionalized, and protected by an unequal power that annuls the possibility of conflict by denying the right of combat. In this case, denial is achieved through Big Other’s hidden techno-economic systems that steal, know, and shape human behavior for the sake of others’ gain. These capabilities depend on the evasion of human awareness. This entails the denial of the epistemic rights that confer individual sovereignty over self/knowledge. Instrumentarian power is the guarantor of epistemic inequality, the hammer of epistemic injustice, and the usurper of epistemic rights.
Because one’s self and all the selves are meant to sleepwalk peacefully through this known unknown, sudden news from behind the veil can have an electrifying effect, if only for a while. This helps to explain the force with which the story of Cambridge Analytica broke on the world in March 2018, when Chris Wylie, the young mastermind-turned-whistleblower, unleashed a torrent of information on that company’s secret efforts to predict and influence individual voting behavior, quickly riveting the world on the small political analytics firm and the giant source of its data: Facebook. There are many unanswered questions about the legality of Cambridge Analytica’s complex subterfuge, its actual political impact, and its relationship with Facebook. My interest here is restricted to how it replicated surveillance capitalism’s ordinary practices, and the implications of that fact.Footnote 44
Academic researchers had already demonstrated the predictive power of behavioral surplus culled from Facebook pages, the insights into human personality that it can yield, the resulting opportunities for behavioral manipulation and modification, and the commercial value of such methods. Wylie recounted his fascination with these studies, especially as they might be pivoted from commercial to political outcomes.Footnote 45 Through a complicated chain of events, it was Wylie who persuaded Cambridge Analytica’s owner, the secretive software billionaire and active enemy of democracy Robert Mercer, and his operatives, including anti-democracy’s dark theorist Steve Bannon, to use Facebook data to advance Mercer’s political aims.
Cambridge Analytica’s operations followed the surveillance capitalist playbook. They were designed to produce ignorance through secrecy and the careful evasion of individual awareness: “We exploited Facebook to harvest millions of people’s profiles,” Wylie admitted, “and built models to exploit what we knew about them and target their inner demons.”Footnote 46 The objective was “behavioral micro-targeting … influencing voters based not on their demographics but on their personalities.”Footnote 47 “I think it’s worse than bullying, because people don’t necessarily know it’s being done to them,” Wylie reflects. “At least bullying respects the agency of people because they know … if you do not respect the agency of people, anything that you’re doing after that point is not conducive to a democracy. And fundamentally, information warfare is not conducive to democracy.”Footnote 48
Wylie describes Cambridge Analytica’s operations as “information warfare,” correctly acknowledging that this form of shadow warfare originates in significant asymmetries of knowledge and the power produced by such knowledge. In other words, information warfare exploits epistemic inequality, while its effects intensify epistemic injustice.
However, the Cambridge Analytica narrative suggests an even more disturbing conclusion. The political firm was only able to operate as an information warrior because the conditions for successful warfare and its weapons had already been developed by surveillance capital. Surveillance capitalist operations like Facebook, Google, Amazon, Microsoft, and countless others are best understood as examples of information-warfare-for-profit. These firms are information mercenaries that leverage unprecedented asymmetries of knowledge/power for the sake of revenues, which, in turn, fund their continued dominance and the intensification of epistemic inequality.
Consider how a small firm such as Cambridge Analytica was able to enter the fray of information war. The so-called political consultancy functioned as a parasite buried into the host of Facebook’s vast behavioral data supply chains, while adapting its host’s foundational mechanisms and methods: secret data capture, extraction of behavioral surplus, predictive computational analysis, behavioral microtargeting in the service of economies of action.
Cambridge Analytica channeled these methods and mechanisms, merely pivoting the surveillance capitalist machinery from commercial markets in human futures toward guaranteed outcomes in the political sphere. Its strategies of secret invasion and hidden conquest were the same standard operating procedures to which billions of innocent “users” are subjected each day. What better description of the unsavory treatment of 6.6 million young people in Australia and New Zealand whose social anxieties were extracted and manipulated for profit than to say that we “built models to exploit what we knew about them and target their inner demons”? What more apt reflection on “loyalty prediction” interventions based on trillions of data points, or Pokémon Go’s manipulative game within the game than, “I think it’s worse than bullying, because people don’t necessarily know it’s being done to them”?
It is worthwhile noting that it was Google’s Eric Schmidt who first pried open this Pandora’s box, transferring surveillance capitalism’s core mechanisms of behavioral microtargeting to the Obama presidential campaigns, where Wylie enjoyed some of his early training under Obama’s Director of Targeting.Footnote 49 In little over a decade, Schmidt’s innovations have become the envy of every enemy of democracy, well within reach of the plutocrat’s wallet or the more modest budgets of other non-state actors. Indeed, information warfare is widely assumed to originate in the State for the purposes of political, cultural, and, or, military destabilization, just as we once considered behavioral modification or surveillance as projects of the State. But recent theories of information warfare have begun to recognize the growing ease with which non-state actors, such as Robert Mercer or ISIS, can undertake information warfare.Footnote 50
What has not yet been adequately recognized is that surveillance capitalism has already institutionalized information warfare as a market project. It is only on the strength of this construction that states and non-state actors alike can succeed as information warriors. Such operations exist as parasites on the host of the larger surveillance capitalist body. A simple set of distinctions framed by US Naval Academy professor and cyber-security expert Martin Libicki are useful here as they help to describe the contributions of the surveillance capitalist host that deliver triple nourishment by providing what Libicki identifies as (1) the conditions, (2) the weapons, and (3) the opportunity to wage information war.Footnote 51
Conditions
Surveillance capitalism’s economic imperatives increase the range of societal vulnerabilities for parasitic exploitation. Libicki observes that US companies lead the world in the collection and processing of personal information. Pervasive datafication and connectivity, largely driven by surveillance capital, substantially increases society’s “attack surface” leaving it more vulnerable to a range of assault capabilities. Libicki asks: “[W]hy collect what can be stolen?”Footnote 52
Weapons
The surveillance capitalism host also provides the “weapons” (data, methods, and mechanisms) necessary to exploit the vulnerabilities that it creates. “Ultimately it has been the evolution of the information economy that has provided the means by which hostile others can run a pervasive harassment campaign,” Libicki acknowledges.Footnote 53 He cites “data-mining techniques” that construct “realistic simulations of individuals, indeed perhaps of most of a population … integrating data streams with enormous cloud-based storage, powerful processing, and a dash of artificial intelligence.” Such simulations, he notes, “may be used to test every individual’s reaction to events,” including “advertising, political campaigns, and psychological operations, and even to guess what might go viral through person-to-person interactions,” just as we saw in the case of Facebook’s contagion experiments.Footnote 54
Libicki catalogues some of these weapons, which are already essential methods in surveillance capitalism’s arsenal: “exquisite psychological operations,” “messages tailored to one person at a time,” and data-mining able to characterize individuals precisely enough for “crafting the message most likely to resonate with them.” These achievements permit the “optimization” of “psychological operations,” “new conduits for persuasion,” and “the manipulation of fear,” all of which are amply on display in surveillance capital’s expanding zones of experimentation as it learns to translate knowledge into power.Footnote 55
Opportunity
Libicki notes that information warfare unfolds in an atmosphere of “deep secrets” protected by “a dense fog of ambiguity.”Footnote 56 These conditions are structurally enabled. First, they reproduce and extend the asymmetries of knowledge and power already compelled by economic imperatives. Second, as long as the surveillance capitalist host operates from the perspective of radical indifference, it is like a Cyclops whose single line of sight leaves it blind to everything but its prey. Parasitic operations succeed because they fall on the blind sides of radical indifference. This means that parasites can persist unchallenged for long periods. It is difficult to find them, disable them, and to confirm their destruction. Such was the case with Cambridge Analytica, which fed off illegitimately collected Facebook data that were illegitimately sold for nefarious purpose – all of it secreted in the shadow of Facebook’s single eye. Radical indifference creates a void where social reciprocities once thrived. Surveillance capitalists cannot fill this void because doing so would violate the logic of accumulation on which everything depends. The rogue forces of disinformation grasp this fact more crisply than anyone else, as they cleverly exploit the Cyclops of radical indifference and escalate the perversion of information in an open society.Footnote 57
Surveillance capitalism’s antidemocratic and antiegalitarian consequences are best described as a market-driven coup from above. It is not a coup d’état in the classic sense but rather a coup de gens: an overthrow of the people concealed in the technological Trojan horse that is Big Other. On the strength of its audacious annexation of human experience, this coup achieves exclusive concentrations of knowledge and power that undermine democracy at root and crown. It poisons democracy at its roots by usurping the epistemic rights that confer individual sovereignty over self/knowledge, thus weakening self-determination and undermining human autonomy without which democratic society is unimaginable. It poisons democracy from above by imposing a new axis of epistemic inequality that now threatens to remake society while unmaking the structure and function of democratic institutions. Surveillance capital wages a quiet information war for epistemic hegemony and the power over human behavior that it promises, thus channeling capitalism’s adversarial bloodline not toward groups like workers or consumers who are defined by their economic function, but rather toward the widest possible category of people: “users.” This broad target of all people engaged in all forms of life is as all-encompassing as the economic imperatives that compel surveillance capitalism toward societal domination. It bears a single message: CAVEAT USOR.
V. The Poisoned Crown: The Division of Learning in Society
When the young Emile Durkheim wrote The Division of Labor in Society, a treatise that would become a foundational text of modern sociology, the title itself was controversial. Why this was the case is relevant to our predicament today. Because the transformation that we witness in our time echoes many of the century-old observations in Durkheim’s seminal work, a few key points are reviewed here.
The division of labor had been understood as a critical means of achieving labor productivity through the specialization of tasks. Adam Smith memorably wrote about this new principle of industrial organization in his description of a pin factory, and the division of labor remained a topic of economic discourse and controversy throughout the nineteenth century. Durkheim recognized labor productivity as an economic imperative of industrial capitalism that would drive the division of labor to its most extreme application, but that was not what held his fascination.
Instead, Durkheim trained his sights on the social transformation already gathering around him, observing that “specialization” was gaining “influence” in politics, administration, the judiciary, science, and the arts. He concluded that the division of labor was no longer quarantined in the industrial workplace. Instead it had burst through those factory walls to becoming the central organizing principle of industrial society: “Whatever opinion one has about the division of labor,” Durkheim wrote, “everyone knows that it exists, and is more and more becoming one of the fundamental bases of the social order.”Footnote 58
Economic imperatives predictably mandated the division of labor in production, but what was the purpose of the division of labor in society? This was the question that motivated Durkheim’s analysis, and his century-old conclusions are relevant for us now. He argued that the division of labor accounts for the interdependencies and reciprocities that link the many diverse members of a modern industrial society in a larger prospect of solidarity. This new principle of social order was an essential response to the breakdown of traditional communities as the old sources of meaning that had reliably bonded people across space and time melted away. What would hold society together in the absence of the rules and rituals of place, clan, and kin? Durkheim’s answer was “the division of labor.” Society’s need for a coherent new source of meaning and structure was the cause, and the effect was an ordering principle that enabled and sustained a healthy modern community. The reciprocities of the division of labor would breed mutual need, interdependence, and respect, all of which imbue this new ordering principle with moral force. As the young social theorist explained:
The most remarkable effect of the division of labor is not that it increases output of functions divided, but that it renders them solidary. Its role … is not simply to embellish or ameliorate existing societies, but to render societies possible which, without it, would not exist …. It passes far beyond purely economic interests, for it consists in the establishment of a social and moral order sui generis.Footnote 59
Durkheim’s vision was neither sterile nor naive. He recognized that things can take a dark turn and often do, resulting in what he called an “abnormal division of labor” (sometimes translated as “pathological”) that produces social distance, injustice, and discord in place of reciprocity and interdependence. In this context, Durkheim singled out the destructive effects of inequality on the division of labor in society, especially what he viewed as the most dangerous source of inequality: extreme asymmetries of power that make “conflict itself impossible” by “refusing to admit the right of combat.” Such pathologies can only be cured by a politics that asserts the people’s right to contest, confront, and prevail in the face of unequal and illegitimate power over society. In the late nineteenth and most of the twentieth centuries, that contest was defined by economic inequality and led by labor and other social movements that asserted rights to economic justice through new institutional constructions: unions, collective bargaining, public education.
But now it is a division of learning that follows the same migratory path from the economic to the social domain once traveled by the division of labor. The progress of digitalization and information intensification began in the offices and factories of the 1980s, when workplaces mobilized around the new questions concerning knowledge, authority, and power, thus drawing labor and capital into a novel and poorly understood crisis of epistemic equality.Footnote 60
Forty years later it is possible to see that the labor crisis of the late twentieth century was an early phase of a longer struggle over the division of learning in society that would engulf twenty-first century societies as the dilemmas of knowledge, authority, and power broke through the boundaries of the economic sphere to overwhelm and finally saturate everyday life. Now the division of learning “passes far beyond purely economic interests,” as it establishes the basis for a new social order and its moral content. But scientists warn that the world’s capacity to produce information has substantially exceeded its ability to process and store information.Footnote 61 Information is digital, but its volume exceeds our ability to discern its meaning.
As the solution to this problem, Martin Hilbert counsels, “The only option we have left to make sense of all the data is to fight fire with fire,” using “artificially intelligent computers” to “sift through the vast amounts of information … Facebook, Amazon, and Google have promised to … create value out of vast amounts of data through intelligent computational analysis.”Footnote 62 The rise of surveillance capitalism, however, necessarily turns Hilbert’s advice into a damning vision of social pathology. Although he does not mean to, Hilbert’s suggestion merely confirms the self-authorized epistemic dominance of the surveillance capitalists and the institutionalization of epistemic inequality as the division of learning in society is bent to the commercial interests of private surveillance capital.
Surveillance capitalism’s command of the division of learning in society begins with the problem of what may be called “the two texts.” The first is the public-facing text, familiar and celebrated for the universe of information and connection that it brings to our fingertips. We are its authors and its readers. Google Search codifies the informational content of the World Wide Web. Facebook’s News Feed binds the social network. Much of this public-facing text is composed of what we inscribe on its pages: posts, blogs, videos, photos, conversations, music, stories, observations, “likes,” tweets, and all the great massing hubbub of lives captured and communicated.
Under the regime of surveillance capitalism, however, the first text does not stand alone; it trails a shadow close behind. The first text, full of promise, actually functions as the supply operation for this second shadow text. Everything that is contributed to the first text, no matter how trivial or fleeting, becomes a target for surplus extraction. That surplus fills the pages of the shadow text, hidden from view and “read only” for surveillance capitalists.Footnote 63 In this text, private experience is dragooned as raw material to be accumulated and analyzed as means to others’ market ends. The shadow text conceals more about us than we can know about ourselves, exemplified in Facebook’s ingestion and calculation of trillions of behavioral data points each day. Worse still, it is nearly impossible to refrain from contributing to this vast concentration of shadow knowledge, as Big Other feeds on the normal and necessary routines of daily life.
Finally, shadow knowledge ricochets back into lives, morphing into the instrumentarian power to shape what is seen, learned, and done. As Frank Pasquale describes Google: “The decisions at the Googleplex are made behind closed doors … the power to include, exclude, and rank is the power to ensure which public impressions become permanent and which remain fleeting … Despite their claims of objectivity and neutrality, they are constantly making value-laden, controversial decisions. They help create the world they claim to merely ‘show’ us.”Footnote 64 When it comes to the shadow text, we are the objects of its narratives from whose lessons we are excluded. As the source from which all the treasure flows, the shadow text is about us, but it is not for us. Instead it is created, maintained, and exploited outside our awareness for others’ profit.
Just as Durkheim warned his society a century ago of an abnormal division of labor, we now enter the third decade of the twenty-first century with our societies already disfigured by a division of learning that drifts into pathology marked by epistemic inequality and injustice at the hands of the unprecedented asymmetries of knowledge institutionalized in the shadow text. The pathology does not stop here. Asymmetries of knowledge feed the progress of instrumentarian power as exclusive knowledge is translated through the networked layer of digital instrumentation to produce new capabilities of actuation at scale – influencing, tuning, herding, and modifying human behavior toward others’ commercial ends. The division of learning is thus both the ascendant principle of social order in the twenty-first century and already hostage to surveillance capital’s privileged position, empowered by its ownership of the texts and its exclusive command of analysis and prediction capabilities.
More than thirty years ago, the legal scholar Spiros Simitis published a remarkable essay on the theme of privacy in an information society. Simitis grasped early on that the already visible trends in public and private “information processing” harbored threats to society that transcended narrow conceptions of privacy and data ownership. “[P]ersonal information is increasingly used to enforce standards of behavior,” he wrote. “Information processing is developing, therefore, into an essential element of long-term strategies of manipulation intended to mold and adjust individual conduct.”Footnote 65 Simitis argued that these trends were incompatible not only with privacy but with the very possibility of democracy, which depends on a reservoir of individual proficiencies associated with autonomous moral judgment and self-determination.
Building on Simitis’s work, Paul Schwartz warned in 1989 that computerization would transform the delicate balance of rights and obligations upon which privacy law depends: “Today the enormous amounts of personal data available in computers threaten the individual in a way that renders obsolete much of the previous legal protection.” Most important, Schwartz foresaw that the scale of the still emerging epistemic crisis would impose risks that exceed the scope of privacy law. “The danger that the computer poses is to human autonomy,” he warned. “The more that is known about a person, the easier it is to control him. Insuring the liberty that nourishes democracy requires a structuring of societal use of information and even permitting some concealment of information.”Footnote 66
Both Simitis and Schwartz sensed the ascent of the division of learning as the axial principle of a new computational societal milieu, but they could not have anticipated the rise of surveillance capitalism and its consequences. While the explosive growth of information territories shifts a crucial axis of the social order from a twentieth-century division of labor to a twenty-first century division of learning, it is surveillance capitalists who command the field and unilaterally lay claim to a disproportionate share of the epistemic rights that shape the division of learning in society.
Instead of the long anticipated explosion of democratization, the competitive struggle over surveillance revenues has dragged our societies into a regressive “pre-Gutenberg” pattern, in which a pathological division of learning is captured by private capital, presided over by a narrow priesthood of privately employed computational specialists, their privately owned machines, and the economic interests for whose sake they learn. This epistemic violence runs free of law, of market restraints, and of organic reciprocities with its communities, which are no longer required as sources of customers or employees but rather as a passive unwitting cornucopia of raw material for production and sales.
The result is best understood as the unauthorized privatization of the division of learning in society. Just as Durkheim warned of the subversion of the division of labor by the powerful forces of industrial capital a century ago, today’s successful prosecution of information warfare aimed at citizen-users by surveillance capital now exerts private power over the definitive principle of social order in our time. Epistemic inequality is enshrined as the signature deformation of this epoch as the pathologies of the division of learning infect the societal superstructure.
Here is democracy’s poisoned crown: As things currently stand, it is the surveillance capitalist corporations that know. It is the market form that decides. It is surveillance capital that decides who decides. Experts in the disciplines associated with machine intelligence know this, although they have little grasp of its implications as the signal of the defining axis of social inequality in our time. One data scientist writes: “Whoever has the best algorithms and the most data wins …. Google with its head start and larger market share, knows better what you want … whoever learns fastest wins.”Footnote 67 In 2018, the New York Times reported that Google CEO Sundar Pichai had located his office on the same floor as the company’s AI research lab, noting it as a trend among many CEOs – a literal take on the concentration of knowledge and power.Footnote 68
Here is the paradox in which citizens are caught: Democracy is the target of this epistemic poison, and its only antidote.
VI. Remedies
Despite the variation of motives among the ninety Spanish citizens, their collective assertion of a “right to be forgotten” announced a new twenty-first-century contest over once elemental epistemic rights now under global assault from private surveillance capital. The European Court of Justice’s decision on the “right to be forgotten” so often reduced to the legal and technical considerations related to the deletion or delinking of personal data, was in fact a key inflection point at which elemental epistemic rights successfully sought the protection of democratic institutions as they undertook the long migration toward law. It was the Court of Justice that wrote an early chapter in what is now an epoch-defining struggle to claw back epistemic rights from the powerful forces of surveillance capital and its determination to assert authority over what can be learned and known. There is evidence that citizens and lawmakers around the world are finally picking up the pen, as a new wave of public “techlash,” legislative initiatives, and regulatory actions begins to take shape.
The past offers good counsel at this key juncture. In his Pulitzer prize-winning history, Prophets of Regulation, Thomas McCraw recounts the phases and distinct purposes of regulatory regimes in the US: the 1870s and the initial period of industrialization; the early twentieth century, especially 1900–1916; the 1930s and the New Deal; and the onset of deindustrialization during the 1970s and 1980s. The challenges of each era brought distinct forms of law and regulatory leadership. At the dawn of the twentieth century it was the muckrakers and progressives who defined the regulatory paradigm. Later, the lawyers dominated. It was only the past few decades that saw the economists as framers of the regulatory vision.
McCraw observes that this “economists’ hour” will certainly end and wonders what will follow. In considering the arc of this history, he finds clues, noting that concerns for justice and fairness have generally overshadowed the more narrow aims of economic growth in the construction of regulatory regimes. “Regulation,” he writes, “is best understood as a political settlement.”Footnote 69
This perspective suggests that the chartering frameworks of a digital future compatible with the principles of a democratic society are most likely to be defined and led by champions of democracy. The key principles here must be: (1) the redistribution of epistemic rights to the sovereign individual, (2) under the authority of the rule of law, and (3) sustained by the power of democratic institutions and their governance. Elected officials, citizens, and specialists can seize this opportunity, allied in the knowledge that despite its failures and shortcomings, democracy is the one idea to emerge from the long human story that enshrines the peoples’ right to govern themselves and asserts the ideal of the sovereign individual as the single most powerful bulwark against tyranny.
McCraw delivers a warning with his observations, however, and it is significant for us now. The historical record shows that regulators failed when they were unable “to frame strategies appropriate to the particular industries they were regulating.”Footnote 70 The lesson is that today’s new legislative and regulatory challenges will not be met effectively without a clear grasp of surveillance capitalism as a novel economic logic defined by distinct economic imperatives and the specific practices and consequences that they compel. Twenty-first century solutions to our twenty-first century challenges may build on existing paradigms of privacy and antitrust but will also have to move beyond and even transform those paradigms, as we learn how to interrupt and outlaw surveillance capitalism’s key mechanisms, methods, and markets. As the European Commissioner for Competition Margrethe Vestager recently put it: “One of the things I have learned from surveillance capitalism … is [that] it’s not you searching Google, it is Google searching you. And that gives a very good idea about not only what you want to buy but also what you think. So we have indeed a lot to do.”Footnote 71
The prospects of a new regulatory paradigm are improved with a clear grasp of the forces that have impeded its emergence during the first two decades of the twenty-first century. Lawmakers have been reluctant to challenge surveillance capitalism for many reasons.Footnote 72 Among these was an unwritten policy of “surveillance exceptionalism” forged in the aftermath of the September 11 terrorist attacks, when the government’s concerns shifted from online privacy protections to a new zeal for “total information awareness.” In that political environment, the fledgling surveillance capabilities emerging from Silicon Valley appeared to hold great promise. Another reason has been the unparalleled lobbying infrastructure pioneered by Google and later joined by Facebook and others. A third is the value of behavioral microtargeting to political campaigns.Footnote 73
As a new wave of public mobilization and lawmaking gathers force, doing “a lot” will require overcoming these old impediments. It also means confronting surveillance capitalism’s strategic propaganda campaigns and its mass communications tactics based on gaslighting and mendacity. Strategies and tactics have been designed as defensive fortifications intended to undermine and intimidate lawmakers and citizens alike, confounding judgment and freezing action. What follows are three examples of strategic propaganda that have received relatively little scrutiny compared to the damage they do.
The Innovation Defense
Surveillance capitalist leaders vigorously portray democracy as the enemy of innovation. Facebook’s Head of Global Policy and Communications, Sir Nick Clegg, warned in 2019 that any restrictions resulting from “tech-lash” risked making it “almost impossible for tech to innovate properly,” invoking the threat of Chinese ascendance as the price the West would pay for law. “I can predict that … we will have tech domination from a country with wholly different sets of values,” he insisted.Footnote 74
Clegg was only repeating what surveillance capitalist leaders had been proselytizing for years. In 2010, Mark Zuckerberg announced that privacy was no longer a “social norm,” celebrating Facebook’s explosive new “privacy policies” that publicly displayed personal information by default as evidence of his determination to innovate rather than “be trapped” by conventions or law. “We decided that these would be the social norms now and we just went for it.”Footnote 75 In 2011, former Google CEO Eric Schmidt warned that government overreach would foolishly constrain innovation: “We’ll move much faster than any government.”Footnote 76 That year Google founder Larry Page complained that “old institutions like the law” impede the firm’s freedom to “build really great things.”Footnote 77 All of this rhetoric is actually a hand-me-down from another era when Gilded Age barons, whom we now call “robbers,” insisted that there was no need for law when one had the “law of evolution,” the “laws of capital,” and the “laws of industrial society.” As historian David Nasaw put it, the millionaires preached “democracy had its limits, beyond which voters and their elected representatives dared not trespass lest economic calamity befall the nation.”Footnote 78
George Orwell observed that the rhetorical silences and blatant contradictions of power are designed so that “to see what is in front of one’s nose needs a constant struggle.”Footnote 79 The tech companies’ innovation rhetoric helped to suppress criticism from users and their lawmakers for many years, despite what was in front of their noses. Facebook and Google were regarded as innovative companies that sometimes make dreadful mistakes at the expense of privacy. Since then the picture has sharpened and we are getting better at seeing what’s in front of our collective nose. It is now possible to recognize that what were once regarded as mistakes – Google Glass, Gmail scanning, Street View’s theft of private data, Facebook’s Beacon program, its sale of private information to developers, and more – were, in fact, the innovations.
The Freedom Defense
Lawmakers also have been held back in their work by confusion about the relationship between knowledge and freedom. Surveillance capitalists are no different from other capitalists in demanding freedom from any sort of constraint. They insist on the “freedom to” launch every novel practice while aggressively asserting the necessity of their “freedom from” law and regulation. This classic pattern reflects two bedrock assumptions about capitalism made by its own theorists: The first is that markets are intrinsically unknowable. The second is that the ignorance produced by this lack of knowledge requires wide-ranging freedom of action for market actors.
The notion that ignorance and freedom are twinborn characteristics of capitalism is rooted in the conditions of life before the advent of modern systems of communication and transportation, let alone global digital networks, the Internet, or the ubiquitous architectures of Big Other. Until the last few moments of the human story, life was necessarily local, and the “whole” was necessarily invisible to the “part.”
Adam Smith’s famous metaphor of the “invisible hand” drew on these enduring realities of human life. Each individual, Smith reasoned, employs his capital locally in pursuit of immediate comforts and necessities. Each one attends to “his own security … his own gain … led by an invisible hand to promote an end which was no part of his intention.” That end is the efficient employ of capital in the broader market: the wealth of nations. The individual actions that produce efficient markets add up to a staggeringly complex pattern, a mystery that no one person or entity could hope to know or understand, let alone to direct: “The statesman, who should attempt to direct private people in what manner they ought to employ their capitals, would … assume an authority which could safely be trusted, not only to no single person, but to no council or senate whatever.”Footnote 80
The neoliberal economist Friedrich Hayek, whose work laid the foundation for the market-privileging economic policies of the past half century, drew the most basic tenets of his arguments from Smith’s assumptions about the whole and the part. “Adam Smith,” Hayek wrote, “was the first to perceive that we have stumbled upon methods of ordering human economic cooperation that exceed the limits of our knowledge and perception. His ‘invisible hand’ had perhaps better have been described as an invisible or unsurveyable pattern.”Footnote 81
In Hayek’s framing, the mystery of the market is that a great many people can behave effectively while remaining ignorant of the whole. Individuals not only can choose freely, but they must freely choose their own pursuits because there is no alternative, no source of total knowledge or conscious control to guide them. “Human design” is impossible, Hayek says, because the relevant information flows are “beyond the span of the control of any one mind.” The market dynamic makes it possible for people to operate in ignorance without “anyone having to tell them what to do.”Footnote 82
When it comes to surveillance capitalist operations, the classic quid pro quo of freedom for ignorance is shattered. The “market” is no longer invisible, certainly not in the way that Smith or Hayek imagined. The competitive struggle among surveillance capitalists produces the compulsion toward totality. Total information tends toward certainty and the promise of guaranteed outcomes. These operations mean that the supply and demand of human futures markets are rendered in infinite detail. Surveillance capitalism aims to replace mystery with certainty as it substitutes datafication, behavioral modification, and prediction for the old “unsurveyable pattern.”
The result is a fundamental reversal of the classic ideal of the “market” as intrinsically unknowable. Now the market is visible. As the head of Facebook’s data science team once reflected: “This is the first time the world has seen this scale and quality of data about human communication. For the first time, we have a microscope that … lets us examine social behavior at a very fine level that we’ve never been able to see before.”Footnote 83 A top Facebook engineer put it more succinctly: “We are trying to map out the graph of everything in the world and how it relates to each other.”Footnote 84 The same objectives are echoed in the other leading surveillance capitalist firms. As Google’s Eric Schmidt observed in 2010: “You give us more information about you, about your friends, and we can improve the quality of our searches. We don’t need you to type at all. We know where you are. We know where you’ve been. We can more or less know what you’re thinking about.”Footnote 85 Microsoft’s Satya Nadella understands all physical and institutional spaces, people, and social relationships as indexable and searchable: all of it subject to machine reasoning, pattern recognition, prediction, preemption, interruption, and modification.Footnote 86
Although there is nothing unusual about the prospect of capitalist enterprises seeking every kind of knowledge advantage in a competitive marketplace, the surveillance capitalist capabilities that translate ignorance into knowledge are unprecedented because they rely on the one resource that distinguishes the surveillance capitalists from traditional utopianists: the financial and intellectual capital that permits the actual transformation of the world, materialized in the continuously expanding architectures of Big Other. More astonishing still is that surveillance capital derives from the dispossession of human experience, operationalized in its unilateral and pervasive programs of rendering private experience as computational data.
This new condition unravels the economic justification for the triumph of raw capitalism: its free markets, free-market actors, and self-regulating enterprises. It suggests that surveillance capitalists mastered the rhetoric and political genius of the neoliberal ideological defense while pursuing a novel logic of accumulation that belies the most fundamental postulates of the capitalist worldview. It’s not just that the cards have been reshuffled; the rules of the game have been transformed into something that is both unprecedented and unimaginable outside the digital milieu and the vast resources of wealth and scientific prowess that the surveillance capitalists bring to the table. Surveillance capitalism’s command and control of the division of learning in society is the signature feature that breaks with the old justifications of the invisible hand and its entitlements. The combination of knowledge and freedom works to accelerate the asymmetry of power between surveillance capitalists and the societies in which they operate. This cycle will be broken only when we acknowledge as citizens, as lawmakers, as societies, and indeed as a civilization that surveillance capitalists know too much to qualify for freedom.
The Success Defense
A third propaganda strategy is the argument that the financial success of the leading surveillance capitalist firms reflects the real value they bring to people. In this view, financial success is prima facie evidence that no laws are required. But data from the demand side evident in a range of research conducted over the last decade and a half suggest a more disturbing picture. In forty-six of the most prominent forty-eight opinion surveys on the subject of privacy administered in the US and Europe between 2008 and 2016, substantial majorities support measures for enhanced privacy and user control over personal data. (Only two early surveys were somewhat less conclusive, because so many participants said they did not understand how or what personal information was being gathered.)
By 2008 it was well established that the more information people have about “Internet privacy practices,” the more they are “very concerned” about privacy.Footnote 87 A major 2009 survey found that when people were informed of the ways that companies gather data for targeted online ads, more than 73 percent rejected such advertising.Footnote 88 A substantial 2015 survey found 91 percent of respondents disagreed that the collection of personal information “without my knowing” was a fair trade-off for a price discount.Footnote 89 Fifty-five percent disagreed that it was even a fair exchange for improved services. By late 2019 the disjuncture was even more pronounced: An important survey from PEW Research reported that 81 percent of Americans believe the potential risks of companies’ data collection outweigh the benefits, compared to 66 percent who felt that way about government data collection.Footnote 90 A similar Swedish study in 2020 found that 55 percent of Swedes were most concerned about data collection by private companies, compared to 11 percent concerned about government data collection.Footnote 91
The surveillance capitalist firms typically dismiss these results, pointing instead to users’ actual behavior and the spectacular revenues it produces as justification for the status quo. Recall former Google CEO Eric Schmidt’s infamous 2009 privacy brushoff: “If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place.”Footnote 92 Scholars have called the gap between attitudes and behavior “the privacy paradox,” but there really is no paradox here, only the predictable consequence of the pitched battle between supply and demand expressed in the difference between what surveillance capitalism imposes on people and what they really want.Footnote 93
The data suggest that surveillance capitalism is best understood as a market failure that would not survive a genuinely competitive commercial environment. Few people who get a glimpse of surveillance capitalism’s hidden operations actually want to be their target. Most want an alternative path to the digital future, one that will fulfill their needs without compromising privacy and usurping epistemic rights. This is one of those disjunctures in economic history, when nearly everyone wants something that they cannot have, just as early twentieth-century farmers and shopkeepers wanted automobiles too, but at a price they could afford. Instead of a close alignment of supply and demand, people use surveillance capitalism’s services because they have no comparable alternatives and because they are ignorant of its shadow operations and their consequences. Corporate success is best understood as the result of coercion and obfuscation, practices that are only sustainable when they are conducted in secret.
Surveillance capitalism has thrived in the absence of law and regulation. Rather than mourning this state of affairs, the lack of prior action may be regarded as a positive. Democracy has not failed to reign in this rogue capitalism, it has simply not yet tried. And further, democratic societies have successfully confronted destructive forms of raw capitalism in the past, asserting new laws that tethered capitalism to the needs of people and democratic values. Democracy moderated some of the excesses of early industrialization. It ended the Gilded Age. It mitigated the destruction of the Great Depression. It built a strong post-War society. It protected earth, creatures, water, air, consumers, and workers.
According to Lawrence Friedman’s history of American law in the twentieth century, the appetite for new law and regulation in the 1930s came from decades of anger, frustration, outrage, and helplessness at the growing scale and complexity of the industrial behemoths.Footnote 94 Only law was up to the task of tethering the giant industrial corporations to the needs of a democratic society. The swell of survey data, the gradual awakening to surveillance capitalism’s mass communication tactics, and the drumbeat of novel legislative and regulatory discussions and initiatives appears to point in a similar direction. The question remains: What is to be done? What kinds of new law and regulation are likely to be effective? Will it be comprehensive privacy legislation? Will it be an antitrust approach, as many counsel? McCraw’s warning suggests that we need new economic, legal, and collective action paradigms born of a close understanding of surveillance capitalism’s economic imperatives and foundational mechanisms.
Privacy and antitrust law are vital, but there is reason to believe that neither will be wholly adequate to this new challenge. An example is privacy law’s call for “data ownership” and related data rights. Such formulations legitimate the original sin that is the theft of human experience for rendition into data in the first instance. All discussions that begin with existing data flows serve to institutionalize that theft. Negotiating data ownership is like negotiating how many hours a day a seven-year-old should be allowed to work in a factory, rather than contesting the fundamental legitimacy of child labor. Data rights also fail to reckon with the realities of behavioral surplus. Even if “users” achieve “ownership” of the data that they provided to a company, they will not achieve “ownership” of the behavioral surplus that floods the shadow text, the predictions gleaned from it, or the fate of those predictions in markets that trade in human futures. Finally, data ownership is a recipe for a new epistemic underclass, in which economically disadvantaged individuals, families, and groups sell their data in the same way that one might sell one’s organs in an illicit market. The Google contractors who induced homeless people of color to “sell” their faces for $5 offered a portent of that bleak future.
The prospect of “breaking up” the largest surveillance capitalist firms also fails to reckon with the actual mechanisms of this economic logic. Surveillance capitalists achieve scale by cornering behavioral surplus supplies and driving up the value chain for more predictive forms of surplus. If there are monopolies here, they are monopolies of behavioral surplus supplies, of scientific labor, and of material infrastructures for predictive analytics. These features do not correspond neatly to conventional monopoly criteria, neither do they reflect the conventional categories of “consumer harms” that most antitrust laws are designed to combat.
It is necessary to rethink the meaning of “size” and “monopoly” when a company with relatively few employees but huge capital reserves can corner data flows from large domains of human experience (e.g., “search” or “social networking”) while simultaneously cornering the capabilities to compute those data flows. Assistant Attorney General Makan Delrahim offered an initial analysis of such distinctions in 2019 noting: “Broadly speaking, in some digital markets, the competition is for user attention or clicks. If we see the commercial dynamics of Internet search, for example, in terms of the Yellow Pages that were delivered to our doors a generation ago, we cannot properly assess practices and transactions that create, enhance, or entrench market power – and in some cases monopoly power.”Footnote 95 Breaking up the largest surveillance capitalists – Google, Facebook, Microsoft, and Amazon – can address important anti-competitive problems, but without measures tailored to the actual mechanisms of surveillance capitalism, it will not prevent the emergence of smaller and more efficient surveillance capitalist firms, while opening the field for new surveillance capitalist competitors.
The most efficient legislative and regulatory strategies would be aimed at disrupting the surveillance dividend by disrupting the raw material supplies and financial incentives that sustain it. In other words, it means legislative and regulatory strategies that interrupt and in some cases outlaw surveillance capitalism’s mechanisms of supply and demand. Such measures would create space for alternative citizen-based and commercial action, building ecosystems that realign with individual needs and democratic practice.
Supply:
The relentless expansion of extractive supply chain operations is only likely to be constrained when the legal and regulatory focus shifts from data ownership and management to the originating processes of datafication: the secret theft of private human experience as free raw material for rendition into data. The boundary in dispute must move upstream, from contest over data as property to the codification of epistemic rights that link individual sovereignty to inalienable rights of self/knowledge and establish a rights-based moat around private experience.
This codification of epistemic rights to self/knowledge would interrupt data supply chains by safeguarding the boundaries of human experience before they come under assault from the forces of datafication. It assigns the choice to turn any aspect of one’s life into data as a right that adheres to individuals and communities by virtue of law sustained by democratic governance and as an extension of elemental rights now explicated and translated into juridical rights. This means, for example, that companies cannot claim the right to your face as you walk down the street, or use your face as free raw material for analysis, or fabricate, own, or sell any computational products that derive from your face and its depths of personal information. Such epistemic rights can be understood as the cornerstone of individual freedom under twenty-first-century conditions of existence.
The conversation on epistemic rights has already begun. In the absence of a comprehensive epistemic right to self/knowledge, for example, legal scholars, practitioners, and neural scientists have begun to frame epistemic rights claims to “freedom of thought.” They cite the sanctity of the “forum internum,” the interior space of human awareness and thought, and the need for codified individual rights that protect this domain of human experience from unwanted intrusion, theft, and manipulation.Footnote 96 Some of this work is a direct response to surveillance capitalism’s unrelenting drive to eliminate every barrier to the datafication of human experience, including recent breakthroughs in translating brain signals into speech.Footnote 97 Columbia University is home to a “neurorights initiative,”Footnote 98 the OECD issued formal recommendations for “responsible innovation in neurotechnology,Footnote 99 and Amnesty International issued a path-breaking report on the human rights implications of the “surveillance business model.”Footnote 100 As contests over epistemic rights multiply, there is a strong likelihood that surveillance assets will be reinterpreted as toxic “fruit of the poisonous tree” that can only be acquired at the price of fundamental epistemic rights.
Demand:
The opportunity on the demand side is to disrupt or eliminate the financial incentives that sustain the surveillance dividend. This can be accomplished with sanctions that outlaw the trade in human futures. This is not a radical prospect. For example, societies outlaw markets that trade in human organs, babies, and slaves. In each case, such markets are recognized as morally repugnant operations that produce predictably violent consequences and violate democratic principles. Human futures markets can be shown to produce equally predictable outcomes that challenge human freedoms, violate epistemic rights, and undermine democracy.
****************
Consider the Aware Home, a collaboration between computer scientists and engineers in the year 2000, intended as a “living laboratory” for the study of “ubiquitous computing.”Footnote 101 The project envisioned a “human-home symbiosis” in which animate and inanimate processes would be captured by an elaborate network of “context aware sensors” embedded in the house and by wearable computers worn by the home’s occupants. The Aware Home information system was designed as a simple closed-loop controlled entirely by the home’s occupants. Because the house would be “constantly monitoring the occupants’ whereabouts and activities … even tracing its inhabitants’ medical conditions,” the engineers concluded, “there is a clear need to give the occupants knowledge and control of the distribution of this information.” All the information was to be stored on the occupants’ wearable computers “to insure the privacy of an individual’s information.”
In 2017 University of London scholars published a detailed analysis of a single “smart home” device, the Google-owned Nest thermostat.Footnote 102 They determined that the purchase of a single Nest thermostat entails the need to review nearly 1,000 so-called “contracts,” each with its own burdensome and audacious terms of service for third-party data sharing.Footnote 103 Should the customer refuse to agree to Nest’s stipulations, the Terms of Service indicate that the functionality and security of the thermostat may be deeply compromised, no longer supported by the necessary updates meant to ensure its reliability and safety. The consequences can range from frozen pipes to failed smoke alarms to an easily hacked internal home system.Footnote 104
Today we might mourn the innocence of the Aware Home but, like a message in a bottle from a bygone age, it tells us something important. Once we were the subjects of our lives, now we are its objects. We once looked to the digital future as a human future – a place we could call home. The Aware Home is testimony to what we have lost and what we can find again: the rights to know and decide who knows and decide who decides who knows about our lives: Individual epistemic sovereignty, law, and democracy. Such rights and principles have been and remain the only possible grounds for human freedom, a functional democratic society, and an information civilization founded on equality and justice.
Open access