Is another punctum possible?

In this article I will take up Eric S. Jenkins’ (2013) suggestion that we need to learn from, but ultimately go beyond, Roland Barthes's concept of the punctum to describe a new kind of impersonal affective experience of animation. A new punctum needs to be a novel concept that grasps how animated images produce an illusion of movement by affecting perception and an ideological level of comprehension or miscomprehension. To achieve this aim we need to start, as Jenkins does, with Barthes's analysis of the photographic image. To understand how he arrived at the punctum, we need to imagine Barthes sitting at his writing desk, fixing his attention on an image, possibly with a blanket over his head, so as to block out any interference (Figure 4.1).

As he gazes deeply into the photograph, Barthes experiences something that goes beyond the subject of the image. His immersion in it is penetrated by the punctum. This is, as Jenkins points out, part of a now renowned distinction Barthes makes between the studium (the subject of the photo) on one hand, and the way the photo itself pierces the viewer's observation of the photo and in doing so transforms their perception of the image on the other. This rupture in the experience of the still image is considered, in Barthes's poststructuralist terms, as part of the language of the image: a punctuation in the speech images use to communicate with their audiences.

What is astute about Jenkins referencing of the punctum is his grasping of the opportunities Barthes seems to offer to go beyond a linguistic reading of still images, and address, as such, affective experiences, which might similarly pierce the perception and influence comprehension. This is a complex theoretical move, since although Barthes explicitly points to a media experience based on intensity rather than just detail (p 575), he distances his analysis from moving images. Indeed, as Jenkins notes, Barthes argues that the experience of photography is not at all like that of a moving image (p 576). In fact, one gets the impression that this distancing has something to do with Barthes's concern for the eventual death of the photographic image. It is certainly a mournful Barthes that we encounter here since he suggests that it is the new technology of movement that threatens the old punctum.

Introduction

Complex code and advanced algorithms are traversing nearly every part of our mediatized everyday life: automotive workshops do not only replace spare parts but run software to detect programme errors; search engines are not only useful sources of knowledge but evaluate and exploit every query; today's virtual assistants are smart conversation partners with a personality instead of simple programmes with very limited dialogue options. Therefore, research on algorithmic systems has become essential for understanding the world we are living and acting in. But code as part of a smartphone that accompanies us every second, longing for gaze and touch, and causes anxieties when untraceable, evoke different affective responses than code implemented in objects that demand little or no interaction. Always implemented in a processing machine that we encounter, there is no such thing as pure code without matter. This also applies to video games, which are not only simple programmes but programmes running on platforms with specific controls, captivating our senses with their multifaceted output. Alexander Galloway (2006) describes video games as an action-based medium consisting of machine and operator actions, both equally important for the gaming experience: ‘People move their hands, bodies, eyes, and mouths when they play video games. But machines also act. They act in response to player actions as well as independently of them’ (p 4). The human encounter with game hardware and software takes place on eye level instead of being controlled or dominated by the operator input: ‘Humans do not simply manipulate or control machines, data, and networks any more than machines, data, and networks simply manipulate or control us’ (Paasonen et al, 2015:2).

This change of perspective has brought algorithms more and more into the focus of research. Nevertheless, the machinic act in form of processing algorithms should not be overstated in the experience with video games. In her book Playing with Feelings, Aubrey Anable (2018) criticizes the current fetishization of code in combination with the simultaneous neglection of the audiovisual output and haptic quality, that stimulates our senses, in game studies. Her critique is a reaction to the formalist approach of proceduralism which privileges strategically utilized code and computational architecture over its actual realization through hardware coming together with a sensitive and active subject. This transforms the player into a mere button-pushing entity, following the predefined interaction system of the game designer.

Mud mood

An observation on human existence today might benefit from hurtling backwards to some sort of origin, mythic or not. We are golems, mud people, inanimate matter that had life breathed into it, once long ago. We crawled from the mud. We shaped it around us eventually to make a shelter. Now we might think we are elevated above it, have asphalted over it or fenced it off. Mud is from another time, if not another place. Mud is not conceived of as modern, as the stuff that builds up our digital worlds. Mud surrounds us still, but it is not meant to figure in our futurist dreams, our fully automated futures, our clean technologies or shiny screens and interfaces that are all we are meant to desire. The opposite of mud must be the liquid crystal. Mud is opaque, unlike liquids and crystals. Mud is dirty, unlike our dreams of liquids and crystals. Mud is brown and sticky, unlike our flowing liquids, our translucent crystals. Mud must be the contradiction of the liquid crystal environments that illuminate our cities, our homes, our own bodies – greeny, bluey, cool whitish light reflected chin-ward – which are not brown, opaque, sticky, dirty. But sometimes mud floods in on us. We find ourselves stuck in the mud, actually or metaphorically. Landslides, mudslides occur periodically, the result of development, the stripping away of trees, encouraged by extreme weather events. But mud sticks to us in another way, is our inheritance, is what we came from but cannot brush off easily. To start in the mud, mud-bound, in any reflection, is to evoke a primal state before our time, when there was only mud. This mud was a context for life and, indeed, has been found in its kaolinite form to hold traces of the soft, microbial life of 540 million years ago, which did not leave fossils, because it lacked hardness. In time, life hardened, and in time too, mud became clay, became the vessels and objects of something called civilization.

Mud is old, but mud is still present, is still our ground, even if we rarely touch it. To reflect on it allows an acknowledgement of how much there is, in our time, in theory, a love affair developing with matter and materiality, with its capacities, its possible liveliness, its unpredictability, its self-orientations, that can be mobilized for us.

So Captain George Smith, who was appointed Inspector of the Royal Military Academy at Woolwich in 1772, described the use of the verb ‘to animate’ in his An Universal Military Dictionary (1779). In late-eighteenthcentury military jargon, ‘animation’ responded to the needs of galvanizing soldiers to overcome their fears when facing a battle, or of maintaining troops in ordered formations in the fog of war. Somewhat paradoxically, animation meant enlivening, indeed, giving life to individuals in the face of their potential death. Smith was undoubtedly aware of the word's etymological roots in Latin animare, which Samuel Johnson's (1755) English dictionary described as ‘to quicken; to make alive; to give life to’, as well as ‘to give powers to; to heighten the powers or effect of any thing’, and ‘to encourage; to incite’. But etymology aside, noteworthy here is how a couple of hundred years before theorists like John Langshaw Austin or Gilles Deleuze, Captain Smith recognized how linguistic utterances could shape souls and transform states of affairs. By the sheer power of discourse, Smith contended, bodies could be rendered as elastic, strong, and speedy; by the power of discourse, they could be made to perform tasks they were not able to or did not want to do. By means of what Smith called ‘animation’, individuals could be programmed to become operative units of the war machine.

Fast-forward about 240 years: commanders’ enlivening elocutions have been replaced by more mundane logistical operations of images and sounds. We are familiar with how contemporary war machines abound with digital visuals of all sorts, especially computer-generated animations circulating on various video games and virtual reality platforms. Their programming functions have expanded from elevated cries as incitements to battle, to encompassing the whole cycle of warrior production from recruitment to post-combat therapy.

‘Deepfakes’ is the popular word term for faked videos (synthetic audiovisual media productions) produced by using deep learning techniques (artificial neural networks). According to Deeptrace, an Amsterdam-based company that provides tools to detect deepfakes, in 2019 there were 14,608 deepfake videos online, which drew more than 134 million views (Ajder et al, 2019:7). There is no doubt that the phenomenon generates fear and resistance. In November 2018 the Guardian published an article with the headline, ‘You thought fake news was bad? Deep fakes are where truth is going to die’ (Schwartz, 2018). The American government is working to ‘combat the spread of misinformation through restrictions on deep-fake video alteration technology’ (Clarke, 2019). The recent, rapidly accelerating developments in the field of artificial neural networks have made it possible for ordinary users to easily play with face-swaps. The Zao app, which is currently available only in China, enables its users to digitally swap faces with movie actors. The seamless integration and growing accessibility of face-swap technologies generate fear about their abuse: for example, in connection with fake news, advertisements, or hate crime strategies. The fear is that epistemic claims (‘He said this’, ‘She did that’) in a medium that is historically associated with authenticity may affect people differently from other media (writing, audio, still images). Deepfakes are not used primarily for fake news, but to create non-consensual pornography. According to Deeptech, 96 per cent of the deepfake videos online have sexual content (Ajder et al, 2019:7). American researcher Danielle Citron, an internet hate crimes expert, states, ‘Deepfake technology is being weaponized against women by inserting their faces into porn. It is terrifying, embarrassing, demeaning, and silencing. Deepfake sex videos say to individuals that their bodies are not their own and can make it difficult to stay online, get or keep a job, and feel safe’ (Ajder et al, 2019:6).

Just as it is illegal to write falsehoods about people, the United States Congress states that each and every person should be protected from being falsely visually presented (Clarke, 2019). Scientists, lawmakers, and journalists seem to concur in their criticism of deepfakes, but as Danielle Citron stated during a hearing, ‘A ban on deep fake technology would not be desirable. Digital manipulation is not inherently problematic. There are pro-social uses of the technology.

Animation is an art of affect, and not only in the human register of enticing feelings of nostalgia, marvel, or wonder but in the nonhuman or more-than-human one as well, because animation experiments in practices of affecting and being affected, of pen to paper, ink to celluloid, hand to clay, photo to projector, light to shadow, colour to colour, and, today, line of code to line of code, or digital model to 3D simulated environment. Each of these relations of affecting and being affected pose implicit questions. How might light and shadow be related such as to convey a sense of three-dimensionality? How might figures be drawn that enable an illusion of movement and even life? How might photographs be taken to enable flat drawings to appear as if a moving figure occupies the foreground of a more distant background? The evolution of animation is the discovery of different answers: shading, squash and stretch, the multiplane camera.

Although CG animation borrows many techniques and practices from the history of hand-drawn and stop-motion animation, it also introduces new questions. Indeed, CG animation presents Spinoza's famous question in stark relief: what can a body do? CG animators must ask, first, how can a digital body (model) be rigged? What are its potential relations of movement and rest, to invoke the ‘longitudinal’ aspect of bodies understood as modes outlined by Deleuze (1988) via Spinoza? Second, how might these digital bodies relate to the three-dimensional simulated environments? What are the capacities to affect and be affected that constitute the ‘latitude’ of modal relations? The spectacular power of CG animation rests in the breadth of potential answers to these questions, in the fact that relations of movement-rest and affective capacities are less limited by the laws of physics because the animator works in the realm of the virtual, first constructing potentials that will become actualized by the rendering of computer software. Whereas drawn and stop-motion animation begin from the actual (character sketches, material objects), CG animators begin with the virtual, modelling a range of potential movements and crafting rules of affection that will serve as the physics within which the characters and animators work. Analogue animators, in contrast, must work with rather than craft the laws of physics, bound as they are to material actualities.

Introduction

The chapter attempts to examine the (human) reception of moving images by considering the role of ‘affect’ – ‘the affective psycho-sensory complex’ – which people simply feel but find difficult to articulate in language. The paper focuses on the differences between digital aesthetics created by computer animation and analogue aesthetics in hand-drawn animation. Both are different ways to provide representations of reality. The development of 3D computer animation has sought to produce photorealistic images and brought about broader possibilities for creating verisimilar naturalistic films. In contrast, analogue animation, especially ‘conventional’ hand-drawn animation, depends on expressive form. The aim is not to duplicate or produce photographic reality, rather to appeal to our emotions, sensation, memory, imagination, and fantasy.

Recently, computer-generated imagery has become a part of the fabric of social life, from computer games to medical data images, from military training simulation to stock market data visualizations, from various advertisements to private photoshopping. Contemporary society is unable to escape the continuous flows of millions of computer-generated images that appear on ubiquitous screens. These images restlessly stimulate and influence our perception. The prime question here is the extent to which machine-generated images influence human embodied practices and feelings, such as experiences, emotion, sensation, and judgement.

The animation industry has often been characterized as a central part of the ‘fantasy factory’ since its emergence, with this feature further emphasized today through the use of advanced computer technologies. Yet hand-drawn animation still attracts a considerable audience and is often appraised as an art form that involves a sophisticated capacity to express human emotions and senses. Here, attention will focus on the influential Japanese animator Hayao Miyazaki.

Conventional animation depends on an ordered sequence: hand-drawing images on cells; placing the set of image cells under the camera; and shooting a whole sequence of frames. With computer-generated imagery (CGI), the rendering process involves pixels and mathematical calculations for lighting, staging, framing, performance, and camera movements. Digital figures do not need to have a substantial form in the real world, relying on pixels and mathematical formulas for their existence. All these processes ‘are carried out within the computer’ (Furniss, 2007:174).