Visual languages and the problems with ideographies: A commentary on Morin

Abstract

Morin argues that ideographies are limited because graphic codes lack a capacity for proliferating standardization. However, natural graphic systems display rich standardization and can be placed in sequences using complex combinatorial structures. In contrast, ideographies are not natural, and their limitations lie in their attempts to artificially force a graphic system to behave like a writing system.

In “The puzzle of ideography” Morin argues that ideographic systems of pictorial signs are limited in their communicative capacities because “Graphic codes can only be standardized for a limited number of meaning–symbol mappings” (target article, abstract). Although we agree that ideographies are limited systems, we disagree with Morin's reasons why, especially the notion that the graphic modality itself is limited in its semiotic capacities. Specifically, ideographies attempt to artificially force a graphic system to behave like a writing system, which itself is an adaptation of the vocal modality into the graphic modality.

First, it is important to recognize that “standardization” or “conventionality” is orthogonal to symbolicity (Peirce, 1940). These notions are often conflated (de Saussure, 1972 [1916]), but standardization or conventionality is how much a signal is patterned and recognized across individuals. These signals (idiosyncratic or patterned) correspond to meanings through various interfaces characterizing their signification (i.e., iconicity, indexicality, and symbolicity). To clarify: A signal in sounds or graphics can be standardized or not, and how that signal corresponds to conceptual structures characterizes its signification(s).

Despite their stereotype as “arbitrary symbols,” spoken languages display all types of signification, as in Figure 1a (Clark, 1996; Ferrara & Hodge, 2018), and so does the graphic modality (Fig. 1b). Because all modalities use all types of signification, symbolicity is not the issue, but rather the question is about standardization.

Figure 1. (a) Vocal and (b) graphic signification, and standardized (c) components of drawings, (d) combinatorial signs, and scene templates (e) without and (f) with reference to other graphics.

Contrary to Morin's statements, the graphic modality displays voluminous standardization at multiple levels of complexity. Although small “pictographs” like hearts, stars, or peace or radiation signs are easy to recognize as standardized (as in ideographies), all graphics use standardized building blocks (Arts & Schilperoord, 2016; Cohn, 2013; Wilson & Wilson, 1977). Drawings are constructed from low-level visual patterns, such as how people draw eyes, headshapes, houses, and flowers, as in the conventionalized hands by three comic artists in Figure 1c. Graphics also use classes of combinatorial signs, such as the inventory of elements that float above characters heads or replace eyes, which use systematic and symbolic meaning-making (Fig. 1d), in addition to visual vocabulary like motion lines, impact stars, speech balloons, and other highly standardized, culturally variable, graphic codes. Although these small visual “morphemes” can combine into novel pictures, larger units can also be systematized, whether as templates of abstract scenes (Fig. 1e) or templates in reference to other scenes, which often invoke symbolic meanings (Fig. 1f). These observations contrast with the phenomenologically based idea that drawing and graphics are about articulating one's idiosyncratic vision of what one sees, despite this notion being unsupported by cognitive, cultural, and developmental research (Wilkins, 2016; Wilson, 1988) and having erroneous origins (Willats, 2005).

Visual representations also allow a range of conventionalized sequencing. Many patterns of two-unit sequences persist to show causative before–after relations, contrasts, or analogies (Schilperoord & Cohn, 2022). Longer sequences often provide visual lists of related images, such as what is allowed in a park or on an airplane (Cohn & Schilperoord, 2022). Visual narratives also use recursive combinatorial structures for sequential images displaying structural features of linguistic grammars, but operating at a higher-level information structure than the organization of nouns and verbs (Cohn, 2013; Cohn & Schilperoord, 2022). These visual narrative sequences are natural productions of sequential images, and the specific sequencing constructions they use have been shown to vary across cultures' comics (Cohn, 2019), again indicating culturally relative standardization, not universality. In addition, their processing invokes the same neural responses as linguistic syntax and semantics, and the understanding of these visual sequences requires proficiency that is acquired through exposure to and practice with those cultural graphic systems (Cohn, 2020).

In contrast to these natural graphic systems, ideographies use a basic lexicon of simple graphics attempting to have “word” levels of information, often created top-down by individuals, rather than emerging from a language community. These graphics are intended to be sequenced through a syntax, but because graphics do not naturally afford sentence-level combinatorics, they end up parasitic to spoken languages. Writing systems themselves are adaptations of the spoken into the graphic modality, but ideographies then assume graphics should behave like writing systems – forced into sentence-level sequences – to take on “linguistic” properties. This is why ideographies are largely invented by specific people (e.g., Blissymbolics, as in Morin's Fig. 4), because they do not proliferate instinctively through human history or cultures.

These invented ideographies are thus systems that attempt to mimic the structures of speech/writing which serve as people's reference point for what “linguistic graphics” should be like. However, this denies the affordances and linguistic properties already displayed by natural graphics in the first place. It should be no wonder then that ideographies don't work.

There is a clear analogue to this in the bodily modality. Sign languages are natural linguistic systems that optimize the affordances of the bodily modality and thereby do things in ways that differ from the structure of speech (Liddell, 2003). Yet attempts persist to force sign languages to have the properties of spoken languages, like Manually Coded English, which maps the lexicon and grammar of spoken English onto the body (Supalla, 1991). As adaptations of one modality to another, these systems deny the bodily affordances that natural sign languages display, just like ideographies deny the affordances apparent in natural graphic systems.

To conclude, ideographies are limited because they attempt to make the natural expressive graphic modality behave like writing, itself a conversion of the spoken modality into graphics. This quality undermines Morin's claim that “understanding why ideography has not worked in the past may help us understand how technology could make it work in the future” (target article, sect. 1, para. 8), because ideographies' unnaturalness will never “make it work” (target article, sect. 1, para. 8). Rather, this discussion raises the importance of investigating the affordances of all our modalities and their meaning-making capacities, and especially a greater integration of graphics into the study of the mind and cognition.