Make it or draw it? Investigating the communicative trade-offs between sketches and prototypes

Abstract

Design representations play a crucial role in facilitating communication between individuals in design. Sketches and physical prototypes are frequently used to communicate design concepts in early-stage design. However, we lack an understanding of the communicative benefits each representation provides and how these benefits relate to the effort and resources required to create each representation. A mixed-methods study was conducted with 44 participants to identify whether sketches and physical prototypes led to different levels of cognitive load perceived by a communicator and listener and the characteristics that shape their cognitive load during communication. Results showed that listeners perceived higher levels of mental and physical demands when understanding ideas as low-fidelity physical prototypes, as compared to sketches. No significant differences were found in the cognitive load levels of communicators between the two conditions. Qualitative analyses of post-task semi-structured interviews identified five themes relating to verbal explanations and visual representations that shape designers’ cognitive load when understanding and communicating ideas through design representations. Results indicate that designers should be aware of the specific objectives they seek to accomplish when selecting the design representation used to communicate. This work contributes to the knowledge base needed for designers to use design representations more effectively as tools for communication.

1. Introduction

Owing to the highly social nature of engineering design (Baxter & Sommerville 2011), designers must leverage the tools available to them to communicate effectively. At every stage of the design process, designers are likely to engage in some forms of social interaction, such as collaborating with their team members (Kim, Kim, & Kim 2007), interacting with stakeholders (Lauff et al. 2020) or obtaining feedback from users (Siu 2003). Prior research has highlighted that effective communication between these individuals is a key determinant of design teams’ success (Hales & Gooch 2004). One of the tools used by designers to facilitate communication is a design representation. These representations, such as sketches and prototypes, have been referred to as “boundary objects” (Bucciarelli 2002). Boundary objects were defined by Star & Griesemer (1989) as “objects that are both adaptable to different viewpoints and robust enough to maintain identity across them.” During communication, these boundary objects allow designers to bridge divides caused by differences in knowledge both within design teams and between designers and stakeholders (Bucciarelli 2002). As a result, these design representations support communication by allowing designers to communicate design ideas (Gerber & Carroll 2012), justify design decisions (Lim, Stolterman, & Tenenberg 2008) and promote stakeholder buy-in (Nelson et al. 2019).

There are different representations in design, each with its own spatial properties, level of detail and purpose (Pei, Campbell, & Evans 2011). Prior research suggests that the modality of representations influences both creativity (Toh & Miller 2014) and analogical reasoning capabilities (Linsey, Wood, & Markman 2021) of designers during idea generation. The current work investigates the effect of representation modality, namely sketches vs. physical prototypes, on design communication. Each design representation may offer unique benefits in the context of design communication. Prior work points to the benefits of physical prototypes in facilitating communication, as they allow for tactile engagements (Brandt 2007), are useful for obtaining feedback and spurring discussion (Isa & Liem 2021) and “sell” a design concept (Elverum & Welo 2014). However, compared to a sketch, a physical prototype can be more challenging to create due to the resources and effort involved in physical manufacture (Nelson & Menold 2020), even at low fidelities (Nolte & McComb 2020). Designers often face situations where a representation may need to be rapidly generated to communicate a design idea. In these situations, the ability of sketches to be quickly generated (Martin-Erro, Dominguez, & del Mar Espinosa 2016) may make them a more viable communication tool. Designers often communicate with one another and other stakeholders with the intent of achieving shared understanding, which consequently improves design team performance (Mathieu et al. 2005). Synthesizing this past work, we note that the costs to create an artifact to communicate, such as the time and materials needed to build a prototype, may not result in significant communicative benefits. While this past work demonstrates that trade-offs exist between the costs of creating a representation and the communicative benefits gained from it, the specific nature of the relationship between facets of the representation and outcomes of the communicative act remains unclear.

In this work, we specifically focus on the cognitive resources expended by individuals during communication. While a shared understanding facilitates negotiation between individuals, it is a process that requires individuals involved in communication to expend cognitive resources to construct mental models of the presented design information (Mayer 2005). Gaining a deeper understanding of how design representations shape the cognitive resources used by listeners and communicators involved in communication would allow designers to weigh the communicative benefits of specific representations versus the costs involved in creating them. To this end, this work seeks to equip designers with the knowledge needed to leverage design representations as communication tools. Consequently, this would improve design communication between stakeholders and increase design team performance.

2. Literature review

2.1 Design representations as communication tools

Henderson (2014) categorized engineering design as a “visual culture” that inherently involves the externalization of design concepts into visual representations. These representations, such as sketches and physical prototypes, are crucial representations that allow designers to physically manifest design information (Ferguson 1977). Subsequently, when interacting with other designers and stakeholders, these design representations become “vehicles” of design information (Eckert & Stacey 2000) and allow designers to communicate design concepts and rationale (Lim et al. 2008). Recent work has highlighted how design representations support communication within and outside design teams across various social situations with different stakeholders, underscoring the criticality of these representations to effective communication (Lauff et al. 2020).

A sketch is a commonly used design representation, and sketching has been described as a process of visual thinking and reasoning that allows designers to externalize and reflect on design concepts during early-stage design (Schmidt, Hernandez, & Ruocco 2012). Sketching acts as an accessible method of visualizing design concepts regardless of sketching ability, as prior work has shown no relationship between how well a designer sketches and their design outcome (Yang & Cham 2006). However, not all sketches are effective communication tools. For instance, some sketches may be ambiguous in nature (Ferguson 1977), as designers may visualize only certain information through sketches while storing other information internally (Yang & Cham 2006). While this ambiguity may spark reflective conversations and creativity, it may also make the represented idea less comprehendible or more challenging to understand by other individuals (Self 2019).

In addition to sketches, low-fidelity prototypes are commonly used design representations in early-stage design (Gerber & Carroll 2012). The physical nature of prototypes makes them a design language of their own (Yang 2005), and the tangibility of physical prototypes elicits different types of communicative patterns as compared to sketches (Oviatt, Coulston, & Lunsford 2004). Physically realizing a design concept allows for the identification of new knowledge and the facilitation of communication between designers and stakeholders (Jensen, Elverum, & Steinert 2017). Deininger et al. (2019) compared how stakeholders provided feedback when communication was supported with physical prototypes, computer-aided design (CAD) models and sketches. They found that acquiring feedback using physical prototypes led to stakeholders giving more extended responses, more useful design input and backing their opinions with justifications. However, while prototypes act as a tangible medium through which communication is supported, researchers have acknowledged the additional “costs” involved in prototyping. Prototyping is one of the most expensive design acts in terms of monetary and material resources (Nelson & Menold 2020). Furthermore, prototyping may induce design fixation due to the sunk cost involved in realizing physical models (Viswanathan & Linsey 2011). Cognitively, prototyping can be even more expensive, as prior work suggests that creating a physical model of a design concept induces more stress than other design tasks, such as concept generation or selection (Nolte & McComb 2020).

Prior research exploring the differences between design representations has focused on individual designer cognition during the creation of the representation. For example, researchers have established how prototypes and sketches differ in their effects on designers’ exploration of a design space exploration (Bao, Faas, & Yang 2018), users’ perceptions of novelty (Häggman et al. 2015) and designers’ fixation on certain ideas (Viswanathan, Tomko, & Linsey 2016). In this work, however, we focus on the cognitive experiences of individuals during a communicative act and seek to understand how these cognitive experiences differ between design representations, namely sketches and low-fidelity physical prototypes.

2.2 Cognitive theory of multimedia learning and cognitive load theory

This work is grounded in the theories of Mayer’s cognitive theory of multimedia learning (CTML) (Mayer 2005) and Sweller’s cognitive load theory (CLT) (Sweller & Chandler 1991). CTML proposes a comprehensive framework that details how individuals simultaneously process information through spoken words and pictorial media. CTML is based on three core tenets – active processing, dual processing and limited cognitive capacity. The first tenet, active processing, states that humans are active processors who process multimedia material with the intention of selecting and organizing information and then using this information to create cogent mental representations.

The second tenet, dual processing, similar to other theories of information processing such as Baddeley’s model of working memory (Baddeley 1992) and Paivio’s dual coding theory (Paivio 1990), states that individuals process information through two distinct channels (Figure 1). First, information is separately processed through the individual’s sensory systems – ears in the case of spoken words and eyes in the case of printed words and pictures. Next, the individual selects relevant verbal and visual information required to construct knowledge and transfers this information to their working memory. Here, the individual organizes the verbal and visual information separately to construct verbal and pictorial models. For instance, in the case of a verbal model, the individual may organize a text describing the stages of a phenomenon to indicate which stage activates another. Finally, the verbal and pictorial models are integrated. In other words, cognitive resources are expended to create connections between verbal information and the corresponding pictorial information, and the final mental model is obtained and integrated with prior knowledge.

Figure 1. Mayer’s Cognitive Theory of Multimedia Learning (Mayer 2005).

The third tenet of CTML, limited cognitive capacity, states that the processes of selecting words, organizing them and constructing a final mental model require the individual to expend cognitive resources. Each channel has a limited cognitive capacity, and if the resources required during information processing exceed this capacity, the individual is likely to experience cognitive overload and is subsequently unable to process any additional information effectively. This tenet of CTML is drawn from Sweller’s CLT (Sweller & Chandler 1991). CLT states that individuals have finite cognitive resources. If the cognitive load experienced (or cognitive resources expended) by an individual is more than their cognitive capacity, cognitive overload is induced (Sweller & Chandler 1991). Cognitive overload is associated with reduced task performance (Haji et al. 2015) and stress (Nguyen & Zeng 2014), and minimizing cognitive load that does not contribute to learning is critical to developing coherent mental models (Mayer et al. 1999). Sweller (2010a) states that there are three facets of cognitive load: intrinsic load, which is determined by the complexity and element interactivity of the task at hand; extraneous load, which is determined by the format through which information is presented; and germane load, which refers to any remaining cognitive resources available for constructing schema in long-term memory. However, it should be emphasized that only intrinsic and extraneous loads contribute to cognitive load (Sweller 2010b).

A wide body of prior literature highlights the relationship between CLT and CTML. For example, Tabbers, Martens, & Van Merrienboer (2001) showed that students perceived their mental workload to be lower when information was presented as pictures and spoken word (bimodal) as compared to pictures and printed text (unimodal). From the perspective of CTML, the lower mental effort in the bimodal group can be attributed to the cognitive resources being divided between the auditory and visual channels, rather than using only the visual channel, as was the case of the unimodal group. From a CLT perspective, the cognitive load experienced by participants in the unimodal group is extraneous, as it is driven by the format of the instructional material and does not contribute to learning. A similar result was found by Mayer & Moreno (1998) in their work studying the retention abilities of students who were asked to learn the functioning of a car’s braking system. Students who were presented with narration and pictures performed better at recalling steps of the process as compared to those presented with on-screen text and pictures. This outcome was likely driven by learners being unable to select relevant information due to their visual channel being overloaded and subsequently not having enough cognitive resources to make connections between words and pictures and generate a coherent mental model.

2.3 Design communication and cognitive load

We draw a parallel between CTML and CLT and how communication in design takes place between individuals through design representations. Design communication is often bimodal, as designers combine design representations (such as sketches and prototypes) with verbal information to communicate design concepts (Bracewell & Wallace 2003). The listener must select and organize the presented information and construct a mental model of the design concept being communicated with the intention of developing a shared understanding with the communicator (Maier, Eckert, & Clarkson 2005).

In this work, we focus on how differences in visual information, or the modality of the design representation, influence the process of communication between two individuals. In particular, extending theories of CTML and CTL to design communication, we investigate the differences in the cognitive experiences of communicating design concepts with sketches and low-fidelity physical prototypes. Being three-dimensional representations, it is possible that low-fidelity physical prototypes lead to a reduction in cognitive resources needed to imagine an object in a three-dimensional space, as the listener can interact with and manipulate the object directly (Dunn & Risko 2016; Risko & Gilbert 2016). At the same time, however, Huk (2006) argues that information processing through three-dimensional objects may involve greater cognitive resources, as additional spatial information is processed in the visual channel of the listener. The processing of additional visual information may also occur in the case of sketches. It is known that annotations (printed text) are often used as accompaniments to sketches (Rodgers, Green, & McGown 2000), and listeners may need to use cognitive resources to process this textual information through their visual channel. Furthermore, prior literature has highlighted the benefits of animations over static imagery when presenting visual information, as animations allow listeners to create connections between visual and verbal mental models (Kühl et al. 2011). However, sketches are often static, and it is possible that communicators and listeners may need to use greater cognitive resources to communicate and understand how different components of a design concept move with respect to one another.

While prior work from other fields, such as instructional design, has investigated the differences between two-dimensional and three-dimensional representations of information processing, the contexts of these studies differ significantly from engineering design. For instance, Foo et al. (2013) found that medical students were able to localize anatomical structures more effectively with lower cognitive effort when using three-dimensional (3D) visualizations as compared to two-dimensional (2D) visualizations. However, the 3D visualizations used in the work by Foo et al. (2013) were 3D representations on a digital display and not physical objects – the latter of which is the focus of this work. Pillay (1998) found that assembly tasks were accomplished the fastest when a physical object was used as a reference and attributed this to the lower extraneous load associated with encoding information through physical objects. However, the participants in Pillay’s work were 14-year-old children who may have different cognitive processes compared with designers – even those at the novice level. The work closest to ours was by Dadi et al. (2014), who asked participants to reconstruct a simple structure presented to them as 2D drawings, 3D CAD models and 3D physical models and found no differences in participants’ perceived cognitive demands between representations.

No studies, to date, have compared how three-dimensional physical prototypes and two-dimensional sketches differ with respect to the cognitive resources expended by designers either when communicating or understanding a design concept. By addressing this gap in the literature, we will provide designers with the knowledge needed to appropriately select a design representation to effectively build a shared understanding with stakeholders, without expending unnecessary cognitive resources. Synthesizing the literature above, we aim to answer the following research questions (RQs) in this work:

RQ1: How does the modality of a design representation, specifically two-dimensional sketches and three-dimensional low-fidelity physical prototypes, affect the cognitive load of the communicator and listener when communicating design concepts?

RQ2: What characteristics of design representations alleviate or add to the perceived cognitive load of the communicator and listener when communicating design concepts?

3. Methods

To answer our RQs, a controlled laboratory-based mixed-methods study was conducted at The Pennsylvania State University. The study was conducted in dyads and simulated the communication of a design concept between two individuals. Each participant was asked to create a representation of a design concept (a sketch or prototype depending on the condition) and use this representation to explain their design concept to the other participant in the dyad. The aim of this experimental design was to simulate the act of a single designer communicating a new design concept to a colleague; in particular, if we imagine an engineering firm working on multiple projects, we can envision a designer discussing their project with a colleague and sharing their ideas to solicit feedback on the concept itself. Following the construction of the design representation, a survey to query the participant’s explanations of their own design concepts was distributed. In addition, a survey to capture the cognitive load involved in communicating was distributed after each communicative act. The listener’s survey also included questions to capture perceived paradigm-relatedness and the constructed mental model of the presented design concept. These variables will be reviewed in detail in the next section. Finally, a semi-structured interview was conducted with each participant. The experimental procedure followed in the study is shown in Figure 2.

Figure 2. Experimental procedure of the study.

3.1 Metrics

Cognitive Load : Cognitive load theorists have established that subjective mental workload can be used as an index for the actual cognitive load imposed during a task (Paas & Van Merrienboer, 1994). Subjective mental workload is defined as the perceived load, or effort, experienced by an individual’s cognitive system when performing a task (Paas & Van Merriinboer, 1994). Furthermore, the use of subjective scales has been shown to be more accurate and sensitive and is easier to administer compared with more intrusive physiological measures of cognitive load (Paas & Van Merrienboer, 1994). Among subjective measures, most prior work in engineering design research has used the NASA Task Load Index (TLX) instrument (Hart & Staveland 1988) to assess the perceived mental workload of design tasks. The instrument consists of six scales – mental demand (the mental and perceptual activity required by the task), physical demand (the physical activity required by the task), temporal demand (the time pressure felt during the task), performance (how successfully the individual thought they were accomplishing the task), effort (how hard the individual worked to accomplish their level of performance) and frustration (how insecure, discouraged or irritated the individual felt during the task). The respondent is required to rate, on a scale of 0 to 20, the perceived load in terms of the six constructs described previously. In this work, we use the ratings on the individual constructs in our analysis.

Paradigm-Relatedness: The cognitive load experienced during a task is known to be dependent on the amount of prior knowledge relevant to the task being completed (Mayer & Moreno 2010). Fewer cognitive resources are expended when relevant prior knowledge is integrated into the working memory during the formation of a mental model, as compared to the cognitive resources expended without any prior knowledge (Kirschner 2002; Cook 2006). In this study, relevant prior knowledge refers to possessing a mental model of the presented design solution. We posited that building a mental model of a common design solution a listener already knows about would require fewer cognitive resources as compared to building a mental model of a design solution they have no knowledge about. As a result, any observed differences in cognitive load could be attributed to the possession of knowledge of the presented design solution, rather than the modality of the design representation.

To capture prior knowledge about a design solution, we used the concept of paradigm-relatedness. Paradigm-relatedness examines the perception of a design solution within the boundaries of the original design problem and captures elements such as the surprise or novelty of a design concept (Silk et al. 2018). In other words, an existing and common solution that operates within the constraints of a design problem would be “paradigm-preserving.” While an unexpected solution that breaks all constraints of the problem would be “paradigm-breaking.” In the study, listeners were asked to categorize the solution presented to them into one of three levels of paradigm-relatedness, namely paradigm-preserving (solution resembles an already existing, common design, stays well within constraints defined by the problem as given and typical assumptions), paradigm-challenging (a solution that integrates an uncommon element or relationship, or begins to stretch the boundaries of the problem) and paradigm-breaking (solution violates all boundaries of the initial design problem and shifts focus of the problem to a larger problem) (Silk et al. 2018).

Idea Complexity: As posited by CLT, there exist two sources of cognitive load – intrinsic cognitive load (ICL) and extraneous cognitive load (ECL) (Sweller 2010b). While ECL is induced by how the material is designed, or the media through which information is presented, ICL is determined by the complexity of the information being processed. As ECL and ICL cannot be measured independently, differences in one can only be identified by controlling the other. In the context of this work, it then becomes important to measure and control for any differences in the complexity of ideas that were communicated (ICL), to isolate the differences in cognitive load between the types of representation (ECL).

Idea complexity was calculated through a two-step process. First, each participant’s explanation of their idea from the survey was converted to a functional structure using the functional structure taxonomy by Stone & Wood (1999). Once each participant created their sketch or prototype of their design concept, they completed a survey where they were asked to describe the design problem they solved, their generated design solution and how their solution works. This written explanation from the survey was used as the basis for the functional structure of each design concept rather than participants’ verbal explanations to one another. This is because during verbal communication, participants may use demonstrative pronouns such as “this goes there,” which makes it challenging to extract specific functions of the design concept. We do emphasize, however, that the researcher conducting these studies specifically instructed participants to keep their verbal explanations similar to the written ones they had just provided. To validate the generated functional structures, the first and second authors reviewed 20% of the dataset and discussed the functions and flows of the functional structures, how accurately they represented the written explanations and the consistency between different functional structures. The second author is proficient in generating functional structures and has published multiple papers on the use of functional models to identify potential human errors in user–product interactions (Soria Zurita et al. 2018; Soria Zurita et al. 2022, 2019). Any changes from the second author were discussed during the review of functional structures. The first author then reviewed all the functional structures again to incorporate said changes consistently.

Following the creation and validation of the functional structures, the size complexity of each functional structure was calculated using the method described by Ameri et al. (2008). In their work, Ameri et al. define size complexity as the “information content contained within a representation” (p. 165). Using the number of functions and flows present in each functional structure, size complexity is calculated through equation (1). Table 1 shows examples of written explanations from communicators, the generated functional structures and the associated complexity scores.

(1) $$ \mathrm{C}{x}_{\mathrm{size}\_\mathrm{func}}=\left(\mathrm{Dv}+\mathrm{Dr}\right)\mathrm{xLn}\left(\unicode{x03C1} +\unicode{x03BD} \right) $$

Table 1. Examples showing the conversion of explanations to function structures and associated complexity scores

where

Dv = number of instances of function blocks and I/O types.

Dr = number of instances of primitive relations.

ρ = the number of primitive modules (operands) available within the representation (35, as there are 35 possible functions in the taxonomy).

ν = the number of primitive relationships (operators) available between all available modules (3, as there are three I/O types, namely material, energy and information).

3.2 Participants

Forty-four participants (22 men and 22 women) completed this study, all of whom were enrolled in the College of Engineering. Students in the College of Engineering at The Pennsylvania State University complete a series of design courses in which they are exposed to design thinking methods, solve various design problems and are trained to develop design representations such as sketches and prototypes. Participants were recruited through purposeful sampling methods, such as reaching out to students enrolled in undergraduate design classes through mass emails and flyers and snowball sampling methods, where each participant was asked to inform their peers about the study. Fourteen participants were graduate students, and 30 participants were junior- or senior-level undergraduate students. Twenty-eight participants identified as White, 10 identified as Asian, 2 identified as Hispanic, Latino or of Spanish origin, 1 identified as Black or African American, 1 identified as White and Asian, 1 identified as Middle Eastern or North African and 1 identified as White and Hispanic, Latino or Spanish origin.

3.3 Procedure

Participants were paired together to complete the experiment. Figure 2 graphically depicts the experiment groups and procedure. Pairs were randomly generated, and each pair was randomly assigned to one of two groups: prototyping and sketching. To start the study, both participants were brought into the same room. In accordance with Institutional Review Board Guidelines, participants were introduced to the study and were informed that their participation was voluntary. Participants then took a short pre-survey where they generated their unique participant ID.

In an initial pilot study with 14 participants, participants also completed the visuospatial ability test by Peters et al. (Peters & Battista 2008) in the pre-survey. Prior research has highlighted how an individual’s visuospatial ability determines the amount of cognitive resources they need to form connections between presented information and mental models of said information (Huk 2006). As compared to an individual with high visuospatial ability, an individual with lower visuospatial ability would require greater cognitive resources to create mental models of presented visual or spatial information and, as a result, be more prone to cognitive overload. The visuospatial ability test was administered to ensure that individual differences in visuospatial ability were not confounding any results. However, no significant relationship between visuospatial ability and cognitive load was found during pilot testing, and this test was removed from the protocol for the remainder of this research. Following the pre-survey, participants were briefed about the design task. Participants in the prototyping condition were allowed to generate sketches initially as prior research has noted the detrimental effect on problem-solving when designers prototype solutions without sketching (Bao et al. 2018). As physical modeling is known to be more time-consuming than sketching (Viswanathan & Linsey 2010), participants in the sketching and prototyping conditions were provided with 20 and 45 minutes, respectively:

“You will now be given a design task to complete. You can use as many materials that are given to you and you have forty five minutes to complete the task. You can feel free to sketch out as many ideas as you want, but you will only be allowed to bring your final prototype with you when explaining your design solution. Your final design can be a single idea or a combination of your ideas generated.” (Prototyping condition)

“You will now be given a design task to complete. You have twenty minutes to complete the task. You can feel free to sketch out as many ideas as you want, but you will only be allowed to bring your final sketch with you when explaining your design solution. Your final design can be a single idea or a combination of your ideas generated.” (Sketching condition)

Following this, participants were led to different rooms, where they completed the design tasks. Participants were in different rooms to ensure that they would not be able to see each other’s design solutions before they explained their solutions to one another. Participants were then provided their design prompts; each participant was given a different design prompt than their partner. This was done to ensure that participants presented distinct solutions to one another. The two prompts were selected from prior work that validated their similarity in terms of their structure, complexity and solvability (Patel, Elena, & Summers 2019). The design prompts given to participants were as follows:

“Design an automatic clothes-ironing machine for use in hotels. The purpose of the device is to press wrinkled clothes as obtained from clothes dryers and fold them suitably for the garment type. You are free to choose the degree of automation. At this stage of the project, there is no restriction on the types and quantity of resources consumed or emitted. However, an estimated 5 minutes per garment is desirable.”

“Design an automatic recycling machine for household use. The device should sort plastic bottles, glass containers, aluminum cans, and tin cans. The sorted materials should be compressed and stored in separate containers. The amount of resources consumed by the device and the amount of space occupied are not limited. However, an estimated 15 seconds of recycling time per item is desirable.”

During the design task, each participant was provided with the same materials. For the sketching condition, this included pencils, papers and a ruler. For the prototyping condition, in addition to materials for sketching, participants were also given foam core, cardboard, popsicle sticks, rubber bands, wire, thread, utility knife, scissors, tape, cotton balls, tube cleaners and hot glue.

Once participants completed their design task, they were given a survey through Qualtrics where they rated the paradigm-relatedness of their own solutions and provided a written description of their design problem, solution and how it works. Once both participants completed the survey, they were brought into the same room along with their final sketches or prototypes. All subsequent interactions between the participants were audio- and video-recorded. The facilitator then asked the participants to present their design solution to each other: “You will both now present your design solution to each other and you can use your design representation to do so. Please remember to go over your design problem, solution, how it works, and how you arrived at it, and keep the explanation of your solution consistent with your written explanation in the survey you just completed.”

First, one participant was assigned as the communicator and the other as the listener. The communicator was given 5 minutes to explain their design problem and solution to the listener. Following this, the listener had 3 minutes to ask the communicator any clarifying questions. Next, each participant was given a survey through Qualtrics. The communicator was asked to rate the cognitive load experienced while communicating their concept (measured by the NASA-TLX instrument). In the listener’s survey, the listener was asked to rate the paradigm-relatedness of the solution presented and was asked to describe the design concept that was presented to them: “In as much detail as possible, please recall the solution that was presented to you, and describe what the solution is, the problem it solves, and how it works in your own words.” They were also asked to rate the cognitive load experienced while listening to and recalling the communicator’s explanation (measured through the NASA-TLX instrument). Following this, the participants switched roles, i.e., the participant who was previously the listener became the communicator and vice versa. Identical to the previous stage, the communicator had 5 minutes to present their solution, following which the listener had 3 minutes to ask any clarifying questions. Finally, participants were given the same post-task surveys for the communicator and listener, an example of the experiment is shown below (Figure 3).

Figure 3. Communicator (right) using their prototype to present their design solution to the listener (left).

Finally, each participant was taken to a different room for a short semi-structured interview aimed at understanding their experience of communicating with and understanding design concepts using either a prototype or sketch. To conduct the interviews simultaneously, a graduate researcher experienced in conducting human-subjects design research was trained in the specific interview protocol for this study. The questions asked during the interview related to participants’ perceptions of the interaction (“Can you describe the interactions between you and the other participant when you explained your design concept?”), their experiences communicating their design concept (“Do you think the prototype/sketch helped you in explaining your concept? Why?”) and understanding the other participant’s concept (“Do you think the prototype/sketch helped you in understanding the concept? Why?”) and any challenges in communicating or understanding (“What do you feel inhibited your ability to understand the design concept?”). These interviews were audio-recorded and transcribed by a third-party service. Participants were debriefed, thanked for their time and allowed to leave. All sketches and prototypes were photographed and stored, examples of both physical prototypes and sketches are shown below (Figure 4).

Figure 4. Example prototypes (top) and sketches (bottom) generated by participants.

3.4 Data analysis

RQ1: How does the modality of a design representation, specifically sketches and physical prototypes, affect the cognitive load of the communicator and listener?

All statistical analyses for RQ1 were performed on R version 4.1.1. In addition to p-values, we also report the effect sizes for the statistical tests performed (Hedge’s g for t-tests and r for Mann–Whitney U-tests) (Ellis 2010). Before data analysis, one participant’s cognitive load ratings (when they were the communicator) were removed from the dataset due to incorrectly entering their ratings. Additionally, one participant misunderstood the prompt during the design task, and the cognitive load ratings of the communicator and listener during the explanation of this design solution were also removed from the dataset.

Before conducting any statistical analysis, an outlier analysis was conducted; outliers were defined as values greater than 1.5 times the interquartile range (Ghasemi & Zahediasl 2012). For listeners’ cognitive load ratings, one participant’s temporal demand and frustration ratings were identified as outliers, and another participant’s performance rating was identified as an outlier. These datapoints were removed before the analysis. For communicators’ cognitive load ratings, one participant’s rating on the frustration dimension of the NASA-TLX scale was identified as an outlier and removed.

To isolate the effects of the representation (prototypes vs. sketches), we also checked for any relationship between paradigm-relatedness and idea complexity on the cognitive load of communicators and listeners. A series of Kruskal–Wallis tests showed that the listeners’ ratings on all dimensions of the NASA-TLX scale did not differ based on their perceptions of the paradigm-relatedness of the presented solution, and paradigm-relatedness was not included as a covariate in the analysis. Similarly, a Spearman correlation was performed to assess the relationship between the complexity of the presented idea and listeners’ cognitive load ratings. No relationship was found between idea complexity and any dimension of the NASA-TLX scale, and idea complexity was not included as a covariate in the analysis.

RQ2: What characteristics of design representations alleviate or add to the perceived cognitive load of the communicator and listener when communicating design concepts?

In this study, we followed a follow-up explanatory QUANT -> qual approach, implying that qualitative data are collected and analyzed to explain the quantitative results in further detail (Giddings & Grant 2006). To understand what characteristics of design representations affect the cognitive load of communicators and listeners, we conducted interviews with 30 of the 44 participants (i.e., those not part of the initial pilot studies). These interviews were transcribed using a third-party transcription service and then validated for accuracy by the first author. Unfortunately, due to a technical issue with the recording of one participant, this interview could not be transcribed accurately and was subsequently removed from the dataset. All transcripts were anonymized using each participant’s unique ID.

We began coding each transcript using an open and axial coding approach (Charmaz 2006) paired with an abductive coding paradigm. Unlike a grounded theory approach that seeks to generate a theory or framework as a product of the qualitative analysis, in abductive coding, the researcher uses prior theory to guide their coding process while also being receptive to themes and codes that may go beyond prior theory (Timmermans & Tavory 2012). In particular, in this work, the first author kept in mind Mayer’s CTML (Mayer 2005) and Sweller’s CLT (Sweller & Chandler 1991) during the qualitative analysis process. Using these theories as a lens through which coding was performed allowed him to identify how participants communicated and understood design representations using both verbal information and design representations and which characteristics contributed to cognitive load during communication. The first author coded the entirety of the dataset. The first author maintained memos throughout the coding process and entered observations and codes on Miro, a visual collaborative platform for teams, on a weekly basis. The first and third authors met on a weekly basis to review and organize codes and identify the themes that answered the RQ of interest. Through this iterative process, we arrived at the themes that contextualize our quantitative results and provide a holistic picture of the effect of representation modality on design communication.

4. Results

RQ1: How does the modality of a design representation, specifically sketches and physical prototypes, affect the cognitive load of the communicator and listener?

We first sought to identify whether the cognitive load reported by listeners differed between the prototyping and sketching conditions (Figure 5). To determine the appropriate statistical test, we verified the assumptions of normality and homogeneity of the variances of listeners’ ratings on each dimension of the NASA-TLX. Only the dimension of mental demand met both assumptions, and a t-test was run to analyze differences between conditions on this dimension. For all other dimensions, Mann–Whitney U-tests were performed. In both cases, the tests were conducted with the condition (prototyping and sketching) as the independent variable and the dimension of the NASA-TLX as the dependent variable. The results of the tests are shown in Table 2, with the significant results highlighted in yellow. We found a significant difference (t = 2.047, p < 0.05) and moderate effect size (g = 0.612) in the mental demand perceived by listeners between the prototyping and sketching conditions. Listeners who were presented ideas as prototypes perceived higher mental demand in understanding and recalling the presented design solution than those who were presented ideas as sketches. We also found a significant difference (U = 315.5, p < 0.05) and moderate effect size (r = 0.366) in the physical demand perceived by listeners between the prototyping and sketching conditions. Listeners who were presented ideas as prototypes perceived higher physical demand in understanding and recalling the presented design solution than those who were presented ideas as sketches.

Figure 5. Significant differences were found between the mental demand and physical demand perceived by listeners in the sketching and prototyping conditions.

Table 2. p-values and effect sizes for t-tests assessing the effect of the representation (sketch versus prototype) on listeners’ cognitive load levels

While the difference in the perceived performance of listeners was not significant (U = 147, p = 0.065), we found a moderate effect size (r = −0.332) between listeners in the two conditions. Based on this effect size, we posit that with a larger sample size a significant difference may be observed in the performance of listeners between the two conditions; we do not make any claims regarding the effect of representation modality on the performance of listeners solely based on the observed effect size.

Next, we sought to identify whether the cognitive load reported by communicators differed between the prototyping and sketching conditions (Figure 6). Once again, to determine the appropriate statistical test, we verified the assumptions of normality and homogeneity of variances of communicators’ ratings on each dimension of the NASA-TLX. Only the mental demand and effort dimensions met both assumptions, and t-tests were run to analyze differences between conditions on these dimensions. For all other dimensions, Mann–Whitney U-tests were performed. In both cases, the tests were conducted with the condition (prototyping and sketching) as the independent variable and communicators’ ratings on the dimension of the NASA-TLX as the dependent variable. The results of the tests are shown in Table 3. While no significant results were found, we do highlight the difference in communicators’ perceived frustration between the prototyping and sketching conditions. While this difference was not significant (p = 0.087), a moderate effect size was observed (r = 0.31). Based on this effect size, we posit that with a larger sample size a significant difference may be observed in the frustration levels of communicators between the two conditions; we do not make any claims regarding the effect of representation modality on the frustration levels of listeners solely based on the observed effect size.

Figure 6. No significant differences were found between the cognitive load experienced by listeners in the sketching and prototyping conditions.

Table 3. p-values and effect sizes for t-tests assessing the effect of the representation (sketch versus prototype) on communicators’ cognitive load levels

RQ2: What characteristics of design representations alleviate or add to the perceived cognitive load of the communicator and listener when communicating design concepts?

Our quantitative analysis found that listeners who were presented ideas as prototypes perceived higher levels of mental and physical demands as compared to those who were presented ideas as sketches. No differences were found in the perceived cognitive load of communicators. In this section, we discuss the results of the qualitative analysis aimed at contextualizing these quantitative results. Using an abductive coding paradigm and viewing our data through the lens of CTML and CLT, we sought to understand how participants communicated and understood design concepts through visual representations and verbal explanations and identify key characteristics that may affect individuals’ cognitive load during communication. Table 4 lists the themes identified in our analysis.

Table 4. Thematic codebook

Visual or physical representation: Through our qualitative analysis, we identified three themes, or characteristics of visual representations, that could affect the cognitive load of communicators and listeners during communication – physical affordances, information content and visual anchor. In the theme “ Physical Affordances ,” we identified participants’ descriptions of a representation’s ability (or lack thereof) to allow for demonstrations of their design ideas and show multiple views of their ideas simultaneously. These physical affordances, or the degree of interactivity provided by a design representation, were noted by participants as one of the key advantages of communicating with low-fidelity prototypes. For instance, one participant noted how the low-fidelity prototype she created allowed her to demonstrate the movement of the iron press and folding mechanism in her design:

“I think having the prototype that moved a little bit and was interactive was really helpful, because explaining this without demonstrating I think would be somewhat difficult. I think without that it would have been a lot harder for me to explain.”

The participant who was presented with this idea also commented on how the prototype allowed him to understand the functioning of the design:

“The dynamic aspect of the model was super helpful. I love exploded diagrams on CAD and things like that, or assemblies that move and things like that. I thought it was super helpful to just visualize the actual pressing of the garment and the folding mechanism was super cool.”

Participants in the sketching condition who had similar designs with moving components noted that a low-fidelity prototype would have been beneficial to demonstrate motion:

“I guess that it would be good to have certain materials, maybe not building the entire thing, but even just showing this folding table, ’cause I could easily make that with a piece of construction paper or something like saw material, and that would even show more visually about how I was thinking about how the folding part would work.”

Other participants in the sketching condition with similar design concepts spoke about “storyboarding” their sketch in lieu of a prototype – using multiple sketches of the same design feature to demonstrate its position at multiple timepoints. For example, consider the quote below from a participant describing how the sketch presented to them could have been improved:

“Although she had the numbers on the ones get showing one, two, three, the folds, I think even a timeline almost separating them. And then if she did use heat in between the folds and putting that into the timeline and saying, ‘Okay, it’s gonna fold here now, and then add heat and press, and then now it’s gonna fold here,’ and kind of separate it like that, which could help.”

Communicators also emphasized that one of the key benefits of a low-fidelity physical prototype was its three-dimensionality, as it allowed communicators to show different views of their design ideas. For example, a participant in the sketching condition noted how one of the components in his design solution was challenging to communicate as a sketch is “not very explicit” and noted the problem of “looking at this in one dimension.” Another participant stated how different views of his design concept could be visualized using a single prototype, rather than “multiple sketches to showcase different angles,” which he then perceived as making the idea “easier to understand.”

In the theme “Information Content ,” we explore how participants perceived the ability of sketches and physical prototypes to represent design features and other relevant design information during communication. Communicators described challenges in visualizing design information in low-fidelity physical prototypes, which they felt affected their ability to explain the design concept. This observation is consistent with how low-fidelity prototypes are used in design practice – often these prototypes are created in early-stage design and are not always visually similar to the final design concept. However, it is interesting to note the perceived trade-off between the amount of information a representation carries and the effort needed to build it. This trade-off may be associated with more complex ideas that had more design information to be communicated, as was noted by one participant who stated that his design was “too complicated to really form a feasible low-fidelity prototype.” Another participant who built a prototype of an automated recycling machine noted how, in addition to sketching being “a lot faster than building the physical model,” a sketch could also communicate a greater level of detail than a low-fidelity physical prototype. This trade-off between the effort to build a prototype, even at low fidelity, and the information it represented was echoed by a participant assigned to the sketching condition:

“I was able to kind of break down each sub-system into their own sketches. So like I said, I had three components to the design. And so where I have the compressor located, I have a separate sketch for the compressor. And as opposed to that taking 15 minutes to fabricate, it took me about five minutes to sketch all the important parts, and then I could start to add annotations to it to be like, ‘This is how this works,’ ‘This is how this is supposed to work.’ And I was able to also annotate better notes for the presentation, things that I didn’t have necessarily with the prototype.”

The absence of design information in representations also impacted how listeners built mental models of design concepts. Not only was missing information cited frequently as a challenge when understanding design concepts, but the absence of design information also led to listeners “filling in gaps” during communication. In other words, listeners were found to make assumptions about what the missing information may visually look like, and these assumptions were often based on mental models of designs the listeners already possessed. For example, one participant, who was presented with a low-fidelity prototype of an automatic recycling machine, which only showed the form of the design, stated how she “ended up trying to imagine what [the] original prototype was gonna be with the button to crush [items].” When asked how she was imagining these design features, the participant said she was:

“…visualizing [the other participant’s] prototype in my head based on what I already know about different trash cans that just have the signs that say, oh, okay. Put aluminum in here. And I was trying to visualize the difference between what she had and what is currently on out there.”

The next theme, “Visual Anchor ,” is derived from one of the core aspects of CTML, which discusses how separately processed visual and verbal materials are combined to form an integrated mental model. In this theme, we explore how low-fidelity prototypes and sketches facilitated (or inhibited) the construction of this final mental model. Participants articulated how annotations played a key role in providing visual cues to participants during communication, particularly when communicators did not feel confident in the quality of their sketches. For example, one participant stated how annotations in his sketch were useful to “remind probably both myself and whoever’s looking at it what they’re looking at.” Another participant detailed how the presence of annotations in the sketch presented to him helped in linking the verbal information (being described by the communicator) and the visual information represented in the sketch:

“When you put in the trash then she had a different box for the compression chamber, and she labeled that, and then she drew four ducts essentially going into four different boxes and they were also labeled. And then she had another drawing for the screen where she was like, ‘Okay, these are the four input options.’ And so it was, yeah, it was pretty easy to understand. Everything was well labeled and everything was drawn out.”

We also identified participants describing certain aspects of low-fidelity prototypes that may have inhibited the creation of connections between physical and verbal information. These challenges primarily had to do with the inherent nature of low-fidelity prototyping. Participants stated challenges with visualizing certain design features using materials conventionally used during low-fidelity prototyping (such as cardboard and foam), and only a few participants resorted to using annotations in their prototypes. For example, when asked whether anything was challenging to explain using her prototype, one participant said that the inclusion of a conveyor belt in her design concept was challenging due to “not having a nearly working or very similar looking piece to my prototype that looks like a conveyor belt.” Another participant, when asked what he would change about his prototype to make explaining his idea easier, stated how he would add labels to clearly indicate what different parts were meant to represent: “So one thing, I think I should label everything better. So, like right now everything is just cardboard, like styrofoam, styrofoam, styrofoam on top styrofoam.” Another participant, who was presented with the idea of an automated recycling machine represented as a low-fidelity prototype, stated how understanding what certain materials were meant to represent was challenging:

“I guess just looking at the machine itself, it wasn’t totally clear what some of the materials meant. There was cotton balls at the bottom of one of the tubes, and I wasn’t really sure what that was supposed to convey. Maybe like a cushion, so the plastic doesn’t shatter. I don’t really understand what that was about.”

Through the quotes above, we observe how the physical affordances of a design solution, the information content it can represent and its ability to act as a visual anchor may shape individuals’ cognitive load during design communication. We also note how participants perceived trade-offs between these aspects of a design representation and the effort needed to make them. For instance, the ability to demonstrate motions was seen as an important “return” gained from low-fidelity prototyping. However, when the objective was to communicate a greater level of design detail, a sketch was perceived as more useful than a low-fidelity prototype.

Verbal explanation: Design representations do not exist in isolation during communication, and designers often combine visual or physical representations with verbal explanations. We observed two themes related to participants’ verbal explanations that shaped the communicative process: “Ambiguities in idea” and “Links to prior knowledge.” In the theme “Ambiguities in idea ,” we identified how the concreteness of an idea (or how “fleshed out” an idea was, as stated by one participant) affected participants’ communication and understanding of design concepts. For instance, one participant when asked whether any parts of the design concept presented to her were hard to understate stated, “I don’t think the mechanisms behind the actual function of the steamer were thought out, I think it was more of the system of how it’s going to progress.” A similar sentiment was echoed by another participant, who had design features in her idea that were, in her opinion, not thoroughly thought out:

“I would say maybe the hardest part to explain was, my design has an external component that, you’re supposed to just use the touch screen and it will automatically sort things out. I didn’t really figure out the details of the how that’s gonna happen. So I was like, ‘Just imagine it happens and it did’”

Participants’ perceptions of their ideas being less “flushed out” may have been a facet of the limited duration of the design task. However, in the theme “Links to prior knowledge ,” some participants attributed ambiguities in their design solution to their own lack of background knowledge. For instance, one participant said that the inclusion of a compressor in her design was challenging to explain as she did not have the knowledge to “actually properly design” a compressor, which then led to her giving a surface-level explanation of the idea. Here, we observe how the background knowledge of the individuals communicating affects the communication process. This effect also extended to the background knowledge of listeners. One participant stated how he found it challenging to understand the idea presented to him as he did not have a good understanding of how hydraulic presses worked, one of the features of the presented solution:

“The hydraulic press I had a hard time a little bit in the beginning, and then I figured out a little more. But I’m not familiar with hydraulic presses, so I think it was something that I didn’t have a basis understanding. It was hard to picture it even with the prototype.”

While the absence of background knowledge may have inhibited communicators from explaining their designs in-depth, in the theme “ Links to prior knowledge ,” we also observe how some communicators were able to compare their solutions to existing ideas and knowledge. This deliberate use of analogies during communication was likely used to provide listeners with a reference point during communication. Consider the participant below who discussed using analogies to trash chutes and bank teller machines in her explanation:

“I think it also helped that I was comparing certain things to things that we know, like the trash chute or like I said, those bank teller things comparing them to something that people are familiar with makes it more effective in communication because they have something tangible to tie that idea to. Say, oh, I’ve seen how that works before, or I at least know how that mechanism works to some extent.”

This strategy of using analogies was likely helpful for listeners, as seen in the quote below. This participant, who presented an idea for an automated cloth ironing and folding machine, stated how the communicator’s use of analogies helped him understand how clothes would be fed into the machine:

“I think she did a really good job explaining it and how it basically worked. Obviously, she used a good analogy whenever you’re inserting it, you can’t just throw it in like a jumbled-up piece, but actually inserting like the dollar bill, I thought that was really good on her part.”

Through these qualitative results, we observe how the different characteristics of verbal explanations shape the communicative process between participants. While ideas in early-stage design are likely to be less detailed, from a communicative perspective, these less thought-out ideas were perceived as being challenging to explain and understand. Furthermore, the role of background knowledge in communication manifested itself in two ways. While the absence of knowledge inhibited communicators’ ability to explain the technical details of their design solutions, some communicators were able to leverage connections to commonly known design concepts to communicate effectively.

5. Discussion

5.1 Summary of results

This work sought to investigate how prototypes and sketches differed in their effects on communication in design. In particular, we used a mixed-methods approach to study whether, and how, the cognitive load of communicators and listeners was different when prototypes and sketches were used to communicate design concepts. We conducted a controlled study with 44 participants; each participant was asked to create either a prototype or sketch of their design concept and explain and communicate their idea to a partner. Participants were asked to rate their perceived mental workload (indicative of cognitive load) when understanding and communicating design concepts using the NASA-TLX instrument; these ratings were then quantitatively analyzed. Additionally, we also conducted interviews with each participant to understand their experiences of communicating and understanding design concepts using sketches and prototypes. These data were then qualitatively analyzed through an abductive coding paradigm using CTML and CLT.

Our quantitative results found that listeners who were presented ideas as low-fidelity prototypes perceived higher levels of mental and physical demands when understanding and recalling the design solutions presented to them. One of the benefits of communicating with physical prototypes is the tactile engagement they allow for (Brandt 2007), and it is likely that these tactile engagements led to perceived higher levels of physical demand in the prototyping condition. The difference in physical demand is expected, and based on the low median value of physical demand ratings of listeners in the prototyping condition (M = 1), we do not believe there is any indication of cognitive overload. We additionally observed a significant difference in the mental demands of listeners. In particular, listeners who were presented ideas as low-fidelity prototypes perceived greater levels of mental demand than those who were presented ideas as sketches. These results are contrary to the work by Dadi et al. (2014), who found that participants perceived similar levels of mental workload when extracting information from 2D drawings and 3D physical models. We emphasize, however, that it is challenging to make accurate comparisons between this work and the work by Dadi et al. due to the differences in the characteristics of the task and representations used in both studies. First, participants in the work by Dadi et al. were required to create mental models of and subsequently reconstruct a simple, static building structure. This differs from the design solutions created by participants in this work that were far more complex, each with its own multiple subsystems and moving components. Participants would have needed to build a detailed understanding of the components of a design solution, the processes involved in its functioning and the movement of components relative to one another. Additionally, the representations in Dadi et al.’s work were 3D-printed models and schematic drawings, which differ from the sketches and low-fidelity prototypes in this work with respect to their quality and level of fidelity. 3D-printed models are commonly of greater fidelity (Deininger et al. 2019) as they capture greater levels of detail than prototypes made from cardboard, foam and other common materials used in low-fidelity prototyping. Additionally, the sketches created by participants were hand-drawn, which were likely of lower quality than the computer-generated sketches used in the work by Dadi et al.

5.2 Linkages to cognitive load theory and cognitive theory of multimedia learning

To identify the characteristics of low-fidelity prototypes and sketches that may have affected the cognitive resources expended by participants during communication, we qualitatively analyzed the data gathered during interviews with participants. During the qualitative analysis, we leveraged an abductive coding paradigm leveraging Mayer’s CTML (Mayer 2005) and Sweller’s CLT (Sweller & Chandler 1991). As described by participants, the physical affordances of a prototype, such as the ability to demonstrate how components move relative to each other and show multiple views of a design concept, helped them understand the presented design concepts. This aligns with much of prior work in design research, highlighting that a prototype’s physicality, even at low fidelity, lends itself to tactile interactions and subsequently improves communication (Brandt 2007). Additionally, from the perspective of CLT, the use of dynamic visualizations (such as animations) has been associated with a lower cognitive load than static visualizations when these animations are relevant to the knowledge to be acquired (Höffler & Leutner 2007). According to Kirsh (1996), these physical actions in the real world, termed complementary actions, prevent the need to perform the same actions on images stored in an individual’s memory. Researchers have argued that these complementary actions lead to lower cognitive resources needed to process visual and spatial information (Newman et al. 2018). Hence, we posit that the physical affordances of prototypes may lead to lower cognitive resources when visualizing a design solution in three dimensions and the motions of components.

Conversely, participants expressed how they were able to show more details via sketches, as compared to low-fidelity prototypes. The lack of detail in low-fidelity prototypes was attributed to the challenges of physically constructing certain design features. We identified that even at low fidelities, physical prototyping is perceived as more time-consuming and effortful than sketching, leading to trade-offs between the effort needed to create a prototype and the level of detail the prototype can convey. This absence of detail also affected how listeners built mental models of the presented design solutions. In particular, participants stated that the aspects of a design concept that were not visualized in a representation were also the hardest to understand. Viewing this through the lens of CLT and CTML, it is likely that the absence of visual information may have led to an increase in cognitive resources needed to build mental models of design representations. Conventional wisdom in the field of instructional design states that extraneous load is reduced when both visual and verbal channels are used for processing information (Mayer & Moreno 1998; Mayer 2005; Wouters, Paas, & van Merriënboer 2008). When visual information is missing, individuals may have to expend cognitive resources to translate verbal information into mental images, leaving behind lesser cognitive resources to process additional information and create connections between visual and verbal information (Mayer & Anderson 1991; Castro-Alonso et al. 2021). Furthermore, individuals rely on existing knowledge and schema to build these mental images, as dictated by the “imagination effect” (Leahy & Sweller 2004; Sweller, Ayres, & Kalyuga 2011). We also observe a manifestation of the imagination effect in our data. One listener stated that they used their existing knowledge of trash cans to “fill in the gaps” in the missing visual information. This finding also aligns with work by Sauer & Sonderegger (2009), who found that users tended to rate the esthetics of design concepts higher when presented with low-fidelity prototypes as compared to high-fidelity prototypes. The authors attributed this result to the absence of certain design features in low-fidelity prototypes, and users were creating mental images of these missing design features based on their knowledge of existing design solutions. While these assumptions may be accurate in some contexts, they may be inaccurate in others, which could impede the establishment of shared understanding between individuals.

CTML posits that mental models are created through connecting visual and verbal information, which are initially processed in separate channels. Viewing our data through the lens of CTML, we identified the attributes of sketches and low-fidelity prototypes that facilitate or hinder the creation of these connections. One of the facilitators of connections between visual and verbal information was annotations. Rather than annotations overloading the visual channel, our qualitative analysis identified that annotations in representations acted as visual cues during communication. This phenomenon has been noted in prior literature and is termed the “signaling effect” (Mayer & Moreno 2010). It is associated with reductions in ECL (Mayer & Moreno 2010), as these cues direct listeners’ attention toward relevant information and facilitate the connections between verbal explanations of design features and their visual or physical representation (de Koning et al. 2009). However, some participants who explained ideas with low-fidelity prototypes described challenges in understanding what certain materials in the prototypes represented. As indicated by communicators, the use of referential annotations may have clarified the features of their design solution, as these cues would have indicated what each material was meant to represent. We posit that the absence of these visual cues may have led to participants devoting cognitive resources to make sense of what the materials were supposed to convey to integrate the presented visual and verbal information. As this cognitive process does not contribute to learning but is induced by the design of the presented material, it likely led to an increase in ECL (Mayer & Moreno 2010).

Additionally, we also observed how the use of analogies contributed to effective communication between participants. Sifonis, Chernoff, & Kolpasky (2006) highlight analogies as a useful communication tool in product development, as they help listeners understand unfamiliar concepts by leveraging connections to prior knowledge. The use of analogies to link new information to prior knowledge also reduces the cognitive resources needed to process the new information, thereby reducing ECL (Gray & Holyoak 2021). Lastly, we also observe that ambiguities in verbal explanations impede effective communication. In the interviews, participants felt that the features of their design solutions that were the least thought out were also the most challenging to explain. It is important to note, though, that it is unclear to what extent this affected the cognitive load of participants. We found no significant differences in the communicators’ perceived cognitive load, and neither CTML nor CLT suggests how ambiguities in verbal information could affect cognitive load. We highlight this as an important area of future work. Design solutions are rarely fully thought out in early-stage design, and it is important to understand how a design solution’s concreteness (or lack thereof) affects designers’ cognitive load during communication.

5.3 Contribution to the field of design research

In the previous section, we have highlighted the characteristics of sketches and physical prototypes that may have driven participants’ cognitive load during communication. Combining these with our quantitative results, we provided possible reasons for the higher mental demand of listeners when understanding and recalling solutions through physical prototypes. Rather than establishing causal–effect relationships or claiming that one representation is better than the other to communicate, this work builds a greater understanding of the contexts in which sketches and prototypes are effective tools for communication. For instance, if a designer intends to demonstrate the movements of components or show different views of their design solution, a low-fidelity prototype’s physicality is an effective communicative tool. Rather than performing mental rotations, individuals involved in communication would be able to interact physically with the prototype, requiring lower cognitive resources to process. However, if a designer intends to communicate a certain level of detail, a sketch may be an effective communication tool. Rather than spending time and effort to build certain components with low-fidelity prototyping materials, and potentially inaccurately, designers would be able to sketch the level of detail needed to communicate. This would subsequently aid communication, as listeners can rely on both verbal and visual information to build their mental models. Additionally, irrespective of the design representation used, we highlight how visual cues such as annotations would be useful to indicate important design features, which would then facilitate connections between verbal and visual information.

Synthesizing these findings, our results highlight the importance of purposeful prototyping in the context of communication – designers being aware of what they want to communicate would help inform which representation to create for communication. This is analogous to prior work in design research highlighting the need for design tools that encourage purposeful prototyping, such as Lauff et al.’s prototyping canvas (Lauff, Menold, & Wood 2019) and Hansen et al.’s prototyping planner (Hansen et al. 2020). Given the trade-offs that participants perceived between the time and effort needed to create a representation and the communicative benefits it provides, our results can inform designers’ decisions and awareness when selecting the most appropriate design representation to communicate. This could then lead to improved communication between individuals and more successful design outcomes.

6. Limitations and future work

This work is limited by several factors. First, this study simulated communication between two individuals. However, design teams often consist of more than two people, and designers communicate in much larger groups than in pairs. Future research could expand on this work to investigate how these results may vary depending on the group size during communication. Second, all participants were engineering students. It is important to validate these results with experts, as novice and expert designers may have different communicative patterns and strategies while leveraging representations such as prototypes for communication (Lauff, Kotys-Schwartz, & Rentschler 2017). It is also important to consider and study the contexts in which designers communicate with people who do not come from engineering backgrounds, as is often the case in design practice (Darling & Dannels 2003). Such individuals may not have the prior knowledge possessed by engineering students or designers, which would impact how they comprehend design solutions and how designers tailor their representations and explanations for these audiences.

Future work should investigate how the cognitive load in communication may change depending on the contexts in which communication takes place. Third, our study had a relatively small sample size of 44 participants. While we believe we have provided valuable results to the field, future work should increase the sample size to validate these results. Lastly, our study was limited to a single comparison between sketches and low-fidelity prototypes. Future work should investigate how designers and other stakeholders communicate and understand design concepts using other design representations, such as CAD models, high-fidelity prototypes or mixed media. This work also required participants to generate concepts in a relatively short period of time, and future work should study how communication processes may change over longer design projects.

6.1 The validity of cognitive load measurements in design research

We avoid claiming casual–effect relationships in this work due to the inherent differences between the NASA-TLX instrument (used in the quantitative analysis) and the tenets of CTML and CLT. The NASA-TLX instrument characterizes mental workload through six dimensions – mental demand, physical demand, temporal demand, performance, effort and frustration. However, the theories of CTML and CLT discuss cognitive load in terms of its source – extraneous load (from the design of the presented material), intrinsic load (from the complexity of the learning material) and germane load (from the acquisition of long-term schema). As there is no prior theoretical work that has established relationships between these different dimensions of cognitive load, we refrain from claiming that an attribute of design representation, associated with increased extraneous load, would have also manifested as an increase in one of the dimensions of the NASA-TLX, such as mental demand.

The NASA-TLX has also been used in several studies in design research to measure mental workload (Nolte & McComb 2020; Song et al. 2021; Zimmerer & Matthiesen 2021). However, it is important to highlight that the NASA-TLX was originally designed to measure a pilot’s mental workload on a flight deck (Hart & Staveland 1988), a task completely different from the cognitive experience of design. CLT states that individuals experience cognitive overload when they have used all available cognitive resources and are subsequently unable to process any new information (Sweller & Chandler 1991). However, the NASA-TLX requires respondents to subjectively rate the task on six dimensions on a scale from 0 to 20, and it is unclear at what point individuals reach this level of cognitive overload. It also does not yield insight into what the sources of cognitive load are. As highlighted earlier, while qualitative interviews can be used to identify specific factors of presented material that drive cognitive load, there is no prior work linking the dimensions of the NASA-TLX to the categories of cognitive load described in CLT.

Researchers in instructional design have recently called for and led the development of newer instruments that subjectively measure cognitive load and differentiate between extraneous, intrinsic and germane load. An example of such an instrument is one by Leppink et al. (2013). However, the use of these instruments is limited to educational contexts, such as lectures. For instance, one of the items in Leppink et al.’s instrument is “The activity covered concepts and definitions that I perceived as very complex,” and this statement is one of three in the instrument used to isolate perceived levels of intrinsic load (Leppink et al. 2013, p. 1070). The specificity of the context makes it challenging to extend the use of such instruments to design research. It is imperative that design researchers develop instruments to measure cognitive load unique to engineering design tasks, thereby increasing the validity of cognitive load measurements in the field.

7. Conclusions

This study aimed to investigate how representations of different modalities, namely sketches and low-fidelity prototypes, differed as communication tools in terms of cognitive load perceived by the communicator and listener and what characteristics of these design representations affected the cognitive load of communicators and listeners. Our results revealed that listeners who were presented ideas as low-fidelity prototypes perceived greater levels of mental and physical demands when understanding and recalling design concepts, as measured by the dimensions of the NASA-TLX instrument. However, no differences were found between the ratings of communicators between the two conditions. Using CTML and CLT, we identified three characteristics of design representations and two characteristics of verbal explanations that could affect the cognitive load of communicators and listeners during communication. Ultimately, this work contributes to the knowledge base needed for designers to communicate more effectively using design representations and motivates future work to understand the effect of design representations on communication in design.