Analyzing problem framing in design teams: a systems mapping approach

Abstract

Early phases of the design process require designers to select into view elements of the problem that they deem important. This exploration process is commonly referred to as problem framing and is essential to solution generation. There have recently been calls in the literature for more precise representations of framing activity and how individual designers come to negotiate shared frames in team settings. This paper presents a novel research approach to understand design framing activity using a system thinking lens. Systems thinking is the way that we understand a system’s components and the interrelations to create interventions, which can be used to move the system outcomes in a more favorable direction. The proposed approach is based on the observation that systems as mental representations of the problem bear some similarity to frames as collections of concepts implicit in the designer’s cognition. Systems mapping – a common visualization tool used to facilitate systems thinking – could then be used to model external representations of framing, made explicit through speech, and sketches. We thus adapt systems mapping to develop a coding scheme to analyze verbal protocols of design activity to retrospectively represent framing activity. The coding scheme is applied on two distinct datasets. The resulting system maps are analyzed to highlight team problem frames, individual contributions, and how the framing activity evolves over time. This approach is well suited to visualize the framing activity that occurs in open-ended problem contexts, where designers are more focused on problem finding and analysis rather than concept generation and detailed design. Several future research avenues for which this approach could be used or extended, including using new computational methods, are presented.

Introduction

Design problems are ill-structured and complex in nature and have been described as dynamic and networked (Jonassen, 2000; Dorst, 2015). Faced with complexity, designers can redefine (or reframe) the problem situation (Dorst, 2015), bringing new variables, and approaches into focus (Nickel et al., 2022). An alternative perspective has the potential to shift designers’ views about core elements of a problem and may redirect them toward different solutions (Murray et al., 2019). A problem frame will be the designer’s way of reading the situation and when the problem situation is familiar, will likely come to mind early in the process (Dorst, 2011). However, when working in a team each member has their own design frame(s), which are debated and negotiated in a process of co-creating a shared understanding of the problem (Hey et al., 2007) that may directly influence team process and performance (Mathieu et al., 2000).

Framing is considered to be one of the core elements of design thinking and has been studied for several decades (Dorst, 2011; Kelly and Gero, 2022). These insights have brought our attention to how teams of designers grapple with the complexity of design problems; however, there lacks precise and conceptual consensus about what exactly design frames are (Kelly and Gero, 2022). In this paper, we draw inspiration from systems thinking – the way in which we engage with and understand systems – to extend our understanding of problem framing. Systems thinking language and methodologies provide a lens for how we might define elements that are brought into designers’ focus as they frame the problem. We adapt a common system thinking tool – systems mapping – which is most commonly used to visualize complex systems to gain clarity of the elements that are important – to devise a coding scheme for analyzing verbal protocols of teams of designers engaging with design problems. The method and analyses conducted in this paper suggest that the resultant system maps may offer a retrospective representation of framing in design teams.

The rest of the paper is organized as follows. In Section Background, we provide background on problem framing, existing approaches to studying framing, and their limitations, and the relationship to systems thinking and visualization tools. In Section Method we describe in detail the new approach that is developed to analyze verbal protocols of design activity and explain how it was tested on two sets of design protocols. Results of those analyses are presented in Section Results. In Section Discussion, we discuss the main contributions and implications of this work for studying framing activity, identify key limitations, and propose future research avenues, before concluding in Section Conclusion.

Background

Problem framing

According to Simon (1978), design is a planned course of action aimed at changing existing dissatisfactory situations into preferred ones. While design is typically modeled as a multi-phase process spanning different activities – from problem finding and analysis to solution delivery – numerous studies of design activity have demonstrated that designing entails a continuous refinement (or “co-evolution”) of both the designer’s understanding of the problem and their ideas for solving it (Crilly, 2021; Dorst and Cross, 2001; Maher and Poon, 1996). Design is thus a reflective process of devising actions, observing their consequences, and devising further actions, with their own consequences, and so on (Schön, 1984b).

Design problems are distinguished by their ill-structured and open-ended nature (Goel and Pirolli, 1992; Jonassen, 2000, 2011; Simon, 1978). Designing thus requires significant effort in problem structuring to ensure that designers are solving the right problem. According to Schön (1984b), faced with the complexity of a situation designers select into view (or “name”) the “things” of the situation and the boundaries around them that they believe to be relevant to solving the problem (p. 40). Schön described setting the context in which the named things are attended to as “framing” (Schön, 1984a, 1984b). As the designer’s frame of the problem changes, they may pay attention to different aspects, for example shifting their attention to the whole or details, to technological opportunities or constraints, to ergonomic or aesthetic considerations (Goldschmidt, 2014, p. 43–44).

For Dorst (2015), a frame is the proposal through which, by first recognizing and then applying a particular pattern of relationships, a designer can create a desired outcome. Although frames can sometimes be paraphrased by a simple and elegant statement, they are complex thought tools. Proposing a frame includes the use of certain concepts, which are assigned significance and meaning (Dorst, 2015). In this way, frames act as something to hold onto for the activity that occurs following their creation (Valkenburg and Dorst, 1998). According to Cardoso et al. (2016), a problem frame is “the perspective that is imposed by a designer on the design situation at a specific time during design activity” (p. 67). While a problem can be framed in multiple ways, individuals’ interpretations of a frame can vary considerably (Silk et al., 2021). As such, problem frames generated by the designer(s) are situated in their context and depend largely on the experiences of those who frame the problem. In design teams, each member will “see” the design differently (Bucciarelli, 2002), bringing their own frame(s), with varying “degrees of coherence” between team members (Kelly and Gero, 2022). Problem frames are thus both personal and social concepts. In team designing, problem frames change over time via social interaction as individual members impose their own experiences on the collective understanding of the problem (Hey et al., 2007). Team members are thus engaged in the co-creation of shared mental models of the problem and approaches to solving it, a process of negotiating shared frames (Hey et al, 2007). Unfortunately, this type of social interaction does not make the frames observable; rather, frames can be shared by a team or group only to the extent that the individual members’ frames overlap or align (Hey et al., 2007).

Different frames can help designers manage challenging design problems, offering alternative ways to approach and formulate them, and thus making available new solution opportunities (Dorst, 2015; Silk et al., 2021). Frames help to simplify and create alternative views of a problematic situation, enabling alternative outcome spaces that afford a wider range of responses (Paton and Dorst, 2011), including ones that might have otherwise been avoided or overlooked (Bardwell, 1991). For example, recent research has described how designers can bypass design trade-offs through changes to the problem frame that bring new variables into focus and enable new problem formulations (Nickel et al., 2022).

In a recent review of the literature on design framing, Kelly and Gero (2022) acknowledge the important role that framing plays in design and in the design literature and highlight the challenges in arriving at conceptual consensus on what frames are. Specifically, there is ambiguity with respect to where design frames are located - whether they are internal or external to the designer, or somewhere in between. As a synthesis of their review, they offer a convention that “a design frame is the implicit conceptual assemblage that represents the frame within cognition; and that such frames are then made explicit (such as through speech) in what are representations of a design frame” (p. 13). How then, can these representations of the design frame – the concepts and relationships between them - be modelled?

Studying design framing

The most common methodological approach for analyzing design activity is protocol analysis (Ericsson and Simon, 1984), a method that can capture and characterize design activity through a process of recording, transcribing, segmenting, and coding of designers’ verbalizations as they design. In a systematic review of the engineering design education literature, Litster and Hurst (2021) found that most protocol analysis studies used one or more problems from a handful of common design tasks (e.g., a one-handed jar opening device (Lemons et al., 2010); a playground that meets safety, cost, and accessibility requirements (Atman and Bursic, 1998); a cost-efficient and safe method to cross a busy intersection (Cardella et al., 2008)). These prompts provide participants both with goals and some requirements; the problem description imposes a certain frame on the problem and therefore the studies may not adequately capture early problem framing.

In their review, Kelly and Gero (2022) highlighted seven empirical studies of framing. Of those, four used protocol analysis as their primary method of investigation (Schön, 1984a; Valkenburg and Dorst, 1998; Kvan and Gao, 2006; Zahedi and Heaton, 2017). All four used a variation of the coding scheme introduced by Valkenburg and Dorst (1998) which builds on Schön’s (1984a) mechanisms of reflective practice: naming, moving, and reflecting. Framing is identified when something in the design situation is named; framing and reframing happen when designers cycle between moving and reflecting. More recently, computational linguistic analyses of design protocols have also been used to study framing by taking advantage of advances in natural language processing. For example, Chandrasegaran et al. (2022) identified uncommon phrases (bi-, tri-, and quad-grams) used throughout the design discourse to highlight potential evidence of framing. Protocol studies are sometimes supplemented with other data, including drawings or presentations produced by designers (Zahedi and Heaton, 2017) and observations of design teams (Valkenburg and Dorst, 1998). Hey et al. (2007) employed a mixed method approach and analyzed varied data that included interviews, project proposals, free-response survey questions, observations, and design documentation created by the design team.

The studies reviewed above have extended our understanding of problem framing; yet, identifying frames in design conversation, and especially how shared frames emerge in designs teams, remains elusive. This has resulted in calls for more precise investigations of how team coherence on frames is achieved (Kelly and Gero, 2022).

A systems thinking inspired approach for representing framing

Arnold and Wade (2015) describe systems thinking as “a set of synergistic analytic skills used to improve the capability of identifying and understanding systems, predicting their behaviors, and devising modifications to them in order to produce desired effects” (p. 675). This definition of systems thinking also captures more recent thinking about design: as problems become truly complex, design is the creation of “high-quality interventions” that move the system towards a more desired state (Dorst, 2019, p. 123; Irwin, 2019).

Systems thinking can support improved system understanding and problem framing (Elsawah et al., 2015), allowing designers to distinguish areas of interventions more easily by making the components and their connections more explicit. It involves visualizing relationships between parts of systems, examining how those behaviors change over time and drawing out phenomena from the interaction of system parts (Orgill et al., 2019). As part of the systems thinking approach to problem solving efforts, and across different disciplines, a wide variety of visual diagrammatic representations are used to understand and/or communicate ideas. In particular, system maps are tools devised to help the problem solver identify and visualize system components and their interactions as they work to understand the system and its behaviors in order to identify areas where interventions would be most impactful.

We now return to Kelly and Gero’s (2022) definition of problem frames as “implicit conceptual assemblages” in the designer’s cognition. In a team design situation, at different points in time, (sub)sets of these concepts and their connections come to the individuals’ attention and are made explicit through speech and sketches. So how can these external representations of framing (i.e., of the concepts and connections making up the “conceptual assemblages”) be modeled and analyzed?

We note a conceptual similarity between systems as (evolving) mental representations or ways of looking at the world (Espejo, 1994) and frames as conceptual assemblages in the designer’s cognition (Kelly and Gero, 2022). As such, systems mapping, typically used to visualize a system’s components and their interactions, could be used to model the concepts (and their connections), as described by the participants speech, that make up external representations of the frame. This relationship, which constitutes the theoretical grounds for our proposed approach to analyzing framing in design teams, is presented diagrammatically in Figure 1.

Figure 1. Conceptual framework for the relationship between systems and problem frames.

In itself, creating system map visualizations from qualitative data is not a new idea. For example, Love et al. (2009) use systems mapping to visualize omission errors in a project based on interviews, with the aims of uncovering interdependencies and behaviors of the system to make informed improvements. Similarly, Eden et al. (2000) used system maps to identify the effects of delays or disruptions in large scale civil engineering projects. Newberry and Carhart (2024) provide an in-depth account of how to make a particular type of systems diagram from qualitative data (e.g., interviews) in a variety of settings (e.g., urban planning, public health). Each of these examples use systems mapping techniques as they are intended: to uncover the nature of a complex system by examining the elements and connections. The novelty of our approach lies in adapting systems mapping into a coding scheme that can be applied to design protocols to retrospectively model (external representations of) framing activity in a design team setting. That is, instead of using the systems mapping methodology (according to its intended use) to represent a system, we adopt the rules and language of a traditional systems mapping process to represent the unfolding framing activity in the designers’ cognition. In the following section we explain how we developed and applied this approach on two different data sets.

Method

Development of new approach for coding verbal protocols

System maps are typically used to visualize and understand system components and their interrelations as designers work to understand the problem and identify interventions. Earlier, three examples (Love et al., 2009; Eden et al., 2000; and Newberry and Carhart, 2024) were briefly described to demonstrate how system maps can be created from qualitative data. In all three of those studies, researchers used a type of system maps called Causal Loop Diagrams (CLDs), with the intention of making explicit the parts of the system (nodes) being studied and the system dynamics in place that describe the relationship(s) between those parts (Kim, 1994). CLDs are the same type of system map that we adapt in our new approach for analyzing design protocols, as follows:

A node is identified (coded) in the verbal protocol whenever the speaker describes an entity that can influence or be influenced by other entities, and thus has a measurable quality or quantity (Kim, 1994). Each node is assigned a short label that captures its meaning.

A system dynamic is assigned to an excerpt when it can be determined that participants are verbalizing a relationship between two or more previously coded nodes. In CLDs, system dynamics describe the relationship between nodes in precise terms as either increasing or decreasing; this information is then used to identify causal loops in the map. In our approach, identification of causal loops is not meaningful, therefore only the presence of an influence and its direction were noted, and not the exact nature of that influence.

Next, we describe two studies that were conducted to develop and test the approach. Study A served to fine-tune the coding scheme and explore its use in understanding design problem framing in early stages of the design process. Study B was conducted to further validate the approach by exploring its utility in a different design task focusing on later stages of the design process.

Study A

The dataset used in this study consisted of eight verbal protocols of students working in small teams on a design task. Each of the teams were provided with the following prompt:

“Different people have different waking up experiences in the morning. However, a great number of people consider this process as unpleasant. How might you improve the morning waking up experience? As a team of three, generate new and useful ways (a product/system/service) that provide people with a positive waking up experience. If you generate several ideas, make sure you choose one final concept, and make a clear sketch of it. You should spend approximately 30 minutes on this activity.”

The “How might you…” phrasing at the beginning of this prompt intentionally makes this problem statement vague and open ended, which is typically used to encourage further exploration of a given problem (Siemon et al., 2018). As such, participants in Study A would be expected to conduct considerable problem framing to identify suitable and promising aspects of the problem for which they could ideate solutions. The study was conducted in a graduate course in a university in the Netherlands. Student participants (22–26 years of age) were randomly assigned to eight groups of three (hereon referred to as Groups 1–8), each comprising two students with an industrial design background, and one student with either a mechanical or civil engineering background. Video recordings of the sessions were captured by the students themselves and averaged 34 minutes in length. A research assistant generated transcripts.

In Table 1, we present an excerpt from Group 1’s transcript to demonstrate how nodes and system dynamics were defined during the coding process. Figure 2 presents the constructed system map from those nodes and system dynamics. This example also illustrates the importance of the coder having a good understanding of the utterance in the context of the designers’ conversation as the meaning of each node and system dynamic label depends on what was said both before and after a node is identified.

Table 1. Demonstration of coding process using a segment of Group 1’s transcript in Study A

Figure 2. Study A – Partial system map from Group 1’s transcript.

In addition to the labeling of nodes and system dynamics, a short label was also assigned to the solution ideas generated and these were noted when they appeared in the transcript. As might be expected, the participants frequently referred to previously designed products (i.e., a smart light bulb able to change the amount of light emitted based on schedule) that might have influence over the waking up experience. Nonetheless, an effort was made to track the emerging ideas and to provide a short label to them in a similar way to nodes as we anticipated that the solution ideas might be useful when discerning framing activity.

Assessing coding reliability

In most protocol studies, predetermined codes are assigned to the segments by two or more independent coders, which enables assessment of interrater reliability. However, in this case, a common coding scheme is absent because nodes and system dynamics emerge throughout the protocol. Our coding scheme shares similarities to how linkography studies are conducted (Goldschmidt, 2014). In linkography, design moves are brief acts of thinking and are generated sequentially in time. There is an assumption that the moves exist in each segment of data which only requires the coder to identify links between moves, which are not known in advance (Goldschmidt, 2014, p. 47). In the case of our approach, it is not assumed that each utterance produces a node or system dynamic, rather they are revealed based on the interpretation of the coder. Given this similarity we recognize that defining nodes and system dynamics, like identifying links in a linkography study, requires that the coder has good acquaintance with the discipline and with the design episode in question (Goldschmidt, 2014, p. 47). As such, the design transcripts were reviewed several times by all coders before the coding process began so they would understand what occurred in the session.

Though there are no specific methods for determining a reliability score, like you would do in a similar analysis (Goldschmidt, 2014), we tested our approach internally first, by having the first and second author (Coders A and B) independently code two of the eight transcripts – those of Group 1 and Group 2. Coder A then carefully compared the lists of nodes generated by the two coders and checked for similarity. Two nodes were deemed to be equivalent between two coders’ lists if their assigned labels used the same or similar language (e.g., ‘number of times someone snoozes’ and ‘snoozing’) and if they were identified at the same time point in the transcript. This analysis yielded the following results:

• • In Group 1’s protocol, Coders A and B identified 24 and 31 nodes respectively. Of those, 18 were equivalent between the two coders, representing 75% and 58% of the nodes identified by Coders A and B, respectively.

• • In Group 2’s protocol, Coders A and B identified 23 and 19 codes respectively. Of those, 18 were equivalent between the two coders, representing 78% and 95% of the nodes identified by Coders A and B, respectively.

We conducted a similar analysis on the identified system dynamics. System dynamic definitions rely heavily on how nodes are defined so any discrepancy in coding of nodes early on in the process can significantly impact the system dynamics that are defined later. As such, it is more challenging to determine if two system dynamics identified by different coders represent the same relationship. We used the following two-step process:

Step I: For each coder, we first identified all system dynamics that connected pairs of common nodes identified in the analysis above (18 nodes in Group 1, and 18 nodes in Group 2). Coders A and B identified 15 and 7 such system dynamics, respectively in Group 1, and 15 and 18 system dynamics in Group 2.

Step II: From the lists compiled in Step I, we identified equivalent system dynamics (i.e., that connected equivalent respective nodes) between coders. For example, the system dynamic [‘coldness of bed’ influences ‘quality of waking up experience’] identified by Coder A was considered equivalent to the system dynamic [‘coldness of bed’ influences ‘waking up experience’] identified by Coder B. For Group 1, there were 4 such system dynamics, representing 27% and 57% of the system dynamics identified in Step I for Coders A and B, respectively. For Group 2, there were 11 such system dynamics, representing 73% and 61% of the system dynamics identified in Step I for Coders A and B, respectively.

Discussions of discrepancies in the identified nodes and system dynamics between the two coders served to further strengthen Coder A’s ability to accurately identify nodes in the transcript. All six remaining transcripts in Study A were coded by Coder A. The results of Study A are presented in Section “Study A” results. We next explain the motivation for Study B and describe the second dataset on which the approach was tested.

Study B

Study A captured participants tackling a very ill-structured problem prompt, which required them to spend most of the design session exploring the problem and identifying promising aspects for ideating solutions, i.e., framing. Once the coding and analysis process of the eight transcripts of Study A was complete, there was concern that the developed approach might have been tailored too closely to that dataset, potentially leading to overfitting. To address this concern, Study B was conducted to further validate the developed approach in the later stages of the design process. We tested the coding scheme’s generalizability to a new set of protocols: a different set of designers working on a different, more well-defined problem that drove them to spend more of their time in the solution ideation and evaluation phases. We expected that because there was a significant difference in nature of the design prompt, we would observe noticeable differences in the structures of the system maps.

The verbal protocols analyzed in Study B were collected as part of a study conducted during a workshop at the DESIGN 2018 conference (Nespoli and Isaksson, 2018). The dataset was used with permission. In the workshop, design practitioners and academics worked on a case study that challenged them with creating conceptual designs for a mechanical device. The device needed to enable an artist to hold and sculpt glass vessels using her existing tile cutting saw and technique. The case included a full description of the artist’s workshop, photos and videos of her working, responses from an interview where she explained some of her requirements and desires, and details of the equipment, measurements of the work area, and a maximum estimated budget. The case was well defined so participants needed to spend little time on problem finding and could spend more of the design session on generating and evaluating solution concepts.

Four groups each with four participants participated in the design sessions. Participants audio recorded their discussions as they worked on the case study. Those recordings were later transcribed using a professional transcription service. Only the transcripts from two groups (one consisting of self-identified “practitioners” and one consisting of self-identified “academics”) were used for the purpose of Study B, each 46 minutes and 78 minutes in length, respectively. The same process explained in Section Development of new approach for coding verbal protocols was used to code the transcripts, by the same person (Coder A), who coded all eight protocols in Study 1.

In both Study A and Study B, once a transcript was coded, the data comprising the system nodes and dynamics was manipulated into a format that could be read by Gephi (Bastian et al., 2009), an open-source network visualization and analysis platform. Gephi offers a variety of rendering and data analysis tools that are useful for understanding the structure of the system map.

Results

Study A results

The number of nodes and system dynamics (or “elements”) coded in each of the eight transcripts provide two simple attributes by which to characterize the system maps (Figure 3). Though groups were given approximately 30 minutes to work on the task, there is a notable variation in the protocol lengths between groups. The variation is also reflected in the number of nodes and system dynamics: a linear regression model between the number of elements coded and length of the transcript (in words) indicates a positive relationship (β = 0.726, p-value <0.05).

Figure 3. Number of nodes and system dynamics for system maps for each group (G) in Study A

To observe how the system maps evolved over time, each protocol was divided into 20 equal segments (ventiles). Figure 4 presents a cumulative graph of the number of new elements coded in each ventile, for each of the groups. A general pattern is observed: in the early parts of the sessions there is a rapid emergence of new coded nodes and system dynamics, as participants begin to explore the problem. For most groups, emergence of new coded elements plateaus about halfway through the transcript. It is observed that, typically, at this point the participants’ utterances change in focus to the generation of solution ideas or revisiting topics that were already previously coded as nodes and dynamics. Only the first occurrence of coded nodes and systems dynamics are tracked.

Figure 4. Cumulative count of nodes and system dynamics over time for each group’s (G) system map in Study A.

Community analysis

Visual inspection of the resulting system maps reveals that the nodes are organized in “hub-and-spoke” and other structured clusters (hereon referred to as “communities”). A modularity algorithm built into the Gephi program was used to computationally determine the boundaries of communities of related nodes based on map structure. The algorithm utilizes the Louvain method (Blondel et al., 2008). It uses a modularity score (between −1 and 1) that assesses link density inside a community compared to those links outside the community. The process occurs in two phases. In the first phase, each node in the network is assigned to a community of their own. Then a local maximum of modularity is iteratively determined assessing modularity each time the closest neighbor is added to an existing community. Once this local maximum is identified, those communities then become a node in a new network and the process is repeated until modularity is optimized.

To investigate the relationship between nodes clustered in a community, we selected a subset of communities (each made up of a minimum of three nodes) and qualitatively evaluated their nodes. Inspection of the communities indicated that each related to a particular aspect of the problem, prompting different kinds of solutions. To demonstrate this finding, consider the maps of Group 3 (Figure 5) and Group 6 (Figure 6), each containing a total of five communities. Table 2 presents a list of the communities (and respective nodes) in Groups 3 and 6 that share a common theme. For each community pair, a label describing the inferred theme is provided in the left column with the associated colour in the system maps. The table also includes a summary of the solution ideas (coded in the transcripts) that relate to each of the communities.

Figure 5. System map for Group 3 in Study A with colour indicating communities.

Figure 6. System map for Group 6 in Study A with colour indicating communities.

Table 2. Community labels and nodes for Groups 3 and 6

There are noticeable similarities and differences associated with each of the communities found in the two system maps presented here:

• • The two communities that fall under the “Activity before going to sleep” theme (blue) only have one common node, ‘Amount of phone use’. Group 3’s map provides a clearer picture of activities one might complete before going to bed.

• • The two communities that fall under the “Factors that influence the ability to wake up” theme (purple) include nodes that involve the role of the senses (‘amount of sense stimulation’, ‘quality of alarm’, ‘amount of natural sound’) during the waking up experience.

• • Finally, both groups have a community that falls under the “Ability to fall asleep” theme (yellow), including common nodes such as the ‘amount of substance use’, and the amount of ‘stress’ or ‘worry’. In this case, the nodes in the community have a focus on those factors that might prevent a person from sleeping.

After identifying a theme for each community across all eight groups, it was determined that many of the communities were similar between groups. Table 3 presents all detected communities, ordered from most common to least. As was demonstrated in the example above with Groups 3 and 6, although the communities share a similar theme (and may even share some of the same node labels), the entire subset of nodes in the community is unique to that group. That is, no two communities between different groups shared exactly the same nodes. Overall, Table 3 demonstrates that some groups cover a wide variety of communities which may indicate a greater exploration of the problem and diversity in overall themes. For example, Group 6 has covered five out of the seven most common communities, while Group 7 has covered none of them.

Table 3. Communities detected in all eight groups; communities identified in more than one group are marked with (*)

Individual contributions in the teams

We sought to understand how the system maps might help determine the contributions of individual participants to the team’s problem exploration. In this section, individual participants are referred to using their participant ID which consist of their group number (G1–G8) and their assigned participant number (P1–P3). Nodes in the system maps are colored to indicate the participants whose utterance generated each node. The resulting maps are then compared against the community analysis system maps presented in Section Community analysis.

Taking for example Group 6’s system map colored by individual contributions (Figure 7) we note the diversity in participant ownership of nodes within each community. For example, of the six nodes that make up the community circled in red, three are identified in G6P3’s utterances (green nodes), two in G6P2’s utterances (blue nodes) and one in G6P1’s utterances (orange node). This is an example of a community that comprises of nodes from each of the three participants. In contrast, of the four nodes that make up the community circled in black, three are identified in G3P2’s utterances.

Figure 7. Group 6 system map marked up with individual contributions (in color) and communities (encircled).

The ownership of system dynamics can describe whose speech connects nodes in the system. We define and observe three types of system dynamics:

• • Type I, occurs when a participant’s system dynamic connects two nodes generated by that same participant (i.e., self to self).

• • Type II, occurs when a participant’s system dynamic connects one of their own nodes to a node generated by another participant (i.e., self to other).

• • Type III, occurs when a participant creates a system dynamic that connects nodes that were not generated by that participant (i.e., other to other).

To explore any patterns in their prevalence, we calculated the frequency of each system dynamic type in each of the eight groups (as shown in Figure 8). The most common types of dynamics across all groups are those of type I or II. This is unsurprising: often, more than one element, usually a node and system dynamic pair, can be coded within the same segment of a participant’s utterance. In contrast, there is a limited number of Type III system dynamics, which can only occur after two nodes are identified in the transcript and a connection between them is later made by a different participant. Their prevalence ranged from none (in Group 2) to three (in Groups 3 and 8). These patterns, especially with respect the relative frequency of Type II and III dynamics, could hold some significance to the level of ‘cohesion’ (Kelly and Gero, 2022) held by the team members in relation to the problem framing, as further elaborated in the Discussion.

Figure 8. Frequency of system dynamics, by type, for each group (G).

Study B results

This section describes the analyses conducted on the system maps generated from Study B, following a similar structure to Study A (Section Study A results).

System map characteristics in comparison to Study A

Figure 9 presents a comparison of the number of system nodes and dynamics coded in the academics’ and practitioners’ system maps and the average of the eight groups in Study A. It is observed that the number of system dynamics coded in the two Study B groups is noticeably smaller than the number coded on average in the Study A groups. Indeed, the ratio of transcript length (in words) to system map elements was on average 231 in Study B, compared to an average of 86 in Study A. In other words, system nodes and dynamics were coded in the Study A transcripts at a ratio almost three times that of Study B. We elaborate on the significance of this finding in Section Appropriateness of the approach in different stages of the design process.

Figure 9. Comparison of number of nodes, system dynamics, and protocol length between Study A and B.

Figure 10 presents a comparison of the cumulative count of elements added to the system maps over time. We note that the graphs of the academic and practitioner groups of Study B are similar to each other, but quite different from the average of the eight groups in Study A. Though the tapering off pattern can still be observed, it happens much earlier in Study B (approximately in the 3^rd to 6^th ventile) than in Study A (approximately in the 12th ventile).

Figure 10. Cumulative count of elements (nodes and system dynamics) in system maps over time (Study A and B).

Community analysis and individual contributions to teams.

The two system maps in Study B (Figures 11 and 12) share a total of nine nodes with similar labels (i.e., they have the same meaning for both groups). More than half of the nodes (marked with an (*)) in each of the two maps are directly tied to the information found in the design brief. The system maps do not share any common system dynamics.

Figure 11. Study B, academics’ system map.

Figure 12. Study B, practitioners’ system map.

Due to the limited number of system dynamics, there was only one community detected in each of the system maps, each colour-coded in gray in Figures 11 and 12. In the academics’ system map, the sole community is composed of six nodes, and its focus appears related to those elements (e.g., ‘the ability to moderate movement’) that would influence the overall feel for the process of glass carving. This was reflected in the solution ideas developed by the group, which focused on a system able to support the piece of glass, while allowing for movement of the piece with some level of precision. The community found in the practitioners’ map, which is composed of three nodes, seems to focus on the water used in the user’s process and how this might influence the quality of the grip. Not all the solution ideas directly mapped to this central focus, though there was discussion of some kind of sling that could be used to support the weight of the piece. Each of the communities included elements from all participants in each group but the frequencies of different system dynamic types are so low that no meaningful comparisons could be conducted to match those described in Section Individual contributions in the teams.

Discussion

We have described and tested a novel approach to coding verbal protocols of design for representing and analyzing framing activity in design teams. The theoretical basis for the approach is the observation that systems as mental representations of a problem (Espejo, 1994) bear conceptual similarity to frames as cognitive “conceptual assemblages” – collections of loosely connected concepts, implicit in the designer’s cognition (Kelly and Gero, 2022). It follows, then, that a coding scheme that identifies “nodes” and “dynamics” in designers’ verbal utterances during a team design activity may produce some degree of mapping to external representations of those implicit concepts and their relationships that make up the designers’ frame.

Systems mapping techniques are routinely utilized to construct representations of systems (sometimes from qualitative data, such as interviews) to inform possible design interventions to improve on some aspect of the system’s operation (e.g., Eden et al., 2000; Love et al., 2009; Newberry and Carhart, 2024) – this is their intended use. In contrast, the approach described in this work adapts systems mapping techniques into a coding scheme of verbal protocols of design activity – to model not an existing system, but rather speech representations of framing activity in the designers’ cognition.

The approach was tested on two separate sets of verbal protocols – in Study A where participants we presented with a brief and open-ended prompt of a problem area, and in Study B where participants were provided with a very detailed design brief of a well-defined problem. Careful analysis of the verbal protocols from Study A and B was conducted to identify excerpts that could be coded as nodes and dynamics, noting when each node and dynamic was first identified, and the individual participant whose utterance produced the excerpt. These elements are believed to provide a representation for what is within the designers’ attention (or frame) at some point during the session. The system maps, then, generated from all identified nodes and dynamics once all coding is complete are viewed as a cumulative representation of the design framing activity throughout the entire design session.

The aims of this research were exploratory; below we discuss the results from the two studies, suggest ways in which the approach may provide insight into framing in a design team setting, and discuss the study limitations and future research opportunities.

Communities as representations of design frames

In Study A, we observed differences between the eight groups of participants, both in terms of the total number of coded system nodes and dynamics as well as their resulting spatial arrangement in clusters, which enabled the detection of sets of related nodes, or communities. Using the content (label) of each node in the community, a community theme could be qualitatively inferred. Whereas we consider the system map a cumulative representation of the entire framing activity, each community could be considered a cumulative representation of a design frame. We offer two arguments to support this claim. First, returning to the theoretical comparison between systems as evolving mental representations and frames as conceptual assemblages (Espejo, 1994; Kelly and Gero, 2022), recall that the proposed approach is built on the idea that the system nodes and dynamics extracted by the protocol analysis can model the loosely connected concepts that make up a frame in the designer’s cognition; each cluster of connected nodes (i.e., community) can thus be considered to have a direct relationship to a different conceptual assemblage (or frame). Second, we consider how frames are capable of triggering solution ideas (Dorst, 2015, p. 63) and expect community themes (representing different aspects of the problem that came under focus during the design session) to be associated with different solution approaches. This was indeed the case in our data: for example, in their quest to improve the waking up experience, participants in Study A focused on activities before sleep (one community), or the ability to wake up naturally (another community), which prompted solutions like an alarm that counts down before an appropriate time to go to bed, and a device that lets in natural light in the morning, respectively.

If each community is taken to represent a different frame, then examining those communities may offer at least two ways to evaluate framing activity: First, the number of communities in a group’s system map may indicate the extent to which the design space has been explored. Across all eight groups of Study A, a total of nine communities with identifiable themes were detected, with seven of them common between at least two of the groups. The differing number of detected communities and their themes may provide insight into how groups vary in the quantity and range of frames considered. Second, if framing activity can be evaluated using the number and diversity of frames considered, as indicated by the detected communities, it may then be possible to also evaluate differences between two communities that represent a similar frame. One measurement could be the number of nodes - a community with more nodes would indicate a more thorough exploration of that particular frame. For example, in Study A Groups 3 and 6 both had the “activity before going to sleep” community in their maps, but Group 3’s community had twice as many nodes as Group 6’s (Table 2).

Relationship between frames and solution generation

Temporal analyses of the system maps, specifically the rate at which system nodes and dynamics emerge in the transcripts, can provide clues as to shifts in focus for the designers – from problem analysis to solution ideation. For example, given the brief and open-ended prompt, participants in Study A spent most of their time analyzing the problem. While new system nodes/dynamics are identified throughout the duration of the transcripts, the rate of additions is higher in the first half of the session, and new additions begin to plateau in the second half. The change is indicative of a shift in the groups’ focus, who at this point intentionally begin to focus on generating solution ideas. In study B, this shift happens much earlier. As it will be further discussed in Section Appropriateness of the approach in different stages of the design process, given the well-defined nature of the design brief in Study B, the participants begin their solution generation very early in the design session. This is reflected in the temporal distribution of identified system nodes and dynamics, the emergence of which plateaus earlier in the design session compared to Study A, as demonstrated in Figure 10. Cumulative temporal graphs that sustain an increasing slope for a longer portion of the design session may suggest that designers are engaging in a more significant process of problem-solution coevolution (Dorst and Cross, 2001; Maher and Poon, 1996), potentially demonstrating a more mature design process and higher design expertise (Cross, 2004). The approach thus has the potential to be used to compare designers across disciplines and/or levels of expertise, as has been previously done with other verbal protocol analysis approaches (e.g., Atman, 2019; Kavakli and Gero, 2002).

According to Dorst (2015), the central idea of framing is that designers spend time reasoning about their desired outcomes and create possible design solutions via their frames (p. 54). In his view, frames must be actionable, capable of triggering solution ideas and leading to realistic solutions (p. 63). The value of the design frame may therefore by assessed by the utility in relation to generating solutions. In our study, by noting the solution ideas generated by each group throughout the transcript, we were able to relate solutions to specific communities of nodes. A map with many, highly populated clusters may indicate more flexible ideation (Guilford, 1957) – that is, ideas that diverge into new and unusual directions - and thus more promising solutions compared to a map with few and/or small clusters. Though the number of solution ideas related to a particular frame might be useful in determining that frame’s quality, a more rigorous metric would combine an assessment of the quality of the final solutions with an assessment of a frame’s causal influence in creating that solution. Though it is possible that this analysis could be conducted on this study’s datasets, it was determined that this endeavour was outside the scope of this paper.

Appropriateness of the approach in different stages of the design process

Framing is a dominant activity in the early stages of the design process (Hey et al., 2007). If the described systems mapping approach adequately captures framing activity, then the characteristics of generated maps would depend on what stages of the design process are captured by the verbal protocol.

Participants in Study A were provided a very short, open-ended, and ill-defined problem statement comprised of a vague goal about improving the waking-up experience. As such participants needed to spend much of the time engaged in problem searching or scoping, identifying which aspects of the problem offered the best opportunity for a designed solution – that is, they were engaging in framing. Accordingly, the system maps in Study A were dense with nodes and system dynamics, providing rich representations of the groups’ framing activity. A linear regression analysis identified a statistically significant positive correlation between the protocol length (in words) and the number of nodes and systems dynamics identified (see Section Study A results) – the longer the groups spent on the design task, the more framing activity was captured by the protocol analysis.

In contrast, far fewer nodes and system dynamics were coded in the two protocols of Study B, even though the latter were longer. In fact, system nodes and dynamics were coded in the transcripts of Study A at a ratio almost three times that of Study B. This result is explained by the differences in prompts/briefs between the studies, which required participants to work in fundamentally different stages of the design process. The design brief in Study B was a detailed and well-defined mechanical engineering design problem, which required little task clarification and problem definition on the part of the participants. Instead, they could spend more of their time generating, evaluating, and explaining the details of solutions that would satisfy the requirements and constraints clearly set out in the design brief. Therefore, participants in Study B engaged in less framing activity, and as expected, fewer system nodes and dynamics were identified in their protocols.

Understanding framing in design teams

A significant motivator for this work is the need to more precisely characterize how framing unfolds in design teams. When individuals come into a design situation they have their own frames (Silk et al., 2021), which evolve over time over time through the social interaction between team members (Hey et al., 2007), and as members come to shared understandings of the problem (Dong et al., 2013). Yet, there is a lack of clear understanding in the literature about how to differentiate between frames in the designers’ cognition, external representations of those frames as the designer collaborates with others in the team, and frames as tools for thinking shared by the team (Kelly and Gero, 2022).

Here we take the view that the systems mapping approach described in this paper can provide a more precise way of investigating how the team comes to develop coherence in design frames over time, directly answering the call by Kelly and Gero (2022, p. 17). By assigning ‘ownership’ of nodes and system dynamics to the participant whose verbal utterance generated that element, the system map visualizations provide insight into framing in a team in at least two ways: First, the degree to which a frame is shared by the team may be related to the degree to which the community that represents that frame includes nodes contributed by all team members. In Study A, we observed that communities varied, from ones comprised exclusively of one participant’s nodes to others comprised of nodes generated by all group members. A second measure of the degree to which a frame is shared could be the prevalence of different system dynamic types, especially those of Type II and III, where a system dynamic connects at least one node “owned” by a participant other than the one whose utterance produced the system dynamic. This is in contrast to Type I system dynamics, where a participant’s system dynamic connects two nodes owned by that same participant. In Study A, in half of the groups, there were more Type II and Type III dynamics than Type I, a possible indication of better collaboration within the team in how each area of the problem is explored. This type of analysis would complement similar approaches to studying intra-group communication, for example, turn-taking analyses (e.g., Jiang and Gero (2017); Nespoli et al. (2021)).

Evaluating the validity of the approach, limitations, and future research directions

The fundamental claim that this paper makes is that the system maps produced by the novel coding scheme applied to protocols of design activity can model external representations of framing, which are considered to be in the cognitive domain of the designer(s). The approach was validated through testing on two distinct datasets: the method could detect elements (i.e., nodes and system dynamics) from verbal protocols of design generated in two different contexts, by different types of designers (including students, academics, and expert practitioners), working on two very different design prompts/briefs. Importantly, variations in design process (within the six protocols in Study A, and between the protocols in Study A and those in Study B) produced variations in the resulting system maps, for which a causal explanation could be offered - meeting the simple but powerful conception of validity proposed by Borsboom et al. (2004). For example, analyses of the generated maps identified patterns of differences between Study A and Study B that could be reasonably explained by the difference between Study A’s prompt and Study B’s brief and the significant differences in their resulting design processes.

The dataset used in Study B was the subject of a previous protocol study reported in Hurst et al. (2019), where it was analyzed using the Function- Behaviour-Structure (FBS) ontology (Gero, 1990; Gero and Kannengiesser, 2004). In that study, analyses of design issues suggested that participants placed significant focus on the solution space over the entire duration of the design activity (p. 1327–1328). This finding is in line with Study B results; the systems mapping approach is intended to capture activity in the problem space, and therefore produced small and sparse system maps (as also explained in Section Appropriateness of the approach in different stages of the design process). The approach could be further validated by testing it on datasets analyzed in one or more of the prior studies that have used protocol analysis to study framing specifically (e.g., Chandrasegaran et al. (2022), Kvan and Gao (2006), Valkenburg and Dorst (1998), Zahedi and Heaton (2017)), as briefly reviewed in Section Studying design framing.

Three methodological considerations are also worth highlighting. First, there is a need to establish a more robust process for determining coding reliability. In Section Assessing coding reliability we have described a two-step process for comparing nodes and system dynamics identified by two independent coders on two of the eight protocols of Study A and offer some preliminary metrics that are akin to interrater reliability metrics reported in similar protocol analysis studies (e.g., linkography (Goldschmidt, 2014)). However, as explained in that section, because of the nature of the coding process in our approach, interpreting these values is challenging. Second, the protocols on which the approach was developed and tested were on average 34 and 62 minutes long in Study A and B, respectively. The approach requires the coder to track utterances throughout the session, with new system dynamics potentially connecting nodes coded much earlier in the protocol. A potential limitation is thus the degree to which the coding method can be applied to longer sessions of design activity. Such situations are similarly encountered in linkographic analysis, where, for example, a two-hour design session may be divided into tens of units, each only a few minutes long (Goldschmidt, 2014, p. 81). The coder may need to first carefully create partitions that are as self-contained as possible, while also ensuring that any major connections between utterances (nodes) in different partitions are tracked. A third and final consideration is that the approach, as implemented in the studies described in this paper, only codes the first occurrence (Gero et al., 2014) of a node and system dynamic, and does not track the subsequent times in which utterances refer to content that had previously been coded. Tracking of repeat occurrences is feasible - participants sometimes mention the same concepts/topics again later in the session. This temporal data, combined with (for example) centrality analyses in the maps to measure the relative “importance” of different nodes, may provide information about other design phenomena such as fixation.

A related useful direction in this research is the application of advances in natural language processing and more recently large language modelling to enable some level of automating of the identification of system nodes and dynamics from design protocols. This could prove useful not only for alleviating the effort that manual coding requires (especially for longer protocols) but also to provide some degree of objectivity to the generated elements. While generation of system maps may be challenging given the highly specific and contextual nature of system nodes and dynamics, other types of maps appear more feasible in the short term. For example, Gero and Milovanovic (2022; 2023) have demonstrated how the NLTK Python package can be used to characterize the design space and track its evolution over time. Other types of representations such as knowledge graphs (e.g., Mihindukulasooriya et al., 2023) can similarly be used. More generally, use of alternative maps such as concept maps (Novak and Cañas, 2008; Safayeni et al., 2005), whether manually or computationally generated, would require careful evaluation and comparison to the systems mapping approach described in this paper to evaluate how the alternatives differ in the way they represent design framing activity.

A final consideration we highlight is validating the potential causal relationship between the identified communities as representations of different frames and the outcomes of design activity, for example, the number and quality of generated solutions. The latter were identified in the Study A protocols; however, analyses were only exploratory. In particular, the solutions were used in the community analysis to provide some validation for the themes identified in each community but the relationship between generated solutions and associated communities was not investigated in a systematic way. However, the results highlight the promise of the approach for future work in this area of inquiry. For example, solution ideas could be evaluated using existing solution evaluation techniques, like the Analysis of Exploratory Design Ideation (AEDI) (Hay et al., 2019), which provides a framework for assessing the quality of ideas in the face of different problem interpretations.

Conclusion

This paper builds on the observed parallelism between the concepts of system thinking and framing in the designers’ cognition and proposes the use of a systems visualization tools as a means for modelling representations of framing in the designer’s speech. A novel protocol analysis approach that uses systems mapping as a coding scheme has been developed and tested on two distinct data sets of verbal protocols of design. The generated system maps provide unique visual representations of those elements that designers bring into focus as they work on understanding a problem and generating solutions to address it. Analyses of the maps, and comparisons of their characteristics both within and between the two datasets, have been discussed to suggest ways in which they can offer insight on framing activity in a team design setting. This new way of coding verbal protocols, inspired by the tools used in systems thinking practice, can be used in a variety of research contexts. Several research directions have been provided, highlighting the usefulness and significance of this approach. Repeated implementations of this method with different protocols will ultimately determine its usefulness in our ability to extend our knowledge of design thinking, learning, and practice.