A systematic review of protocol studies on conceptual design cognition: Design as search and exploration

Abstract

This paper reports findings from the first systematic review of protocol studies focusing specifically on conceptual design cognition, aiming to answer the following research question: What is our current understanding of the cognitive processes involved in conceptual design tasks carried out by individual designers? We reviewed 47 studies on architectural design, engineering design and product design engineering. This paper reports 24 cognitive processes investigated in a subset of 33 studies aligning with two viewpoints on the nature of designing: (V1) design as search (10 processes, 41.7%); and (V2) design as exploration (14 processes, 58.3%). Studies on search focused on solution search and problem structuring, involving: long-term memory retrieval; working memory; operators and reasoning processes. Studies on exploration investigated: co-evolutionary design; visual reasoning; cognitive actions; and unexpected discovery and situated requirements invention. Overall, considerable conceptual and terminological differences were observed among the studies. Nonetheless, a common focus on memory, semantic, associative, visual perceptual and mental imagery processes was observed to an extent. We suggest three challenges for future research to advance the field: (i) developing general models/theories; (ii) testing protocol study findings using objective methods conducive to larger samples and (iii) developing a shared ontology of cognitive processes in design.

1 Introduction

In his work on the principles of engineering design, Hubka (1982, p. 3) notes that designing ‘is a very personal activity, and can probably only be performed by one person as an internal and somewhat subjective process’. The nature of design as an internal cognitive activity has been a focus of design research for a number of decades, with Cross (2001, p. 79) citing studies by Charles Eastman in the late 1960s as the starting point for much of the enquiry in this area. Since then, there has been a proliferation of empirical studies on design cognition (Dinar et al. 2015). That is, the cognitive processes and information used by designers whilst designing (Visser 2004). Dinar et al. (2015) reviewed empirical design cognition studies published over the past 25 years, noting that the majority have focused on the early, relatively ambiguous stages of the design process known as conceptual design (McNeill, Gero & Warren 1998; Suwa, Gero & Purcell 2000; Goel 2014). However, in spite of the considerable body of empirical work, several authors highlight that the nature of the cognitive processes involved in conceptual design remains unclear (e.g. Dorst & Cross 2001; Jin & Benami 2010; Kim & Ryu 2014).

The systematic literature review is a commonly applied research method in scientific fields. It involves gathering and synthesising all publications on a particular phenomenon that meet prespecified inclusion criteria, ensuring coverage of all relevant evidence and minimising bias. Systematic reviews conducted in accordance with established guidelines (e.g. the PRISMA statement (Moher et al. 2009)) are rigorous, transparent and reproducible, and therefore held to the same standard as empirical research. A systematic review of empirical design cognition studies could clarify the cognitive processes involved in conceptual design, revealing common findings as well as differences in perspectives. However, there has thus far been only a single systematic review in this area (Jiang & Yen 2009), focusing largely on methodological aspects of protocol analysis. Other authors provide extensive and informative reviews of the area (Cross 2001; Coley, Houseman & Roy 2007; Dinar et al. 2015), but these are not systematic and go beyond protocol studies to look at other approaches.

This paper reports findings from the first systematic review of protocol studies focusing specifically on conceptual design cognition, aiming to answer the following research question: What is our current understanding of the cognitive processes involved in conceptual design tasks carried out by individual designers? Protocol analysis involves interpreting subjective verbal reports of a designer’s cognitive processing provided during or after completion of a design task (Ericsson & Simon 1984; van Someren, Barnard & Sandberg 1994; Gero & Tang 2001), along with other aspects such as sketches and motor actions (Suwa, Purcell & Gero 1998a ; Park & Kim 2007). Whilst the merits of protocol analysis are widely debated (Lloyd, Lawson & Scott 1995; Suwa & Tversky 1997; Sarkar & Chakrabarti 2014), several authors suggest that it is one of the only methods capable of directly revealing the internal processing of designers (van Someren et al. 1994; Lloyd et al. 1995; Cross 2001; Sarkar & Chakrabarti 2014). We reviewed and synthesised 47 protocol studies spanning architectural design, engineering design and product design engineering, leading to the identification of 35 distinct cognitive processes. Owing to space limitations, this paper reports a subset of 24 processes investigated in 33 articles, namely those aligning with two viewpoints on the nature of designing discussed in the broader literature (e.g. Logan & Smithers 1993; Maher & Tang 2003; Sim & Duffy 2003): (V1) design as search (10 processes, 41.7%), where designing is viewed as a search process transforming knowledge states in a problem space; and (V2) design as exploration (14 processes, 58.3%), where designing is viewed as an exploratory process operating between problem and solution spaces. Our review forms part of a broader effort to provide a more unified view of the field, and the remaining cognitive processes will be reported in a future paper on this theme.

Methods and sample characteristics are outlined in Section 2, before the review findings are explored in depth in Sections 3 and 4. A discussion is provided in Section 5, and the paper concludes with a brief summary in Section 6.

2 Methods and sample

Our approach was informed by the PRISMA statement, consisting of a generic four-phase flow diagram and checklist providing formal guidance on conducting and reporting systematic reviews (Moher et al. 2009). Figure 1 presents a flow diagram for our review specifically. We undertook four of the recommended review phases: (i) identification of candidate articles; (ii) screening the abstracts of candidates for relevance; (iii) defining inclusion criteria for the review, and using these to assess the eligibility of relevant full-text articles and (iv) qualitatively synthesising the final sample of eligible articles. We consulted the checklist to guide our reporting of the review findings, although we found several elements to be irrelevant. In particular, Moher et al. (2009, p. 1) note that whilst the checklist covers items relating to both qualitative synthesis and statistical meta-analysis of reviewed articles, statistical methods ‘may or may not be used to analyze and summarize the results of the included studies’. Meta-analysis was not appropriate for this work owing to the qualitative nature of the findings reviewed (types of cognitive process), and thus we employed qualitative synthesis alone.

The review centred on conceptual design tasks carried out by individual designers; group-based tasks were excluded. To enable comparison and synthesis of cognitive processes from different studies, we reviewed work from three domains similar in terms of their approach to design problems and their fundamental focus on function (i.e. the purpose of an artefact (Gero & Kannengiesser 2004), as opposed to cognitive function (Chan et al. 2008)), behaviour and structure (Hubka 1982; Gero 1990; Roozenburg & Eekels 1994): architectural design (e.g. Goldschmidt 1991); engineering design (e.g. Lloyd & Scott 1994) and product design engineering (e.g. Dorst & Cross 2001). We consider the latter to include tasks incorporating industrial design requirements (e.g. aesthetics, usability and ergonomics) as well as technical requirements. Two design researchers with expertise in product design engineering (RDes1 and RDes2) selected and reviewed articles, receiving regular input from a cognitive neuroscience researcher (RCog). The article selection process, sample characteristics and qualitative synthesis approach are elaborated below.

2.1 Article selection process

Literature was gathered between 27th March 2015 and 3rd April 2015. Major engineering/design and psychology databases were searched (Compendex, Design and Applied Arts Index, Technology Research Database, Embase, PsycINFO and PubMed), in addition to general scientific databases (Science Direct and Web of Science). As Table 1 shows, we structured our search terms into four groups reflecting the various aspects of our research question. Search terms were generally applied across the title and abstract fields, and searches were conducted across the broadest date range permitted by each database. Where possible, searches were limited to English results only. A total of 6796 articles were obtained (Figure 1).

Figure 1. Flow diagram of systematic review process (based on generic diagram in Moher et al. (2009)).

Table 1. Structure of search terms

As Figure 1 conveys, following de-duplication of search results we arrived at 4996 articles reporting a variety of study types, including controlled performance tests, protocol studies, literature reviews, surveys and case studies. As discussed in Section 1, we sought to answer our research question by identifying cognitive processes from empirical studies. Given the general view that protocol analysis is the most capable method for revealing such processes, we decided to focus the review on protocol studies alone. To maximise coverage, we ran follow-up searches (9th October 2015) through the same databases searched initially using terms reflective of protocol analysis (e.g. protocol study, think aloud and verbalisation). During eligibility assessment (Figure 1), we evaluated full-text articles against six inclusion criteria presented in Table 2. Conference papers published pre-2005 were typically republished in later journal articles and thus were excluded (e.g. Suwa & Tversky 1996; Suwa, Gero & Purcell 1998b ; Suwa et al. 2001). We conducted reference list searches on included articles and assessed identified candidates against the same eligibility criteria.

Table 2. Inclusion criteria

2.2 Sample characteristics

In total, 47 articles were included in the sample. Given the focus on a subset of the review findings, only 33 are discussed in depth in this paper (denoted by * in the reference list). However, the full sample may be downloaded as supplementary material, and we report key characteristics of the sample in its entirety here (visualised in Figure 2). Articles date from 1979 (Akin 1979) to 2015 (e.g. Yu & Gero 2015), with 24 (53.2%) published in the last decade. The sample includes: (i) full protocol studies, where data was gathered, analysed and reported in a single study (36, 76.6%); and (ii) analyses, where previously gathered data was analysed and reported (11, 23.4%). Approximately 350 participants were involved, ranging from a minimum of 1 to a maximum of 36 per study ( $\text{mean}=7$ , $\text{median}=6$ , standard $\text{deviation}=6.30$ ). As Figure 2 shows, a sample size between 1 and 16 participants was most common. Broadly speaking, participants included undergraduate, Master’s and PhD students, as well as practicing designers and architects. Participants’ experience levels ranged from 0 to 38 years, although inconsistent definitions of ‘experience’ were observed. In addition, several authors did not provide information on participants’ experience (e.g. Chen & Zhao 2006; Chandrasekera, Vo & D’Souza 2013).

Figure 2. Key statistics for study characteristics.

We identified 45 distinct design tasks – 20 (44.4%) architectural design, 19 (42.2%) product design engineering and 6 (13.3%) engineering design. In certain cases, the same task was studied by multiple authors (e.g. Suwa & Tversky 1997; Suwa et al. 1998a , 2000; Suwa 2003). The following types of verbalisations were analysed: (i) concurrent (32 studies, 68.1%); (ii) retrospective (11 studies, 23.4%) or (iii) a combination of concurrent and retrospective (4 studies, 8.5%). Verbal protocols ranged from 15 minutes to 600 minutes. Six authors omitted length (Kavakli & Gero 2001; Chiu 2003; Maher & Tang 2003; Chen & Zhao 2006; Maher & Kim 2006) and in one study, the task was self-paced (Sun, Yao & Carretero 2013). Video of external behaviour during the task was additionally recorded in 38 studies (84.4%), and sketches were gathered in 24 studies (51.1%).

2.3 Qualitative synthesis

Cognitive processes and viewpoints emerged from the sample and were formalised through an iterative process of interpretation and refinement. Throughout, cognitive processes were identified on the basis of a definition offered by Poldrack et al. (2011, p. 3) in the cognitive neuroscience literature: cognitive processes are ‘entities that transform or operate on mental representations’. Mental representations are defined as ‘mental entities that stand in relation to some physical entity $[\ldots ]$ or abstract concept (which could be another mental entity)’. An example of a mental representation is a mental image of a visual scene, and a corresponding example of a cognitive process is ‘a process that searches a mental representation of the visual scene for a particular object’.

Articles were initially split between RDes1 and RDes2, who read the full text and identified types of processes studied across the sample (e.g. memory, semantic processes, perception, mental imagery and higher-order reasoning processes). These were discussed with RCog and continually refined. As the categories emerged, RDes1 and RDes2 recorded specific descriptions of cognitive processes pertaining to each in a common synthesis matrix. Several descriptions subsumed multiple process types (discussed in Section 5). Finally, RDes1 analysed the synthesis matrix to identify commonalities in the descriptions provided by different authors (e.g. similarities in descriptive terms and the nature of the representations related to processes), and abstracted more general descriptions where appropriate. The final list of processes was reviewed by all team members.

During the above process, both RDes1 and RDes2 observed persistent differences in the terminology and concepts applied to describe cognition in different studies. These were interpreted as reflecting two viewpoints (V) on the nature of designing: (V1) search, where designing is seen as a largely linear sequence of operators effecting changes to design knowledge states; and (V2) exploration, where designing is viewed as an iterative and situated process of interpreting the problem, proposing solutions and restructuring the problem and/or solution in response to features perceived in each (Newell & Simon 1972; Logan & Smithers 1993; Maher & Tang 2003; Sim & Duffy 2003; Visser 2006). In addition, numerous authors were found to discuss cognition more generally in relation to design activities (V3). Examples of design activities identifiable in the sample include problem analysis (Jin & Benami 2010), concept generation (Jin & Chusilp 2006), synthesis (McNeill et al. 1998), concept evaluation (Jin & Chusilp 2006) and decision making (Kim & Ryu 2014).

To provide a coherent framework for reporting the review findings, RDes1 assigned each of the articles to one of the above three viewpoints. Articles were searched for relevant keywords, e.g.: (V1) search, problem structuring, state transformation and operator; (V2) exploration, perception and situatedness and (V3) activities, concept evaluation and concept generation. Each article was interpreted by RDes1 and classified as pertaining primarily to V1, V2 or V3 based on the usage of associated keywords by the authors. For example, articles explicitly aiming to characterise search/problem structuring processes and evidence operators using protocol data, but making contextual references to keywords associated with design activities, were assigned to V1 (e.g. Stauffer & Ullman 1991; Goel 1995; Liikkanen & Perttula 2009). Articles focusing primarily on the conceptualisation and study of exploratory processes while making brief references to search keywords in background and discussion sections were assigned to V2 (e.g. Dorst & Cross 2001; Maher & Tang 2003). Note that whilst this activity was largely based on the judgement of RDes1, the resulting classification was reviewed by all team members and refined where necessary.

In total, 35 distinct cognitive processes were identified from articles on search (10), exploration (14) and design activities (13), with 2 processes overlapping search and activities, hence 35. This paper reports the 24 processes investigated in studies on design as search (Section 3) and exploration (Section 4), with design activities covered in a future paper. The processes are presented in Table 8 in Appendix A, where each is assigned a code identifier consisting of a viewpoint (V) and process (P) number, e.g. $\text{V}1_{\text{P1}}$ . We adopted the organisation and structure conveyed in Table 8 largely because it aligns with the manner in which processes are discussed by the authors investigating them. In turn, we found it to be the most conducive to clear explanation of the review findings. However, we acknowledge that other researchers may have different interpretations. In this respect, the review raises important ontological questions about how cognitive processes should be defined and organised for study in design research (Section 5.2).

3 Design as search

As discussed in Section 2, designing may be viewed as both a linear search process (V1) and an iterative exploratory process (V2). This section presents the 10 cognitive processes we identified from studies aligning with V1, which are summarised in Table 8 in Appendix A. For reference purposes, the code identifying each process in Table 8 is appended in superscript to the first mention of each process throughout Sections 3 and 4.

3.1 Memory, operators, solution search and reasoning

Studies on design as search tend to view the designer as an information processing system (Chan 1990; Stauffer & Ullman 1991; Dorst & Dijkhuis 1995), where elementary processes called $\mathit{operators}^{V1P1}$ are enacted to transform information from input to output states (Stauffer & Ullman 1991); that is, to effect state transformations (Akin 1979; Goel 1995). The range of operators identified from our sample is summarised in Table 2; a detailed elaboration would contravene article space limitations. Note that instances are presented as stated by authors and have not been abstracted, hence similarities may be observed in certain cases (e.g. create and generate, no decision and suspend, etc.). Nonetheless, they may be broadly grouped into four categories reflecting various design activities: information gathering; comprehending, representing and structuring information; generating and synthesising; and evaluating and decision making. Several authors were found to provide evidence supporting high-level operator execution patterns for search and process management, termed search methods (Stauffer & Ullman 1991) and control strategies (Chan 1990), respectively (e.g. Chan 1990; Stauffer & Ullman 1991; Goel 1995; Kim et al. 2007). These are also briefly summarised in Table 3.

Table 3. Summary of operators, search methods and control strategies identified from reviewed studies

Chan (1990) suggests that during designing, a designer firstly retrieves a schema relevant to the design problem from long-term memory $^{V1P2}$ . Schemas may be viewed as abstract knowledge structures, containing both declarative knowledge about design problems and procedural knowledge in the form of operators (Ball, Ormerod & Morley 2004). For instance, Akin (1979) suggests that schemas comprise an input state (declarative knowledge), an output state (declarative knowledge) and the set of operators required to convert the input state to the output state (procedural knowledge). Following schema retrieval, operators are extracted and activated in working memory $^{V1P3}$ , which supports the maintenance and manipulation of information (Chan 1990; Stauffer & Ullman 1991). In addition to information retrieved from long-term memory, certain operators serve to gather information from external sources (see Table 3). The designer interacts with the external environment via receptors and effectors, which receive afferent information and exert external effects, respectively (Newell & Simon 1972; Stauffer & Ullman 1991). These interactions may involve an external memory system – that is, resources such as sketches and notepads where ideas and thoughts may be recorded and stored externally, as well as information sources such as textbooks, databases, etc. (Stauffer & Ullman 1991; Goel 1995).

Collectively, the above processes may be described as ‘a search through $[\ldots ]$ knowledge states guided by information accumulated during the search’ (Chan 1990, p. 64). That is, the process of solution search $^{V1P4}$ , which is considered to be delimited by a problem space. Conceptually, the problem space constitutes ‘a representation of [a designer’s] task environment’ (Newell & Simon 1972, p. 59), incorporating knowledge of an initial problem state, a goal state and all possible intermediate design states (Newell & Simon 1972; Chan 1990; Stauffer & Ullman 1991; Goel 1995). Solution search may be represented within this space as shown in Figure 3, i.e. a series of state transformations originating in the problem state and culminating in the goal state (Chan 1990; Stauffer & Ullman 1991). Desired states to be attained during the search are specified in design goals (Akin 1979; Chan 1990; Stauffer & Ullman 1991). As Figure 3 illustrates, implementing constraints reduces the extent of the space to be searched (Chan 1990; Goel 1995), which is potentially large owing to the ill-defined and/or unstructured nature of design problems (Chan 1990).

Figure 3. The process of solution search.

Goel (1995, p. 119), proposes that solution search may be more fully described in terms of (i) lateral and (ii) vertical transformations (Figure 4). Lateral transformations involve ‘movement from one idea to a slightly different idea’, and vertical transformations ‘movement from one idea to a more detailed version of the same idea’. Goel (1995, p. 126) identified supporting evidence for both in a practicing architect’s protocol. Lateral transformations are argued to be ‘necessary for widening the problem space’, whilst vertical transformations ‘deepen the problem space’ as Figure 4 illustrates. More recently, Chen & Zhao (2006, p. 258) characterised the ‘cognition mode’ of automobile designers in terms of lateral and vertical transformations.

Figure 4. Lateral and vertical transformations within a problem space.

Eckersley (1988) and Lloyd & Scott (1994) additionally highlight the role of deductive and inductive inference $^{V1P5}$ in a search context, i.e. the process by which logical judgements are made based on pre-existing information rather than direct observations. More specifically, several authors consider the role of a type of induction known as analogical reasoning $^{V1P6}$ , referring to the use of information about known semantic concepts to understand newly presented concepts (Ball et al. 2004; Liikkanen & Perttula 2009). Ball et al. (2004, pp. 495–507) found that this may occur spontaneously during designing via ‘the recognition-primed application’ of knowledge schema retrieved from long-term memory. However, problems ‘resistant to schema-based processing’ may instead be solved via a conscious process of mapping solutions to previous problems onto the current problem. The latter may be viewed as a form of case-based reasoning $^{V1P7}$ , where new problems are solved on the basis of solutions to similar past problems (Chiu 2003; Ball et al. 2004).

3.2 Problem structuring

Akin (1979, p. 204) examined operator usage by a professional architect and concluded that prior to solution search, designers may employ a different set of operators to develop ‘an internally assimilated representation of the problem context’. This process is frequently termed problem structuring $^{V1P8}$ . Goel (1995, p. 114) observed that whilst ‘problem structuring occurs at the beginning of the task, where one would expect it’, it ‘may also recur periodically as needed’. Similarly, Chan (1990, p. 69) found that problem restructuring may be triggered by a ‘critical problem situation’ during solution search, e.g. a decision to abandon a particular solution. Problem restructuring results in changes to the solution path or problem structure, i.e. ‘the format of knowledge representation, goal plan and constraint establishment’. With respect to goal plan formation, Lloyd & Scott (1994), Liikkanen & Perttula (2009) and Lee, Gu & Williams (2014) study problem decomposition $^{V1P9}$ , i.e. the process of breaking a design problem down into sub-problems by specifying sub-goals (Liikkanen & Perttula 2009).

The process of problem reframing $^{V1P10}$ investigated by Akin & Akin (1996) in the context of sudden mental insight (SMI) may be considered to relate to problem structuring, although it pertains to the specification of problem frames as opposed to goals, constraints, and requirements. SMI refers to ‘the moment where the designer gets an insight into the design solution and/or the problem frame’ (Chandrasekera et al. 2013, p. 195), or what may colloquially be termed the ‘aha! response’ (Akin & Akin 1996, p. 344). Akin & Akin (1996) argue that to invoke SMI, designers must first recognise restrictive frames of reference and then specify new frames conducive to solving the problem. They propose that suitable frames of reference are determined using declarative and procedural domain knowledge retrieved from memory.

4 Design as exploration

In addition to the 10 cognitive processes discussed in Section 3, we identified 14 processes from studies aligning with the viewpoint that design constitutes an iterative exploratory process (V2). These are summarised in Table 8 in Appendix A and discussed in the following sub-sections.

4.1 Co-evolutionary design

A key characteristic of problem spaces in design as search is that the nature of the problem to be addressed, i.e. the search focus, does not change significantly over time (Dorst & Cross 2001; Maher & Tang 2003). An alternative perspective on design problems views them as evolutionary, i.e. subject to reinterpretation and reformulation over time as a solution emerges (Maher & Tang 2003; Yu et al. 2014). This is formalised in the co-evolution model of design, where the designer’s task environment is represented as two knowledge spaces (Figure 5): (i) the problem space, incorporating problem requirements and forming a basis for solution evaluation and (ii) the solution space, encompassing design solutions and providing a foundation for evaluating requirements (Maher & Tang 2003). Design is described as a co-evolutionary process $^{V2P1}$ , i.e. one that ‘explores the spaces of problem requirements and design solutions iteratively’ resulting in parallel evolution of design problems and solutions (Maher & Tang 2003, p. 48). Over the course of this process, new variables may be added into each space through interactions between the two (e.g. new requirements and potential solutions). The co-evolution model was originally proposed in computational form by Maher et al. (1996) and studied as a cognitive model by Dorst & Cross (2001), Maher & Tang (2003), Maher & Kim (2006) and Yu et al. (2014).

Figure 5. Co-evolution model of design. Reprinted from Design Studies, Vol 22/Issue 5, Dorst, K. and Cross, N., Creativity in the design process: co-evolution of problem–solution, pp.  425–437, Copyright (2001), with permission from Elsevier.

4.2 Sketch-based design exploration

In studies on sketch-based exploratory design, problem/solution interactions may be understood in terms of situatedness: the notion that what a designer draws and perceives in their sketches affects their interpretation of the problem and vice versa (Suwa et al. 1998a ; Reber 2011; Yu, Gu & Lee 2013). A prolific model of situated designing is the situated FBS framework (Gero & Kannengiesser 2004), applied to code protocols by authors in the sample (e.g. Yu et al. 2013). The following sub-sections discuss 13 cognitive processes we identified from sketching studies, along with findings from several studies on the purpose of conceptual design sketching.

4.2.1 Visual reasoning

Visual reasoning $^{V2P2}$ may be broadly defined as the process of generating and reasoning about ideas whilst engaged in sketching. It is conceptualised in two ways by different authors in the sample (Goldschmidt 1991; Park & Kim 2007). Firstly, Goldschmidt (1991) describes the process as a ‘dialectical’ reasoning pattern continually shifting between two modes:

• ∙ Seeing as $^{V2P3}$ (SA), i.e. the process of proposing attributes and properties for a design based on analogies between sketch elements and mental representations (e.g. semantic concepts and past experiences). Suwa et al. (1998a ) study a similar process called re-interpretation $^{V2P4}$ : assigning new functions to parts of a design through interpreting visuo-spatial elements and relations. Reinterpretation is classed as a type of functional cognitive action $^{V2P11}$ , discussed in Section 4.2.2.

• ∙ Seeing that $^{V2P3}$ (ST), i.e. the process of rationalising design decisions relating to proposals developed through SA.

Goldschmidt (1991, p. 125) characterised design moves (acts of design reasoning) performed by eight practicing architects during a sketching task as involving either ST or SA. Characterisation was based on arguments, or ‘rational utterance[s]’ considered to reveal reasoning modes. Participants were indeed observed to alternate between ST and SA throughout the task (Figure 6). Design moves were either (i) unimodal, i.e. characterised by one mode, or (ii) multimodal, i.e. characterised by two modes. Modal changes were observed in both directions, i.e. from ST to SA and vice versa. Goldschmidt (1991, p. 140) argues that this dialectical pattern of reasoning is ‘rather unique’ to sketching activities. Whilst ST and SA may be observed in designers working without sketching or focusing on abstract visual displays, ‘they are not organised in the dialectical pattern’ illustrated in Figure 6. Regarding the impact of different sketching tools in this respect, Won (2001) observed a dialectical pattern in the visual reasoning of designers carrying out both freehand and computer-aided sketching tasks, with the latter involving more frequent shifts between ST and SA.

Figure 6. Example of Goldschmidt’s (1991) dialectical visual reasoning pattern.

More recently, Park & Kim (2007) proposed a visual reasoning model (Figure 7) that formalises three interacting sub-processes termed seeing $^{V2P5}$ , imagining $^{V2P6}$ and drawing $^{V2P7}$ :

• ∙ Seeing involves perceiving, analysing and interpreting visual information from external representations, resulting in the merging of ‘empirical knowledge’ (e.g. test information) with ‘visual knowledge’ from long-term memory (Park & Kim 2007, p. 4).

• ∙ Imagining involves generating new internal images, which may be transformed according to schemas in long-term memory and maintained for externalisation (see below). Images may be generated using perceptual information produced by the process of seeing, and/or schemas.

• ∙ Drawing involves evaluating and confirming internal representations, and externalising such representations (e.g. through sketching).

Protocols gathered from an expert and a student designer were considered to provide evidence for all three of the above processes and their interactive nature. Park & Kim (2007, p. 10) claim the greatest interaction corresponded with the moment at which the ‘creative idea’ was generated, and in turn conclude that process interaction in visual reasoning leads to the emergence of creativity. However, it seems neither participant nor output creativity was assessed during the study.

Figure 7. Visual reasoning model (Park & Kim 2007). Copyright (2007), The Design Society; reprinted with permission.

4.2.2 Cognitive actions

During visual reasoning, designers think of and perceive various kinds of information in their sketches. Suwa & Tversky (1997, p. 388) propose four information categories: (i) emergent properties, e.g. spaces, things, shapes/angles and sizes; (ii) spatial relations, e.g. local and global relations; (iii) functional relations, e.g. practical roles, abstract features and views and (iv) background knowledge. Whilst the authors acknowledge interdependencies between these categories, they are not elaborated. However, Suwa et al. (1998a , pp. 458–459) argue that understanding these relationships is ‘key to understanding the ways in which designers cognitively interact with their own sketches’. Building on the above categories, they propose a set of cognitive actions $^{V2P8}$ – that is, interdependent cognitive processes argued to be involved in sketching. These are organised into four categories depending on the type of information they relate to (Table 4): (i) physical $^{V2P9}$ actions (sensory information); (ii) perceptual $^{V2P10}$ actions (perceptual information, specifically visual); and (iii) functional $^{V2P11}$ and (iv) conceptual $^{V2P12}$ actions (semantic information). The categories were developed from the perspective that ‘information coming into human cognitive processes is processed first sensorily, then perceptually and semantically’.

Suwa et al. (1998a , p. 458) claim the proposed actions are supported by ‘an enormous amount of concrete examples’ identified from an architect’s protocol. However, it is worth highlighting that the meaning of ‘cognitive action’ from a cognitive psychology perspective is somewhat ambiguous. The term is not clearly defined by Suwa et al. (1998a , p. 455), who appear to use it interchangeably with the (also undefined) phrase ‘design action’. They also argue that a designer’s ‘cognitive behaviours’ may be represented as a set of interrelated cognitive actions. These phrases are problematic for psychologists, who typically refer to internal mental processes as ‘cognitive’ and view ‘action’ and ‘behaviour’ as external phenomena. Suwa et al. (1998a , p. 472) further suggest that cognitive actions may be used for ‘dissecting the structures of a designer’s cognitive processes’, suggesting they constitute interdependent cognitive processes. We adopt this interpretation; however, it may be seen in Table 4 that the nature of the actions as either ‘cognitive’ or ‘behavioural’ in the psychology sense is frequently unclear. For instance, the cognitive actions of looking at previous depictions (physical action) and attending to visual features of sketch elements (perceptual action) likely involve cognitive processes such as selective attention and visual perception, but the actual acts of looking with the eyes and working on sketch elements with a pencil constitute external behaviour.

As discussed further in Section 4.2.4, Suwa et al. (1998a ) gained preliminary insights into the purpose of sketches by coding an architect’s cognitive actions. Cognitive actions have also been examined by several other authors for varying purposes:

• ∙ Kavakli & Gero (2002) coded protocols from expert and student architects, revealing differences in cognitive action trends over time and correlations between actions. For instance, the expert executed 6 simultaneous cognitive actions whilst the student executed 30. Drawing, perceptual, and functional actions and goals (Table 4) were also found to be more strongly correlated for the expert than the student. Kavakli & Gero (2001) offer potential explanations for these differences based on mental imagery and working memory theory.

• ∙ Bilda & Demirkan (2003, p. 49) analysed six architects’ cognitive actions while sketching in digital versus traditional media, and found ‘designers were more effective in using time, conceiving the problem, producing alternative solutions and in perceiving the visual–spatial features and the organizational relations of a design in traditional media’.

• ∙ Sun et al. (2013) analysed protocols gathered from 15 engineering students and proposed a relationship between cognitive efficiency and the number of: (i) cognitive actions executed; and (ii) transitions among different action categories and therefore, information processing levels. Participants with higher cognitive efficiency executed more perceptual actions and fewer conceptual actions, and exhibited more transitions from the physical to perceptual level but fewer transitions from the functional to conceptual level (Table 4).

Table 4. Cognitive action categories proposed by Suwa et al. (1998a )

4.2.3 Unexpected discovery and situated requirements invention

Suwa et al. (2000, p. 540) suggest that when a designer sketches a new feature intended to spatially relate to existing sketch features, unintended spatial relations are also ‘automatically produced’ regardless of whether they are actually heeded. Later in the task, visuo-spatial features created by these unintended relations may be ‘discovered in an unexpected way’ by the designer – a process termed unexpected discovery $^{V2P13}$ , and classed as a perceptual action (Table 4). Suwa et al. (2000) propose three types (Figure 8), namely discovery of: (i) the shape, size or texture of a sketch element; (ii) a spatial or organisational relation among elements and (iii) a space existing between elements, an example of what may be termed ‘figure-ground reversal’ in perception research (Suwa et al. 2000, p. 546). An architect’s protocol revealed that throughout the task, unexpected discoveries were correlated with cognitive actions to set up goals focusing on new issues, which are then abstracted and carried through the design process as requirements. Since the invention of requirements by a designer in this manner is ‘situated in the environment in which they design’ – that is, affected by their perception of sketches and the general context – the term situated invention $^{V2P14}$ or ‘S-invention’ is adopted (Suwa et al. 2000, pp. 539–547). Suwa (2003, p. 222) refers to unexpected discovery and S-invention in an overall sense as ‘problem-finding’, i.e. ‘discovering and formulating one’s own problem to be solved’. In more recent work, Yu et al. (2013, p. 411) compared architects’ responses to unexpected discoveries in parametric versus geometric design environments.

Figure 8. Types of unexpected discovery identified by Suwa et al. (2000).

4.2.4 The purpose of sketching

Generally, as Sections 4.2.1–4.2.3 show, sketching is considered to play a fundamental role in solution development during conceptual design. Suwa et al. (1998a , p. 483) conclude based on an architect’s observed cognitive actions that sketches serve several purposes:

• ∙ an ‘external memory in which to leave ideas for later inspection’;

• ∙ a ‘provider of visual cues for association of functional issues’ and

• ∙ a ‘physical setting in which functional thoughts are constructed on the fly in a situated way’.

However, several studies suggest that designers can still successfully develop a solution when unable to sketch. For example, Athavankar (1997, p. 38) analysed a protocol gathered from a blindfolded designer unable to sketch during a task. They suggest that compared with the use of mental imagery alone, sketching ‘makes lesser demands on the cognitive resource that would be otherwise spent on maintaining the mental image’, supporting the idea that sketches may serve as an external memory. Nonetheless, the overall conclusion is that removing the ability to sketch did not limit the ‘ability to propose design moves, reflect upon them, transform ideas into new spatial configurations, propose and compare alternatives, evaluate them, take decisions and argue them out’. Athavankar’s findings are supported in varying degrees by three later studies:

• ∙ Bilda, Gero & Purcell (2006, p. 604) found no significant differences in ‘design outcome scores, total number of cognitive actions (except for recall activity) and overall density of idea production’ in protocols gathered from three architects engaged in tasks under blindfolded and full-vision conditions.

• ∙ Based on protocols gathered from six architects under the same conditions as above, Bilda & Gero (2007, p. 364) conclude that sketching serves to off-load visuo-spatial working memory. However, they also conclude that for expert designers, the ‘use of tacit knowledge and the pre-existing chunks of spatial models from long-term memory could support the design process without the use of externalizations’.

• ∙ Athavankar, Bokil & Guruprasad (2008, p. 338) analysed protocols gathered from four blindfolded architects who were free to move around a room whilst completing a design task. Participants were observed to ‘work with remarkable dexterity in spite of (the) artificially imposed eye mask constraint’, employing two strategies where they either: (i) ‘moved in the real world space and carried the site (of the building) with them in their mind’s eye’; or (ii) ‘physically moved and paced within the stationary visualization of the site which they created in their mind’s eye’ (Athavankar et al. 2008, p. 329).

5 Discussion

In Sections 3 and 4, we explored 24 cognitive processes aligning with two viewpoints on the nature of designing: search and exploration. These were identified through a systematic review of protocol studies aiming to answer the following research question: What is our current understanding of the cognitive processes involved in conceptual design tasks carried out by individual designers? In this section, key observations relating to the research question are discussed, along with future work and the limitations of the review.

5.1 Key observations

A key advantage of a systematic review is its capability to expose common findings and differences in perspectives. In this respect, studies aligning with search and exploration appear to cover similar design activities at a high level. However, they vary considerably regarding the concepts and terminology used to describe the cognitive processes involved, as Table 5 illustrates in the context of four typical conceptual design activities (Sim & Duffy 2003; Jin & Chusilp 2006). Note that the mapping of cognitive processes to design activities in Table 5 is largely based on interpretation rather than explicit statements by authors. That is, similarities we identified in the definitions and outputs of design activities and cognitive processes. For example, generation activities involve producing ideas for solutions to design problems (Jin & Chusilp 2006). The integration and specification operators (cognitive processes) are described as producing spontaneous information and partial solutions, respectively (Table 5). Thus, we interpreted both as pertaining to the generation of ideas.

Table 5. Concepts and terminology used to describe cognitive processes involved in conceptual design activities

1 Code identifier (ID) elements: V  $=$  viewpoint number; P  $=$  process number. Please refer to Table 8 (Appendix A) for a list of authors investigating each process.

Variation between viewpoints may be at least partly explained by differences in two fundamental design cognition paradigms, namely: (i) the problem solving paradigm, founded in Newell and Simon’s theories and models of human problem solving (e.g. Newell & Simon 1972; Simon 1996); and (ii) the reflective paradigm, influenced heavily by Schon’s work on reflection-in-action (e.g. Schön 1983; Schon & Wiggins 1992). Each paradigm’s key perspectives (Dorst & Dijkhuis 1995) are presented in Table 6. It may be seen from Sections 3 and 4 that studies on search align primarily with the problem solving paradigm, whilst those on exploration align more closely with the reflective paradigm. Thus, it is perhaps unsurprising that the two sets differ so considerably.

In spite of the above differences, several broad commonalities between viewpoints may be observed. For instance, as Table 7 illustrates, studies on both search and exploration suggest the involvement of long-term memory retrieval in conceptual design tasks. Semantic (i.e. pertaining to meaning) and associative processes also appear to be commonly investigated, along with visual perception. Notably, we found that mental imagery is discussed fairly extensively in studies on exploration, but receives virtually no attention in studies on search. One potential explanation for this is the perspective that in design as search, a designer constitutes an information processing system in an objective reality (Table 6). In contrast, studies on design as exploration tend to view the designer as a person constructing their own reality (Table 6) in line with the notion of situatedness.

Table 6. Key perspectives associated with design cognition paradigms

The involvement of memory, semantic and associative processes in conceptual design is consistent with creative ideation accounts in the broader psychology and neuroscience literature (e.g. Mednick 1962; Runco & Chand 1995; Mumford, Medeiros & Partlow 2012; Benedek et al. 2013; Beaty et al. 2014; Abraham & Bubic 2015). Nonetheless, the above inconsistencies make it difficult to rationalise the range of cognitive processes identified from the sample. This obscures the fundamental nature of the processes studied by authors, and makes it difficult to identify specific avenues for future work. Difficulties in comparing and synthesising studies are compounded by terminology with little meaning in psychology, such as seeing as (Goldschmidt 1991), unexpected discovery (Suwa et al. 2000) and cognitive action (Suwa et al. 1998a ). The use of such terms is particularly curious in the case of cognitive processes that are well established and clearly defined in psychology research. For instance, the process of unexpected discovery is likely an instance of what psychologists call (visual) perceptual re-organisation: the process of re-organising visual information to reveal previously unseen features and relations of visuo-spatial representations (Bruce, Green & Georgeson 2003; Tversky 2014). Nonetheless, the term ‘unexpected discovery’ is preferred by Suwa et al. (2000). Another issue impeding rationalisation is that the terminology applied by authors appears at times to subsume multiple process types. For example, the process of seeing as seems to refer to the interaction of visual perception, semantic processing and mental imagery processing during sketching (Goldschmidt 1991).

5.2 Towards general formalisms

Dinar et al. (2015, p. 9) attribute the lack of standard approaches and interpretations across the field of design cognition research to a dearth of ‘cognitive models and theories of designer thinking’. In this respect, we suggest that the inconsistencies discussed in Section 5.1 expose a lack of general models and theories of conceptual design cognition. That is, formalisms that describe design cognition using an established common language, and have explanatory and predictive capability with respect to the general population of designers. Although general cognitive models of creativity such as the Geneplore model (Finke, Ward & Smith 1992) have been influential in design research (e.g. Jin & Benami 2010), they are not models of designing per se. Given that general formalisms are central to generating and testing scientific predictions about design cognition, and explaining the phenomena involved, their development constitutes a major challenge critical to advancing the field.

Table 7. Commonly studied cognitive processes

1 Code identifier (ID) elements: V  $=$  viewpoint number; P  $=$  process number. Please refer to Table 8 (Appendix A) for a list of authors investigating each process.

Models and theories of the type outlined above must ultimately be informed by generalisable and statistically significant findings about cognitive processes and their interactions during design tasks. In this respect, whilst protocol analysis may be well suited to exploratory investigations in underdeveloped research areas, a key question is whether it is the best method for advancing the field towards general scientific formalisms. A conclusive answer is beyond our scope, but we shall comment briefly on some observations arising from our sample. A well-documented limitation of protocol analysis is its resource-intensive nature, arising from the involvement of extensive qualitative data processing. Consequently, protocol studies are frequently limited to small samples (Dinar et al. 2015). This means that whilst statistically significant results may be obtained, they are subject to a high margin of uncertainty (Button et al. 2013). In a sample of 1, uncertainty is so high that the significance (or otherwise) of results is essentially irrelevant. Of the 33 articles reviewed in Sections 3 and 4, 64% employed a sample of five participants or less; 33% employed samples of two or less. Furthermore, column 4 in Table 8 demonstrates that of the 24 cognitive processes we identified, 50% were supported by a mean sample of five participants or less, and 17% by a mean sample of one or two. Thus, a considerable fraction of protocol study findings to date are likely subject to high uncertainty margins.

Based on the above, we suggest testing the results of protocol studies using methods conducive to larger samples is a key challenge for design cognition research. One potential approach, typical of cognitive psychology, is the use of controlled experiments employing cognitive tests and outcome measures (Dinar et al. 2015). This permits considerably larger samples owing to the quantitative nature of the methods; however, there are other limitations including reduced ecological validity, time restrictions, a lack of suitable tests and metrics, and reduced richness of the data generated (Shah, Smith & Vargas-Hernandez 2003; Shah et al. 2012, 2013; Khorshidi, Shah & Woodward 2014; Dinar et al. 2015). Thus, effectively combining rich, qualitative approaches like protocol analysis with more objective quantitative approaches may be a more fundamental challenge for design cognition researchers.

Finally, whilst identifying, defining, and organising cognitive processes to be reported in this paper, we were struck by the following question: what cognitive processes exist in a conceptual design context, and how should they be defined and organised for study? The structure of Table 8 presented us with an effective reporting framework; however, it may be less effective as a foundation for future empirical work. This discussion aligns with current efforts in cognitive neuroscience to develop a shared ontology of mental processes, representations, and tasks (Poldrack et al. 2011; Poldrack 2015). Whilst several design ontologies exist (e.g. Gero 1990; Sim & Duffy 2003; Gero & Kannengiesser 2004, 2007), they are not necessarily intended to describe cognitive processes and those that do are not comprehensive. For example, Gero’s situated FBS ontology describes design in terms of broad interpretation, transformation, and focusing processes, encompassing more specific cognitive processes that are only briefly mentioned (Gero & Kannengiesser 2004, 2007). A general, shared ontology of cognitive processes in conceptual design would not only provide a rational basis for model and theory development, but would also increase study comparability and foster a more integrated body of knowledge on design cognition. Furthermore, an ontology consistent with cognitive psychology and neuroscience would increase the capability for design cognition research to contribute to the broader body of scientific knowledge on human cognition.

5.3 Review limitations

We found that whilst a systematic review is a significant undertaking, it is a valuable approach for building a rich and comprehensive map of an area. Nonetheless, there are limitations and lessons to be learned. Firstly, we encountered considerable difficulties in defining suitable search terms (Table 1). For instance, applying the terms design* and protocol* returns hundreds of thousands of irrelevant hits from other fields owing to the ubiquitous phrases ‘experimental design’ and ‘experimental protocol’. Some of our early test runs returned over a million results. This, coupled with the multitude of potential cognition keywords (e.g. conceive, thinking, memory, fixation, etc.) makes it difficult to conduct an exhaustive search. Secondly, we found that the titles and abstracts of design research articles often lack key details on study focus and findings. This caused substantial difficulties during both the (i) identification and (ii) screening phases of the review (Figure 1): (i) search terms were not always reflected in the titles and abstracts of relevant articles, but searching the full text returned an unmanageable number of hits; and (ii) rather than screening abstracts to determine article relevance as suggested by the PRISMA statement, we were often forced to read larger portions of the paper thereby protracting the process considerably.

As the numbers in Figure 1 convey, we have covered a significant portion of the field; however, owing to the above issues, we have still missed certain relevant publications. For example, Danielescu et al. (2012) report a protocol study on problem structuring (relevant to Section 3.2), which meets our inclusion criteria (Table 2). This article was not captured by our searches as the combination of terms applied did not appear in the search fields. We spent significant time and effort trialling different search terms in an attempt to minimise the exclusion of relevant material. We would advise other systematic reviewers to do the same; furthermore, we would suggest that search terms are informed by a background review covering key terminology and concepts in the area of interest. We would also urge design researchers (ourselves and colleagues included) to consider the possibility that their articles may eventually form part of a systematic review when constructing titles and abstracts, especially given the rising number of these reviews in our field (e.g. Jiang & Yen 2009; Blizzard & Klotz 2012; Erkarslan 2013).

6 Conclusion

This paper reports findings from the first systematic review of protocol studies focusing specifically on conceptual design cognition. We reviewed 47 studies on architectural design, engineering design and product design engineering in order to answer the following research question: What is our current understanding of the cognitive processes involved in conceptual design tasks carried out by individual designers? In total, 35 distinct cognitive processes were identified. This paper reports a subset of 24 processes investigated in 33 studies aligning with two viewpoints on designing discussed in the broader literature: (V1) design as search (10 processes, 41.7%); and (V2) design as exploration (14 processes, 58.3%). Our review forms part of a broader effort to provide a more unified view of the field, and the remaining cognitive processes will be reported in a future paper on this theme.

Our central finding is that protocol studies vary considerably with respect to the concepts and terminology used to describe design cognition. Nonetheless, several broad commonalities may be observed, including a focus on memory, semantic, associative, visual perceptual and mental imagery processes. Differences between studies aligning with search (V1) and exploration (V2) may be partly explained by underlying paradigmatic differences; however, they also highlight a lack of general models and theories of conceptual design cognition. Given that these formalisms can be applied to generate and test predictions about design cognition that may advance knowledge in the field, their development constitutes a critical challenge. Two further suggested challenges for the field are: (i) testing protocol study findings using objective methods conducive to larger participant samples and (ii) developing a shared ontology of cognitive processes in conceptual design, to foster a more integrated and scientifically valuable body of knowledge on the phenomenon.