Feature, specification and evidence framework for communicating design rationale

Abstract

Design rationale is the justification behind a product component, often captured via written reports and oral presentations. Research shows that the structure and information used to communicate and document rationale significantly influence human behavior. To better understand the influence of design rationale on engineering design, we investigate the information engineers and designers include in design rationales in written reports. Eight hundred and forty-six pages of student engineering design reports from 28 teams representing 116 individuals were analyzed using a mixed-methods approach and compared across project types. The rationales from the reports were coded inductively into concepts and later applied to five industry reports consisting of 218 pages. The findings reveal a spectrum of rationales underpinning design decisions. Grounded in the data, the feature, specification and evidence (FSE) framework emerged as a feature-based and low-effort capture approach. We discuss the need to improve design communication in engineering design, through structuring rationales (i.e., using the proposed FSE framework or other representations) and improving technical writing skills. Lastly, by enhancing design rationale communication and documentation practices, significant benefits can be realized for computational support tools such as automatic rationale extraction or generative approaches.

1. Introduction

Documenting design rationale is a critical aspect of the engineering design process; however, what to include and at what level of detail is not standardized in teaching or practice. Design rationale has many definitions, capture methods and use cases (Moran & Carroll 1996). We adopt Lee’s definition of design rationale as “not only the reasons behind a design decision but also the justification for it, the other alternatives considered, the tradeoffs evaluated, and the argumentation that led to the decision” (1997). The scope of this research focuses on technical reports, which are an industry standard for engineering design. The reports for this study were gathered primarily from designs with tangible elements instead of entirely software-based. Engineers often compile their design rationales in such reports after the completion of the artifact. The use case for the technical reports and the design rationales in this study, is to communicate to other stakeholders (both internal and external) and serve as documentation of the justifications for the final artifact.

The primary significance behind design rationale usage centers on the artifact’s long-term success and financial implications. Rarely will a single individual be responsible for the entire design process; instead, engineers and designers need to work together and communicate with other members of a firm (e.g., supervisors, sales, marketing) and clients (Hirsch et al. 2001). A team that spends hours developing an innovative product may see the product fail due to poor positioning or poor communication. Additionally, a firm may spend unnecessary resources repeating previous mistakes that were not documented or tracking down rationale from previously completed design iterations. Research in engineering design has also found that information and tone used to reason and explain decisions affect human behavior (Dong, Lovallo & Mounarath 2015; Narayanan et al. 2018; Das & Chernova 2020). Therefore, adequately capturing design rationales is of high importance.

The high-level objective of the work aims to study how engineers and designers communicate design rationales in written technical reports for physical artifacts. A guiding question for the current study is:

What information is contained within design rationale in written technical reports?

To learn about the design rationale trends engineers use, the medium and information contained within rationales need to be investigated. More specifically, this article details how including specific information impacts communication effectiveness. Technical reports were used because they are an industry standard and hold more information than presentations (which tend to be briefer) or informal conversations (usually follow-up questions on specific components). The research was conducted inductively, using document analysis with open coding on 28 student final design reports, followed by deductively coding five industry technical design reports. Information contained in design rationales was derived from the data and compared across report types. Several theoretical concepts emerged, and a select few are the focus of this article.

We propose and discuss the feature, specification and evidence (FSE) framework for documenting design rationale in written reports. While the results from document analysis showed a range of communication effectiveness and rhetoric used, FSE is based on rationales that coherently communicate design justifications behind design decisions. Simplified rationale representations should be leveraged over more complex and time-intensive alternatives. Specific to the nature of technical reports, engineers need to go beyond describing product features or functions and link those features with defined specifications and evidence in a cohesive manner that can seamlessly integrate into their existing processes. The findings and limitations of this work are then situated within the broader field of design rationale. The article closes with implications and next steps for design education and practice.

2. Background

In engineering design, documentation is vital in justifying design decisions, integrating product and process information and facilitating knowledge transfer among stakeholders (Eckert et al. 2017). Design rationale has various definitions, interpretations and perspectives, making it a multifaceted concept that is understood and applied differently across domains (Regli et al. 2000; Sagoo, Tiwari & Alcock 2014). This background section provides an overview of the various definitions, capture methods, use cases of design rationale and the expectations surrounding communication and documentation practices in engineering design. By exploring these topics, we provide insight into the challenges associated with effectively capturing and using design rationales, specifically the design of physical systems.

2.1. Defining, capturing and using design rationale

Over time, the definitions for design rationale have evolved, expanding on the contents to include and the level of detail necessary. Central to design rationale is the “why” behind design decisions. One variation describes design rationale as the “reasoning underlying the design process that explains, derives, and justifies design decisions” (Fischer et al. 1996). A more detailed definition is, “Design rationales include not only the reasons behind a design decision but also the justification for it, the other alternatives considered, the tradeoffs evaluated, and the argumentation that led to the decision” (Lee 1997). While the first is inherently briefer in its purpose, the latter attempts to list what contents rationales should include. Each justifies design decisions, yet the scope of what qualifies as a design decision varies across definitions and domains. For example, a design decision might include every alteration to a CAD model (i.e., extrude in the x-direction for 2 cm, extrude in the y-direction for 6 cm) or simply the outcome of those decisions (i.e., enlarge design). The lack of specificity of what should be included (shared ontologies) in a design rationale can be attributed to the differing use cases and representations (Hay et al. 2017).

Design rationales serve a multitude of purposes, ranging from facilitating effective communication and collaboration among design team members to aiding in decision-making, enhancing knowledge management, and providing a basis for design evaluation and improvement. Which representations and capture approaches to implement depends on the intended use case; Some call for detailed information collected temporally, while others are collected after the completion of a design. Two dominant process-based representations include issue-based information system (IBIS) (Conklin & Yakemovic 1991) and Questions, Options and Criteria (QOC) (MacLean et al. 1991). IBIS frames rationale as issues (design questions), positions (answers to the design questions) and arguments (support/refute positions). An example from the Design Rationale editor (DRed) used by Rolls-Royce is shown in Figure 1 (Bracewell et al. 2009). Alternatively, for QOC, Questions identify key design issues, Options provide possible answers to the Questions and Criteria to assess and compare the Options. An example is shown in Figure 2 (Shum & Hammond 1994). QOC does not record the design process but is a coproduct of the design (created simultaneously with the artifact). Both representations provide examples of rationales branching out at multiple levels. There is rich design information in these representations (expressive); however, notable challenges fall onto the designer responsible for creating them.

Figure 1. IBIS representation from the DRed used by Rolls-Royce to diagnose a problem (Bracewell et al. 2009).

Figure 2. Generic QOC notation is used in Design Space Analysis (Shum & Hammond 1994).

Utilizing design rationale presents inherent challenges, including establishing a universally shared definition, selecting appropriate representations and aligning its diverse uses to effectively support decision-making and collaboration in the design process. Several systems have been developed to capture rationales, yet challenges with industry-wide adoption are observed. The more expressive a representation is, the more laborious the process or annotation required (Lee & Lai 1991). Workarounds have sought to automatically capture the decisions made (Rogers et al. 2015) and even computationally generate plausible rationales (Gruber & Russell 1996). Advances in automation have primarily been noted in software-based systems where the capture tool can be built into the program’s interface (e.g., web development or CAD) (Myers, Zumel & Garcia 2000; Sung et al. 2011). While design rationale poses challenges in its definitions, capture approaches and uses, it remains an indispensable tool for enhancing the design process. The following section contextualizes design rationale in an engineering design domain, including the current standards and challenges.

2.2. Communication and documentation practices in engineering design

Implementing effective communication and documentation best practices in engineering design firms is essential for fostering efficient collaboration, ensuring accurate information transfer, and facilitating the successful execution of projects. Engineering design involves written, oral, interpersonal, numerical and graphical communication (Shwom et al. 1999). The medium of communication influences what an engineer includes as well as the depth of information behind their design actions. In industry, each firm and team usually defines their reporting standards. Therefore, the reports across firms vary, as does when the information is captured (e.g., during vs. after). This research leverages design documentation that designers have completed at their pace and should include sufficient content such that the document stands alone. Educators base course deliverables on the oral presentations and written reports commonly found in industry.

Engineering education utilizes industry presentations and reporting standards for student deliverables. Coupled with hands-on project-based courses, students can gain practical experience working with external stakeholders, collaborating in teams and honing their technical writing skills (Mahan et al. 2000; Hirsch et al. 2001). Educators and researchers understand the importance of communication in engineering and seek to integrate these components as early as the first semester. While presentations and reports are commonly used, what to include varies, mirroring the need for more explicit guidelines in industry (Yarnoff et al. 2010). Addressing challenges related to rationale documentation, including inconsistencies across industries and its labor-intensive nature, is crucial.

In addition to the previously cited challenges with rationale documentation, additional issues arise when designing physical systems. Much design rationale research focuses on software or digital systems (Burge & Brown 2008). For example, website changes can be tracked through the interface. Similar to how CAD software can capture individual interactions (Myers et al. 2000; Sung et al. 2011). In contrast, designing physical systems might move in multiple modalities throughout the process. For example, starting with sketches, then low-fidelity prototypes, then CAD, before moving to higher-fidelity prototypes. For schools of thought that believe each decision should have an accompanying rationale, the richness of data increases exponentially, as does the time required to capture the information. Due to the laborious nature of capturing the rationale for every decision in a software system, engineers and firms end up not using the system altogether. Szykman, Sriram & Regli (2000) found that “user intervention-based design rationale capture has met only with limited success, as designers are typically very reluctant to spend time annotating their designs with rationale.” Thus, gaps in rationale occur, and consequently, successful reuse of rationale diminishes. Furthermore, gaps in rationale can also be attributed to the communication abilities of designers and engineers.

Beyond the challenges associated with tangible designs, research highlights a notable issue in engineering communication, exposing a significant gap in engineers’ proficiency in technical writing skills. Previous research on technical writing has found that complicated sentences, lower accuracy, and less organizational structure (often in student’s work) resulted in decreased effectiveness in areas that practitioners in engineering considered important (e.g., accurate and unambiguous content; fast, predictable reading; liability management; and attention to detail) (Conrad 2017). Documents tend to favor the technical aspects of the final solution but lack an explanation of the information acquired through the design process (Hertzum & Pejtersen 2000). As a result, follow-up conversations are often needed to uncover the context and missing details. For example, these missing details might arise during a presentation’s question-and-answer period or by meeting with your colleague. We believe the nuances of design decisions should appear in written documentation without requiring follow-up meetings. The following sections provide an overview of the research materials and methods.

3. Methods

This section details the research design and methods used to address our central research question: What information is contained within design rationale in written technical reports? The study combines document analysis, thematic analysis and a grounded theory approach to extract the proposed framework for representing design rationales and their contents. The choice of these methods is rooted in their ability to capture nuances and detailed insights from a diverse range of documents and allows for the emergence of patterns and themes directly from the data. This section provides a systematic account of our research process, covering key aspects such as research design, sampling strategy, data collection process, document characteristics and analyses.

3.1. Research design

This mixed-methods study used thematic analysis and a grounded theory approach to construct a framework for representing design rationales within written technical reports (Braun & Clarke 2006; Charmaz 2006). The reports (n = 28) are standalone entities assumed to encapsulate critical reasoning processes for a final design. The data analyses used qualitative methods to identify emergent concepts, while quantitative methods were used to visualize trends and relationships. As part of the verification stage, industry reports were coded until saturation (n = 5) to enhance the generalizability of our proposed framework. This research design enabled detailed analyses of design rationale representations, leveraging the advantages of document analysis and grounded theory to derive nuanced understandings in a new framework.

3.2. Sampling strategy and data collection

The sampling strategy for document selection was driven by the aim of capturing comprehensive and in-depth information about design reasoning patterns. Having previously explored reasoning factors through a controlled study and post-task surveys (Mirabito & Goucher-Lambert 2024), the survey medium did not provide the desired depth. Unpublished preliminary research probed reasoning factors further through a small-scale document analysis on course deliverables (e.g., weekly reports and presentations) followed by a subset of interviews on reasoning factors. The reasoning captured in interviews was closely tied to the interview technique (i.e., asking sufficient follow-up questions). Final reports were selected since they document the problem, solution and design process, and are the most comprehensive course deliverable.

3.2.1. Student reports

To ensure a diverse dataset, reports from graduate and undergraduate courses were collected that addressed various design challenges (see Table A1 for the complete list of reports). Colleagues from the research group, who were course instructors, provided access to the reports after course completion. Twenty-eight total reports (846 pages) were collected from three project-based design courses at a University in California. Students were not told who the researchers were. Identifiers for the technical reports were removed before downloading and analyzing the data. The institutional review board was notified and approved the data sources and methods. Overall, the sampling strategy was guided by a thoughtful consideration of available alternatives and the pursuit of depth and richness in the captured information.

3.2.2. Industry reports

Five documents containing approximately 200 pages would serve as the verification corpus. The reports were found using keywords such as “engineering design report” and “design reports,” along with similar search terms via the Engineering Village and NASA Technical Reports Server databases. Additional searches on the Aerospace Research Central, Army Corps and Ballistic Research Laboratories were also explored. Sixteen reports were initially collected and compiled into a spreadsheet, including industry, source, year and page information. The search focused on documents published in 2000 or later. Ten reports remained, ranging from nine to 260 pages. Furthermore, while NASA reports were neatly structured and linked to supplementary material, we wanted a spread of reports and project types and ranked the 10 reports in the order in which they would be coded. Due to reaching saturation after five reports, the remaining five reports were not coded. The five reports were collected without direct coordination with the respective firms, and thus, information regarding the participants’ demographics remains unknown.

3.3. Document characteristics

3.3.1. Student reports

Fourteen undergraduate technical reports were gathered from two summer design courses that lasted six weeks (two units) in 2018 and 2019. Fourteen graduate technical reports were collected from a Fall 2020 course that lasted 16 weeks (three units). The undergraduate students included a variety of disciplines (i.e., engineering and non-engineering) and mostly upper-level undergraduate students. Meanwhile, students were primarily mechanical engineering master’s students in the graduate course, although a few undergraduates were in this course. Twenty-eight design teams comprising 116 students (Table 1) were taught about and engaged with the human-centered design process, including researching, analyzing, ideating, building and communicating stages. Due to a shorter course timeline, the undergraduate reports were much shorter and less detailed (a total of 153 pages; approx. 10.9 pages/report) than the graduate reports, which also included extensive appendices (a total of 693 pages; approx. 49.5 pages/report). Please refer to Table A1 for the complete list of reports, years and pages.

Table 1. Breakdown of design team participants for the 28 student reports.

The instructors facilitated team formation in each class, resulting in design teams consisting of three to five students. Each design team could select their problem space and utilize any design methods and tools taught in the courses. The instructors and project mentors helped guide students throughout the design process. Each course had a series of learning modules and project deliverables. Project deliverables included but were not limited to design reviews, presentations, prototypes and technical reports. The final write-up for each course had instructions to include the problem space, the final prototype and the design process the team underwent. The undergraduate course deliverable had an emphasis on what and why behind design methods, while the graduate-level course included a comprehensive list of sections they could include in the report, although the list served as a guide and not a strict requirement. Neither course had page limits nor specific framework for communicating rationale was provided.

The typical elements within the summer design courses (undergraduate) contained the problem, background, solution, design process and retrospective. Rationales appeared in the design process portion and when their final prototype was introduced and explained. The typical report elements for the semester-long course (primarily graduate students) contained an executive summary, table of contents and sections split by phases: 1-Identify, 2-Understand, 3-Conceptualize and 4-Realize. The graduate reports included several subheadings for each phase. Relevant information on justifications was primarily found in phase 4 sections when the final concept/features were stated and a discussion on the feasibility and assessment of the design. Both reports contained a multitude of figures showcasing the design process (e.g., images of observations), methods (e.g., specifications table, empathy mapping) and concepts (sketches, CAD, prototypes).

3.3.2. Industry reports

Five publicly available industry design reports were used for the verification stage. Table 2 shows the report name, source, year and number of pages. Reports 1 and 2 focus on redesigning an existing system with contents including an introduction, concept history, specifications, design methods (testing and simulation) and revised design. These reports were shorter and had two or three authors. Meanwhile, reports 3–5 detail the design of a complex system, including document sections such as introduction, specifications, background information, prototypes and final technical design. These reports had between 10 editors and over 500 authors listed. These reports are from government agencies and not-for-profit organizations. The intent of the reports is to showcase the design and relevant science that enables the design to function. The audience is the scientific community who may use the system, perform maintenance, or redesign subsystems. The reports contained figures (diagrams, CAD, images of the system, data visualizations) and tables.

Table 2. Technical design reports from industry used in the verification stage.

3.4. Data analysis

Document analysis is a qualitative research approach in which documents, in this case, technical design reports, are gathered and analyzed. Researchers with experience in engineering design research and design practice reviewed and inductively coded the datasets. More specifically, thematic analysis using a constructivist grounded theory process was used for coding the reports (Charmaz 2006). Figure 3 shows a high-level overview of the qualitative coding process used after collecting and importing reports without any identifiers in MAXQDA, a software program for computer-assisted qualitative and mixed methods data, text and multimedia analysis. The systematic analysis iterated through four main coding stages: initial, focused, axial (i.e., drawing connections between codes) and theoretical (conceptualizing codes into theory) (Charmaz 2006). Explicit design rationales were identified via two approaches: top-down through the designers’ convergent design actions (e.g., “We decided,” “An app was prototyped”), and bottom-up when features were explained (“The cutting board surface serves as”). When little or no rationale was found in the top-down or bottom-up approach, information resembling implicit rationale was identified and coded (e.g., specifications, decision-matrix) even if the designers did not explicitly connect the information to a decision or feature.

Figure 3. Overview of the inductive coding process. Initial codes were raised to tentative categories, used in the focused coding stage, and became two theoretical concepts (Mirabito & Goucher-Lambert 2022). Additional theoretical sampling of industry reports was completed during the verification stage, in which the concepts were further refined via integration work resulting in the proposed FSE framing.

3.4.1. Coding procedure

The first stage of initial coding was performed by closely moving through a portion of the data. Design rationales could appear throughout the technical reports but were mainly present in concept selection, detailed descriptions of the prototype and technical feasibility. In this stage, the researcher inductively coded paragraph-by-paragraph or line-by-line for sections rich with rationale-specific information. These inductive codes use words that prioritize the actions and agency of the designers (gerund coding) and curb researcher tendencies to make conceptional leaps. For this stage, four reports from the graduate-level course (approximately 200 pages) were quickly coded, highlighting any data potentially relevant to the key research question. For example, open codes included “Stating gamification feature,” “Explaining a workaround,” or “Using interview quotes.” Once the first round of coding was complete, observations (i.e., memo-writing on tentative categories) noted possible analytical directions for emergent patterns, such as mediums of design rationale communication (i.e., text, images, tables) and the mix of design processes used to generate evidence for design rationales.

The second stage focused on using the most important codes to sift through a large amount of data (remaining graduate then undergraduate reports). This coding was more selective and conceptual than paragraph-by-paragraph, focusing on using and deciding which tentative categories most aligned with the research question. As focused coding was performed, we decided on two analytical directions that made the most sense to categorize the data and codes. One focused on the categories of communicating design rationale, highlighting the medium and linguistic styles. The other situated the categories as spatiotemporal benchmarks within the engineering design process, highlighting where designers pinpoint rationale development via persons and methods. Our analysis of these categories reached “data saturation” after 16 documents (12 graduate level and four undergraduate reports), where no new information was observed in the student dataset (Hennink & Kaiser 2022). Due to a desire for a quantitative analysis of code frequency across project types (Section 3.4.2), the remainder of the undergraduate reports were quickly coded with respect to the two analytical directions.

Afterward, the third stage (axial coding) was used to establish connections between categories and distinguish the properties and dimensions inherent within each category. Concept charting and flowcharting visualized the emergent tentative categories using Mural, an online whiteboard platform. The integration work focused on how clear and precise the communication of design rationale was and the techniques design teams used to describe and support their rationales. In the last stage, theoretical concepts were created from the tentative categories to group similar codes (i.e., created a code set of “missing information” and associated codes like “missing links” and “overemphasizing language”). The resulting code sets include “communicating clearly,” “missing information,” “inserting self,” “making assumptions,” and “redirecting focus.” The early results on communication patterns (theoretical concepts of “levels of clarity” and “three techniques”) were published in a conference paper (Mirabito & Goucher-Lambert 2022).

The contribution of this work refines those early theoretical concepts through further theoretical sampling of new documents (verification stage), resulting in the proposed FSE framework. This stage aimed to test whether communication of design rationale in an industry setting could be encapsulated by the theoretical concepts of “levels of clarity” (i.e., communication effectiveness) and “three techniques” (i.e., surprising rhetorical strategies). In the verification stage, the industry reports were deductively coded using the following five codes (“communicating clearly,” “missing information,” “inserting self,” “making assumptions” and “redirecting focus”). The first two codes are associated with “levels of clarity” while the latter three are associated with the “three techniques.” A research team member was trained in the coding schema using the student reports before coding the industry reports. After each industry report was coded, reflections and insights were used to develop a more cohesive story of the data. During this stage, the “levels of clarity” concept was verified, while the “three techniques” concept was not verified. Further analyses and integration work sought to better explain the differences across “levels of clarity” by understanding what information was contained in rationales coded as “communicating clearly” (described further in Section 4.1). Essentially, the inclusion or exclusion of “feature,” “specification,” and “evidence” was annotated, giving rise to the FSE framework. In summary, the data collection and analyses helped identify and verify the elements that more effectively communicated rationales contained.

3.4.2. Project classification comparisons

Design team deliverables from the three courses were deductively coded using the innovation type classification from Ceschin & Gaziulusoy (2016), as noted in Table 3. The follow-up analysis was performed using the project classification to determine whether trends in design rationale communication could be attributed to project types, as Rao et al. (2021) used to understand design teams’ justifications behind design method selection. This follow-up analysis required each coded segment from the initial and focused coding stages to be accounted for within the emergent code sets.

Table 3. Number of student projects listed by innovation type (Ceschin & Gaziulusoy 2016).

4. Analysis and results

The FSE framework is a culmination of research that iteratively explores the contents and meanings embedded in design rationales within technical reports. Previous work from our research group explored how engineers and designers communicate design rationale and presented two theoretical concepts: levels of clarity (i.e., communication effectiveness) and three techniques (i.e., surprising rhetorical strategies) (Mirabito & Goucher-Lambert 2022). These two theoretical concepts were based on the five codes of “communicating clearly,” “missing information,” “inserting self,” “making assumptions” and “redirecting focus.” The research objective of this work was to understand what information is captured in design rationales. First, the earlier findings of “levels of clarity” and “three techniques” concepts were deductively coded using the industry dataset. To accomplish verification, new reports were coded using the proposed code sets (Strauss & Corbin 1990). The “levels of clarity” concept was verified. Next, another probe sought to distinguish the “communicating clearly” code from “missing information.” The following results summarize the “levels of clarity” concept, the verification stage using industry reports, and code frequencies across project types.

4.1. Theoretical refinement of levels of clarity to FSE

Preliminary investigations of communication patterns of design rationale resulted in two theoretical concepts, “levels of clarity” and “three techniques.” Levels of clarity sought to position design rationales on a spectrum in which one end was complete and detailed (“communicating clearly”), while the other was incomplete and lacked depth (“missing information”). The three techniques consisted of three code sets: (1) “making assumptions” – Designers make generalizations about their users’ thoughts and actions; (2) “inserting self” – Designers insert their own experiences or values in place of their users; and (3) “redirecting focus” – Designers reference other pieces of information (e.g., figure, table, chapter). Since this article focuses on the FSE framework, example coded segments for three techniques are not discussed but are detailed in Mirabito & Goucher-Lambert (2022). Figure 4a shows a series of example codes grouped under each code set. A simplified version of one of the integration figures created in Mural that began to situate codes on a spectrum is shown in Figure 4b.

Figure 4. (a) Example codes within the two code sets of the “levels of clarity” concept; the list is not exhaustive. (b) The diagram situates the codes from more complete to less complete.

While the codes were grouped under “communicating clearly” or “missing information” code sets, there were more nuances than simply being complete or incomplete. To better understand those nuances, the codes and their corresponding segments were organized on a continuum (four segments included in Figure 5). By analyzing the information within segments, nuances were identified through the content they contained and the depth of information provided. Figure 5 shows how four coded segments were annotated as part of the analysis (feature, form or function explanation, specification, evidence, source for evidence and redirecting focus). A defining characteristic was the ability to cohesively link information within a paragraph or section. Only a subset of rationales was annotated before patterns relating to feature, specification and evidence emerged. Additional nuance noted the depth of information could serve as simply additional descriptions of the feature, linking the source (design method/tool used to generate evidence) or redirecting focus to supplemental material (diagrams, tables, appendices). “Redirecting focus” was one of the codes that appeared in both “communicating clearly” and “missing information.”

Figure 5. Coded segments with elements within them annotated. Repetitive contents include features, specifications and evidence. The depth of information provided was also noted.

4.2. Summary of levels of clarity from student reports

This section summarizes the “levels of clarity” concept (Mirabito & Goucher-Lambert 2022). The analysis revealed a hierarchy in communication effectiveness of design rationales where segments corresponding to the “communicating clearly” code set included more information and detail than those corresponding to the “missing information” code set. Post-coding integration work revealed FSE as key elements that appeared in rationales coded as “communicating clearly.” Meanwhile, “missing information” contained two or fewer FSE elements. Twenty-two of the 28 documents (79%) contained 109 rationales coded under the code set of communicating clearly, indicating that most reports effectively conveyed design rationale for at least one feature. Meanwhile, 27 of the 28 reports (96%) contained 115 rationales coded under missing information, suggesting a high frequency of fragmented or absent design rationale. Note that both codes could be used in a single report since a given product has multiple features and accompanying rationale.

The communicating clearly code set was exemplified by an example design rationale for a product that assists wheelchair users in cooking. This rationale included the product feature (the staging area), the design specification (holding at least 20 lbs.) and evidence (based on user interviews and measurements the team took by filling pots of water) supporting the decision, demonstrating a high level of effective communication. On the other hand, examples of missing information included segments where any combination of the feature, specification, or evidence elements was absent. Even segments with two out of the three elements required readers to search for missing elements in other parts of the report. The presence of ineffective or incomplete rationales was attributed to inadequate communication skills or a lack of understanding of design decisions and processes. The information included in design rationales also indicated the extent to which designers assumed specific knowledge from their readers, reflecting team congruency or a shared knowledge base.

4.3. Verification using industry reports

Across the five industry reports, 438 segments were coded within the 218 pages. For the “levels of clarity” concept, there were 86 instances of communicating clearly and 207 codes for missing information. When information was missing, it often could be attributed to using industry standards or experimental results. Regarding the “three techniques” concept, redirecting focus was most frequently used (n = 110), followed by inserting self (n = 19) and making assumptions (n = 16). The technical design reports used the redirecting focus strategy when referring to graphs, charts and other figures to justify their decisions. The “levels of clarity” concept was verified, while the “three techniques” concept was not verified in the industry reports. Instead, the three techniques helped explain the gaps in missing information. Through the post-coding integration work, features, specification and evidence were the annotated elements that helped distinguish rationales coded as “communicating clearly” from “missing information.”

Segments from Table 5 associated with the feature element are, “To shield NEXT-100 from the external flux of high-energy gamma rays a relatively simple lead castle, shown in Figure 18,” “Double-sided wiring,” and “A highly segmented electromagnetic calorimeter (ECAL).” Often, the feature is described with respect to function, behavior and structure (Gero 1990). Next-100 and Muon System both contain elements regarding requirements and specifications. In addition to describing how the wiring was done and fixed in the design, “The wiring procedure was tested for a 700 mm long detector panel, and the average pitch measured was 1.5 mm with a root mean square of 14 μm.” These reports are rich with information about the features, specifications and tests conducted (evidence) to ensure the design meets the desired specifications. Within the same report, the depth of information for rationale elements varied.

4.4. Cross-comparisons

Student and professional design reports share commonalities in documenting the design process, using visual aids, and presenting their solutions. However, they diverge regarding analysis depth, authors’ expertise and application context. Industry reports are more specialized, in-depth, and tailored to real-world applications, while student reports are typically more generalized, reflecting educational learnings and experiences. To highlight these distinctions, we deliberately chose documents from different situations to thoroughly explore the capabilities of the proposed FSE framework.

4.4.1. Comparisons across undergraduate and graduate reports

Undergraduate and graduate design reports have similarities in their structure and presentation of design processes. Both reports document the design process using natural language, visual aids and structural elements (i.e., headings and subheadings). They guide readers through the identify, understand, conceptualize and realize stages of design. Across each dataset, narratives of their actions with limited depth and detail could be found, “We chose to create a touchscreen user interface for the environmental control, created a wireframe of the concept to demonstrate different capabilities of this idea. However, after we finished creating the tangible prototype and wireframe for the music room idea, we realized that creating a final prototype for this concept would be extremely difficult given our time constraints. Therefore, we chose to move on with the Sound Off idea. We created a sensor and detection system using Arduino, LEDs, and a sound sensor (see Figure 2, below).” This report provides a linear process but is missing information (i.e., it is unclear if the final design meets the intended requirements since there is no information on product testing) and thus is not linked cohesively.

Moreover, the cross-comparisons reveal notable divergences in the depth of information provided in the design rationales and how cohesively different rationale elements were linked. Graduate students engaged in a more rigorous design process (e.g., larger interview samples, finite element analyses). Undergraduate reports had more uncertainty in their design decisions since they had less time than the graduate course – for example, C2 and M1 in Table 4 detail information generated from user interviews and product testing. C2 states “concerns about portability of the device,” which translates (i.e., linked) into design actions that make the product detachable and easily adjustable. Meanwhile, the latter states, “[Users] would prefer this product over their current sanitizing solution.” Details about the current sanitizing solution, why users would prefer the new product (i.e., evidence), and how the team captured user preferences (e.g., interviews, surveys) were not present.

Table 4. Levels of clarity concepts are based on the two code sets, communicating clearly (C) and missing information (M).

4.4.2. Comparisons across student and industry reports

The industry reports were used to verify and refine the theoretical concepts of “levels of clarity” (refined to FSE) and “three techniques.” Despite only the FSE framework being applicable, a few insights and explanations are included in this section. First, structural elements and report length appear to influence the ease of cohesion and completeness of documenting design rationales in the reports. For example, if a document details specifications first, followed by the features, the document tends to contain segments coded as “missing information” since the information is scattered throughout the report rather than cohesively contained within a few paragraphs or referenced logically. Whereas longer reports might have included in-depth details, the content was less cohesive, perhaps due to an overreliance on redirecting the reader’s focus instead of summarizing the specifications or evidence (as noted segments from the MuonSystem and Next-100 report in Table 5).

Table 5. Representative coded segments for industry reports for communicating clearly (C) and missing information (M).

Next, given the specialized nature of the industry reports, assumptions typically revolve around high technical expertise (of the reader) and domain-specific standards. For example, an industry report might list quantitative information as a specification or test result (evidence) without stating how that information is relevant and integrated into their reasoning. Student reports often made assumptions about the design or user preferences. The technique of inserting self was rarely observed in industry reports, which the complexity of the designed systems could explain. In the design course, the products are user-centered, where a student could be part of the target market for the system they are designing. In comparison, the authors of the technical reports are not the direct users of the devices (e.g., Mars vehicle, particle collider).

4.5. Communication patterns of design rationale across project types

An additional analysis of the “levels of clarity” and “three techniques” concepts was segmented by innovation project type for all 33 reports, as shown in Table 6. Product-service and spatio-social have a higher average percentage of communication clearly codes (28–31%) compared to socio-technical and product types, 17% and 23%, respectively. Concerning missing information codes, product and socio-technical innovation types were high in the range of 35% and 40% compared to spatio-social and product-service (22% and 25%). Redirecting focus (224 codes) leads as the most common technique used, followed by making assumptions (101 codes) and inserting self (81 codes). Redirecting focus occurs more frequently within the product innovation type. Redirecting focus in systems with physical components makes sense because of their physicality. However, when reviewing the segments within the redirecting focus code set, the segments often allude to other documents or processes, such as interviews or specification guides. Making assumptions and inserting self were particularly low for socio-technical reports. The analysis of project innovation types reveals noticeable differences across types, suggesting communication challenges could be attributed to project innovation type.

Table 6. Breakdown of the number of coded segments per code set and split by project innovation type.

Note: Product type contains 15 reports and 263 codes. Product-Service has seven reports and 133 codes, and Spatio-Social contains eight reports and 122 codes. Socio-Technical System contains three reports and 378 codes.

5. FSE framework

In the pursuit of enhancing communication practices of design rationale, this research focused on understanding the information contained in design rationales in written technical reports. Using an iterative and inductive approach, the FSE framework emerged from analyzing coded segments of design rationales associated with the “levels of clarity” concept. Within the resulting FSE framework, distinct elements – features, specifications and evidence – have been identified, reflecting the nuanced patterns and insights derived from the coded rationales of students and professionals. This section introduces the proposed framework, its elements, and the type of information contained within each element.

5.1. Elements of the framework

Feature (F) – Feature describes an artifact’s design component or attribute that the rationale serves to justify, such as a brake pad, steering wheel, or tire. The depth of information for this element could be as simple as describing what the feature looks like (i.e., form), while a more detailed description of the feature might include the form and functions. In general, the feature should meet a specification. The breakdown of which features to include in reporting can be best defined by the firm or industry (e.g., brake pad vs. braking system). For example, the exact bolt material might need explicit rationale; however, features are more likely to serve as a critical component of the final solution.

Specification (S) – Specification describes the stated design requirement(s) the feature aims to address, defined in the early stages of the design process, such as slowing down the vehicle within a specified time, ergonomic comfort during prolonged use, or adequate performance in a wide range of temperatures. These specifications are noted early in the process, and existing tables may be referenced; however, authors of design rationales need to be explicit about which specification they are referring to rather than cite an entire table. A feature can address more than one specification, and multiple features can address a single specification. The depth of information for this element could be as simple as compliance with safety standards. At the same time, a more detailed description of the specification might list the exact values and conditions (e.g., braking distance in meters) that the product must meet.

Evidence (E) – Evidence describes the relevant information from that design process that empowered the designer to select the final feature that meets the specification(s), such as interviews, background research or product testing. For example, testing alternative braking mechanisms or brake pad materials is considered. The evidence is the meaningful output acquired using design methods. Thus, designers might include the tests used and the results from those tests influencing their decisions. A list of a few possible evidence pieces include:

• • Interviews – The evidence should include meaningful information from the interview that was used to gain insights. For example, after interviewing 30 users on their ice coolers, 15 interviewees cited the known challenge of the closure mechanism failing and spilling the container’s contents. Five interviewees explained that they now only use ice packs instead of ice in containers, while three noted they use household tape to secure it during transit. The design team opted for a belt-like design that secures the cooler, similar to how one interviewee attached tape.

• • Background research – The evidence should include meaningful information from a specific source that was used. A website, research article, or book could be cited with a quote or paraphrase of the valuable information acquired.

• • Testing – The test should explain why it was run, along with the interpretations of the results. For example, based on a Vickers Hardness test, the three materials were tested to see which is the hardest and should be used in the design. The corresponding results should be listed.

• • Industry or firm standards – The norms should be stated, ideally linking to a page or source. For example, specific aluminum alloys that NASA has previously approved for external usage.

In summary, the three main elements of the framework are feature, specification and evidence. Further information on how to apply the framework to education and practice is shown in Section 6.2. The following analysis and results section connects the theoretical concepts with the empirical richness encapsulated in the coded segments and illustrative quotes. The intricacies of the thematic analysis are examined, and the narrative that resulted in the framework is revealed. Through the process of coding, grouping codes into code sets, identifying theoretical concepts and refining those concepts, the links between the FSE elements and the technical design reports are described.

6. Discussion

Three main contributions were extracted from a qualitative research process using design rationales captured in technical reports. First, the iterative process details the analysis and development of the concept of “levels of clarity” consisting of communicating clearly and missing information code sets that were verified and later refined into the feature, specification and evidence framework. Next, comparisons of the codes across datasets and project innovation types were presented. Lastly, the proposed FSE framework is presented, grounded in technical reports from students and industry professionals. The following sections situate the findings within the broader design rationale research community. The potential applications for FSE, remaining challenges, and next steps are mentioned before closing with a reflection on the methodological approach.

6.1. Situating FSE into broader design rationale research

Documenting design rationale is essential, but what information to include and at what level of detail is not standardized in teaching or practice. We propose a simplified representation of rationale, the FSE framework, that should be leveraged for its clear identification of relevant information and provides examples of how this information can be cohesively documented. Research on design rationale and design rationale capture systems often attempt to collect copious amounts of information, albeit at the cost of increased time and workload imposed upon practitioners (Regli et al. 2000; Bracewell & Wallace 2003; Maier, Eckert & Clarkson 2005). The FSE framework, with its three components, might initially resemble frameworks like QOC or IBIS, which have three variables. However, such representations expand indefinitely, resulting in complex graphical networks. Often, those graphical representations rely on additional software to annotate decisions and artifacts, whereas FSE relies on natural language that would integrate into existing reporting formats. Furthermore, FSE diverges from IBIS and QOC in terms of its origin. Unlike IBIS and QOC, which were conceived as normative methods outlining what should be done, FSE originated as a descriptive pattern, documenting the actual practices of engineers.

The FSE capture approach could be considered a feature-oriented approach (Regli et al. 2000) that is most similar to the Decision Cards template (Gutierrez Lopez 2018). Design Cards capture (1) the title of the decision, (2) a description of the decision in natural language (no clear structure provided), (3) team members associated with the decision and (4) supporting material such as sketches, pictures and notes. Each simplifies the information captured, erring away from increasingly complex representations and added software requirements. Additionally, we acknowledge that alternatives and tradeoffs are not always present when structuring rationale as FSE. In reports, those alternatives often appear in concept selection matrices or product testing results (e.g., plotting the performance of multiple concepts). Future iterations of FSE should explore how detailed evidence should be or how including alternatives considered could fit into the proposed framework.

The FSE framework offers a well-defined content structure for design rationale and allows for capture to take place for convergent design behaviors. This approach aligns with prevailing reporting standards that typically occur after a project by minimizing disruption to existing design practices (Conklin & Yakemovic 1991). We expect that the simplicity of the FSE framework will enhance designers’ willingness to document rationale. While systems aiming to capture every decision hold inherent value, design practitioners often fail to perceive immediate benefits and consequently opt out of documenting rationales at the level of detail demanded by such systems (Regli et al. 2000). Although case studies demonstrate the usefulness of such tools (Bracewell et al. 2009), their implementation within organizations heavily relies on top-level directives. Furthermore, even among firms that employ rationale capture tools, no universally adopted standard has emerged within industry due to the varying information requirements posed by different solutions (Sagoo et al. 2014).

We acknowledge that the proposed framework is less expressive than more intricate capture techniques. However, future research could explore ways to increase expressiveness via the evidence component of FSE. The evidence component is intentionally open-ended, allowing designers to incorporate relevant information regardless of how the evidence was generated. Nevertheless, due to the open-ended nature, practitioners could benefit from a list of possible examples to guide them. For instance, the Product Reasoning Model adapted for a home appliance case study helped designers translate insights into solutions for the following categories: experiences, interactions and technology (Knudsen & Haase 2019; Knudsen, Haase & Goncalves 2020). Future work should explore the types of information possible and strive to create a concise list of common evidence that designers can reference. Alternatively, the focus could be placed on emphasizing the most relevant pieces of evidence. Design rationale usability is paramount, and it relies on effective capture methods. The FSE framing facilitates ease of capture and recognizes potential refinements for the evidence component.

6.2. Potential applications for FSE in education and practice

Applying the FSE framework to engineering education and practice can be integrated into existing standards where designers might use the FSE categories as a guide or checklist for each component of an artifact. The framework can be used to check the completeness of design rationale documentation. Thus helping students and practitioners understand what elements are incomplete. The FSE framework identifies the feature, references specifications and links the information acquired from design processes to evaluate how the updated artifact addresses the specification.

The framework can be explained to students or practitioners using the sample instructions and example outputs in Appendices A.1 and A.2. The intended format to communicate FSE is written in a paragraph format. Another format might consider bullet points or listing in a table. The purpose is to document the rationale for the design’s most critical features, specifications and evidence. The FSE framework provides a guide for the information to include and how it would be linked together. While rationales are regularly under-documented, one must note that specification and evidence can contain multitudes of information. To avoid a time-consuming process for the design engineer and prevent information overload for the reader, designers could perhaps aim to include the top three specifications or pieces of evidence per feature. We acknowledge that limiting evidence for a feature risk omitting essential information, potentially leading to problems if future decisions rely on false assumptions. The amount of evidence provided and the level of detail of information should be investigated in future work. Moreover, improving general communication abilities within engineering design would help with the study of and documentation of design rationales.

Regarding linking features, specifications and evidence cohesively, one might consider traditional reasoning frameworks (deductive, inductive and abductive). Reasoning clarifies how evidence (credible information from valid design methods and tools) supports a claim (feature meets user requirements). Without emphasizing logical structure, designers might only include a specifications table, a list of methods used, and final design photos without articulating how the parts connect. Hernandez developed an engineering reasoning-based course in which logical reasoning was introduced to increase scrutiny of the final design and justifications (Hernandez 2018). Dong et al. (2015) showed that the logical framing structure (i.e., abductive, deductive) significantly influenced design decisions. Deductive reasoning was more likely to cause human participants to reject proposed designs, whereas abductive reasoning was more likely to cause participants to accept a design product or feature. Future work might consider studying how different logical reasoning structures affect the quality of design rationales. Educators could also consider introducing alternative representations (e.g., QOC or IBIS) or asking students to write for a specific audience (Cosgrove 1981). In industry, by incorporating standards to document design rationales, project organization improved, and contrary to expectations, designers found such a system natural and helpful in their communication processes (Bracewell & Wallace 2003). Research finds most design rationale tools useful; the challenge remains with industry-wide adoption (Shum et al. 2006). The proposed FSE framework can help designers and students structure relevant information that articulates justifications behind each component of the designed artifact.

Within engineering design, previous research has shown several advantages to introducing artificial intelligent (AI) design agents to assist human designers at various stages of the design process (Williams et al. 2019; Camburn et al. 2020a, b ; Song et al. 2022). Research from Raina, McComb & Cagan (2019) used deep learning to imitate human designers, where the system performed just as well or outperformed human designers. However, current AI systems struggle to explain the why behind their actions aside from replicating human behavior (Raina et al. 2019; Zhang et al. 2021). Although the agent may recommend an objectively higher-performing design, the consideration and acceptance of the recommendation rely on the agent’s ability to explain its rationale. Research finds that communication style and the level of information provided influence a human’s trust in the system (She, Neuhoff & Yuan 2021).

The framework (feature, specification, evidence) can help structure design rationales provided in written documents, which in turn can be used to assist design rationale extraction algorithms (Rogers et al. 2015) or generative rationale (Gruber & Russell 1996). For example, extraction can search for each element based on a large enough data set annotated as a feature, specification and evidence. A key challenge for previous extraction research is identifying rationale (Siddharth, Blessing & Luo 2022). Here, we define rationale as a justification broken into FSE. Another challenge that needs to be solved is evaluating whether the FSE elements are cohesively linked. Moreover, generative rationale could be structured into FSE with sufficient information on the depth of information each element needs. For example, careful prompting of large language models could use the FSE structure to output rationales. The ability to explicitly link information acquired through design processes to final design solutions, as defined in the FSE framework, could benefit the development of computational design agents to use human-like rationale.

6.3. Next steps to evaluate FSE

We hypothesize that using the FSE framework results in higher-quality rationale than unstructured approaches. However, quality is not well defined. Current research is developing ways to evaluate rationale quality using natural language processing and human evaluations. The two hypotheses developed from this work are: H₁: Rationales using the FSE framework are more cohesive than unstructured rationales. H₂: FSE structured rationales are rated higher than unstructured rationales using human raters and rubrics (e.g., technical writing, critical thinking). Compared to the current design rationale capture processes (i.e., technical reports), we believe that using FSE will result in higher quality rationales in comparable amounts of time. Evaluation of these hypotheses will use quantitative methods, and this research is well underway. One should note that many of the exhaustive rationale capture approaches in literature have been qualitatively evaluated with designer feedback. Future work will move beyond designer perceptions and toward methods to evaluate the quality of design rationales computationally at scale.

6.4. Reflection on methods

The study of design rationale and its components in this study raises potential challenges concerning internal and external threats to validity. Internal validity or confidence pertains to the extent to which the study accurately measures what it intends to measure. In this context, the varying depth of detail teams provide in explaining their design rationales poses a potential threat to internal validity. Some teams used complex finite element analysis as evidence to support their decision. In contrast, others offered only a high-level description without identifying relevant information, potentially affecting the consistency and precision of the study. Moreover, longer reports tended to include many supplementary materials throughout and cited external reports, meaning the relevant information for a rationale became more challenging to link together.

The measures of confidence (i.e., internal validity) and relevance (i.e., external validity) by Atkinson, Bauer & Gaskell (2000) provide a guide for assessing the research materials and methods. For transparency, each step of the coding process, codes, integration work and memo-writing were documented via MAXQDA, lab notes, and Mural. Concerning triangulation and reflexivity, students in the design courses and industry report authors were unaware of the researchers’ presence. The reports were received after the completion of the classes. Industry reports were available through library databases. In the analyses, student identifiers were removed before coding began. The authors and organizations of the industry reports are known and could be easily looked up.

External validity or relevance concerns the generalizability of the study’s findings beyond its specific context. Regarding corpus construction, the student reports were a convenience sample from one university, and the careful selection of industry reports may introduce selection bias, impacting the study’s internal and external validity. The availability of industry reports through library databases may limit the diversity of sources and potentially introduce bias. In terms of thick description, entire segments of rationales are analyzed and compared across datasets and project types in the results and discussion sections. Local surprise arises through the degree of impartial or incomplete information in design rationales, which differed between students and industry professionals by the depth and linkage to design methods and tools. In conclusion, careful attention to potential threats to internal and external validity is essential for accurately interpreting and generalizing the findings.

7. Conclusion

Engineers and designers must provide design rationale when developing new products and systems. Design rationales should go beyond describing product features and functions. Instead, rationales should cohesively link those features with specifications and evidence. Using a mixed-methods approach, 846 pages of technical design reports were analyzed using thematic analysis inspired by the grounded theory process, followed by a verification dataset of five industry reports. Based on the information contained in design rationale data, the main contribution is the proposed feature, specification and evidence (FSE) framework that details a logical structure for communicating design rationales with a low barrier to integration into existing reporting standards. Open challenges in design rationale communication are discussed, as well as how to apply the FSE framework in education and practice. Overall, the FSE framing can help structure rationales for human designers and potentially assist in automated rationale extraction or AI-generated rationales.