Exploring the impact of design tool usage on design for additive manufacturing processes and outcomes

Abstract

Improving designers’ ability to identify manufacturing constraints during design can help reduce the time and cost involved in the development of new products. Different design for additive manufacturing (DfAM) tools exist, but the design outcomes produced using such tools are often evaluated without comparison to existing tools. This study addresses the research gap by directly comparing design performance using two design support tools: a worksheet listing DfAM principles and a manufacturability analysis software tool that analyzes compliance with the same principles. In a randomized-controlled study, 49 nonexpert designers completed a design task to improve the manufacturability of a 3D-printed part using either the software tool or the worksheet tool. In this study, design outcome data (creativity and manufacturability) and design process data (task load and time taken) were measured. We identified statistically significant differences in the number of manufacturability violations in the software and worksheet groups and the creativity of the designs with novel build orientations. Results demonstrated limitations associated with lists of principles and highlighted the potential of software in promoting creativity by encouraging the exploration of alternative build orientations. This study provides support for using software to help designers, particularly nonexpert designers who rely on trial and error during design, evaluate the manufacturability of their designs more effectively, thereby promoting concurrent engineering design practices.

1. Introduction

Design tools can help engineers improve product quality, reduce development times and decrease the potential for product failure (Booker 2012). The use of design tools may be especially helpful for additive manufacturing (AM), where design and manufacturing are intertwined and should be simultaneously considered in the design for AM (DfAM) process (Thompson et al. 2016; Seepersad, Allison, & Sharpe 2017; Simpson, Williams, & Hripko 2017). During DfAM, designers design parts with the understanding that the parts will be created with AM technologies. A range of design tools, such as lists of heuristics and computer-aided design (CAD)-based software, are used in academia and industry to support DfAM, and each tool has its own unique capabilities and limitations. Given the complex interactions between tool design, choices made during the design process and design outcomes, the usage of different tools to support the same task is likely to result in varying levels of quality of the design outcome that is produced. However, the impact of using different design tool types on the DfAM process has yet to be systematically evaluated. This study explores the interplay between the design tool type and the DfAM process and the attributes of design tools that result in feasible yet creative designs.

To understand the relative effect of design tool usage, it is necessary to understand relevant design tool attributes and how these can interact with the design process itself. Prior research has established relationships between the use of different design tools (e.g., sketches, CAD and prototypes) and cognitive processes in engineering design, which in turn affects the quality of design outcomes (Vidal, Mulet, & Gómez-Senent 2004; Yang 2005; Jang & Schunn 2011; Youmans 2011; Häggman et al. 2015). Different design tools may be more compatible with different design stages or tasks. For example, prior research comparing design tools has found that the usage of physical prototypes is associated with faster idea generation and positive user perception when compared with sketching or CAD-based prototypes (Häggman et al. 2015).

Commonly used design tools to support DfAM and to help designers consider manufacturing costs and constraints include lists of design principles and CAD-based software (Pradel et al. 2018; Valjak & Lindwall 2021). Another commonly used tool, the Design for Additive Manufacturing Worksheet (Booth et al. 2017), lists design principles along with a scoring system to allow designers to evaluate a design’s suitability for AM. While these types of tools are used in industry and academic settings, there may be downsides to the use of such tools: prior research shows that many designers oversimplify designs when educated about AM constraints (Prabhu et al. 2020b; Schauer, Fillingim, & Fu 2022) and may fixate on certain features when trying to transfer from one manufacturing process to another (Abdelall, Frank, & Stone 2018b; Bracken Brennan et al. 2021; Brennan et al. 2022). This tendency toward oversimplification and fixation observed in prior research suggests that it is important to evaluate AM design outcomes not just for manufacturability but also for creativity or novelty, to ensure that design tool usage does not push designers toward simplistic changes with minimal design exploration. Some studies have shown that using cards during early design stages to present AM heuristics and design principles, presenting one manufacturability guideline at a time, can promote quality or novelty during ideation or conceptual design (Blösch-Paidosh & Shea 2019; Perez et al. 2019a), possibly counteracting this tendency toward oversimplification, but the effect of these tools has not been compared with other, similar tools.

The use of CAD tools early in the design process has been associated with design fixation in some studies (Robertson & Radcliffe 2009; Edelman & Currano 2011), where designers are hesitant to fully explore the full design space and are biased by the existing geometry defined in a CAD model. Other researchers have posited that the use of CAD can help designers identify new patterns and relationships, improve visualization of designs and support more exploration of esthetics rather than causing fixation (Jonson 2005; Chandrasegaran et al. 2013). Software-based manufacturability design tools can be used to perform tedious calculations and improve technical quality (Mehta et al. 2019) and can help minimize manufacturing process fixation (Abdelall, Frank, & Stone 2018a). There has been little experimental comparison of how the use of different design tools, such as manufacturability software or lists of manufacturability principles, affects design for the AM process and the resulting design outcomes.

This study details the results of an experiment exploring how design tool usage impacts design outcomes and processes with data from 49 engineering student participants who were asked to evaluate a simple design and offer suggestions for improving the design. Participants used one of two design tools that communicated the same set of design principles: a worksheet tool that listed the design principles and a CAD software tool that listed and analyzed compliance with the principles. This study provides a new understanding of how design tool type and associated differences in designer–tool interactions and data visualization impact the technical feasibility and creativity of resulting designs. Based on these results, we derive recommendations for the use of design tools to support the DfAM process.

2. Aims

This study aimed to directly compare process and outcome data during the DfAM process when using a software tool versus a list of DfAM principles presented on a two-page worksheet. The focus of the study is on restrictive DfAM (i.e., DfAM guidance focused on improving printability), specifically applied to fused filament fabrication (FFF), commonly referred to as fused deposition modeling (FDM). This paper details a study that used a controlled design task, with engineering student participants assigned to use one of two different types of DfAM tools: software- or worksheet-based. In the design task, participants were asked to redesign a simple pencil holder part to improve its manufacturability for AM while considering functional design constraints.

This task specifically required participants to conduct DfAM (i.e., to improve the detailed design of a product for successful production using AM) rather than design with AM (DwAM), a broader and more divergent process to help designers consider potential opportunities for innovation around AM (Perez et al. 2019b). In AM and other engineering manufacturing applications, redesigning or improving an existing design is common. Redesign challenges are commonly used for research purposes (Thomas-Seale et al. 2023). Furthermore, during detailed design, it is common for designers to use design tools to improve the manufacturability of their design, rather than starting from scratch and coming up with a new design. Using a redesign for this study is relevant and useful to the industry while simultaneously giving nonexperts a shared starting point and enough constraints to be able to more fairly compare designs. Both types of tools were designed specifically for this study and described the same manufacturability guidelines based on process limitations associated with FFF, such as minimum feature size and the need for support material. This research study explores how tool type impacts redesign performance, as measured by creativity and manufacturability of the design outcomes and participants’ subjective design process data. The hypotheses explored in this study are as follows:

• • H1: Designs produced with the support of software-based manufacturability assessment will have higher creativity and fewer manufacturing problems.

• • H2: Designers who utilize software-based manufacturability assessment will report lower subjective workload and will complete the design task in less time.

3. Significance

Computer-aided manufacturability analysis design support tools have become increasingly prevalent in industry contexts, but there is little evaluation in published research of how the use of such tools influences the design process (e.g., workload and time taken) or the quality of design outcomes. Using a rich data set composed of behavioral data and design outcome evaluation, we provide evidence that the use of such tools is associated with fewer manufacturability violations and identify an association between tool usage and highly creative design outcomes. This study provides insights to inform design practices and future design for manufacturing method development, especially in regard to helping nonexperts balance design performance and creativity. This research study highlights the importance of visualization of design or manufacturing alternatives and provides evidence for how a lack of visualization may reduce creativity by restricting the exploration of build orientations. This study also provides suggestions for teaching design for manufacturing in undergraduate engineering contexts, with guidance on the timing of the use of lists of principles and software tools, separately or in tandem. Finally, details for using the DfAM software tool and access to its freely available source code are provided.

4. Related work

4.1. Representation of information in design tools

AM design principles have been shared in worksheets, lists or card formats. While some sets of DfAM principles focus on opportunistic use of AM to improve product functionality, many also focus on restrictive printability or manufacturability guidelines, providing guidance for how to improve the quality of printed products (e.g., Booth et al. 2017; Perez 2018). Software-based DfAM design tools can provide similar support as lists of AM manufacturability principles to help communicate ways to improve manufacturability. Most existing research studies focus on the presentation of principles themselves, with few studies presenting the application of design principles (Fu, Yang, & Wood 2016). Studies that evaluate the effectiveness of different presentations of design principles can still provide insight into how different design tools might provide varying levels of support for a design task. For example, prior studies have shown that when presenting manufacturing feedback or principles, visual representations are more effective than textual feedback (Barnawal et al. 2017; Fillingim et al. 2020), which may indicate that software tools can be more helpful because they can provide a more visual presentation of information.

Improved visualization of manufacturability information may be especially important for AM because build orientation (i.e., the orientation at which a part is printed) is linked to product quality, risk of printing failure, part strength and other attributes associated with product performance (Di Angelo, Di Stefano, & Guardiani 2020; Bushra & Budinoff 2021). Studies have shown large variability in individual’s spatial visualization abilities (i.e., the ability to mentally rotate or transform 3D shapes) (Yoon 2011), which means it may be difficult for some engineers to effectively visualize a design at different build orientations. CAD-based systems could support designers by displaying a design at any desired build orientation, removing the burden of visualization from the designer. On the other hand, CAD-based systems may also lead to increased fixation (Robertson & Radcliffe 2009; Edelman & Currano 2011), as they rely on an initial CAD model to perform calculations. One of the goals of this study is to explore the interplay between tool usage and design outcome creativity and manufacturability, providing insights into the level of specificity needed in design feedback during detailed design to provide sufficient technical guidance while minimizing design fixation.

4.2. User interaction in design tools

A notable difference between the use of software-based manufacturability analysis and the use of principles and self-evaluation of part geometry is the interaction between the designer and the design tool itself. Interactions between humans and other elements of a system, known as human factors, have been found to play a crucial role in how designers use (or fail to use) design support tools such as CAD (Dillon & Sweeney 1988) and concurrent engineering tools (King & Majchrzak 1996). An important consideration in interactive systems is the delay between user input and system output. In design support tools, the delay in response time has been found to impact effectiveness (Ligetti et al. 2003). For an interactive design support tool to be effective, it needs to be able to give the designer feedback quickly. Software-based systems can use algorithms to assess compliance with principles, while worksheets or lists of principles require the user to manually evaluate their part, which could require more time and mental effort.

Real-time feedback about designs such as those provided by a software-based system could be helpful for novices to understand and retain information about the impacts of selecting different design alternatives and encourage more design exploration (Mueller 2016). Preliminary evidence and widespread industry use suggest that software is useful for improving manufacturability (Barnawal et al. 2017; Mehta et al. 2019), but no direct comparison to self-evaluation exists. Similarly, there is some evidence to suggest that software-based manufacturability analysis can reduce design fixation (Abdelall et al. 2018a), but this evidence is specific to fixation caused by familiarity with a manufacturing process. This study seeks to explore how the design process changes in terms of the subjective mental workload of the designer and the time taken to complete a design when design tool usage is varied.

4.3. Designer behavior during DfAM

Previous DfAM experiments with designers have explored topics such as the impact of lecture format (Schauer et al. 2022) and timing (Schauer et al. 2022), as well as lecture versus laboratory approach (Thomas-Seale et al. 2023) on student design results. Several studies find that a number of different methods of teaching and learning (lectures, laboratory sessions or workshops, set of design heuristics and digital learning game) can help students and engineers successfully incorporate DfAM into their design process, although the impact of lecture content varies significantly between studies (Bracken et al. 2021; Blösch-Paidosh & Shea 2022; Schauer et al. 2022; Van et al. 2022; Thomas-Seale et al. 2023). Manufacturing fixation, especially during part redesign, which is an important part of the industry design process, is shown to make it challenging for designers to use newer technologies, such as AM (Bracken Brennan et al. 2021; Brennan et al. 2023). There are some preliminary results suggesting that designers with more manufacturing experience may be better at generating designs for AM (Pearl & Meisel 2022; Prabhu et al. 2022). Many existing DfAM human subject experiment results are constrained by limited sample size (Bracken Brennan et al. 2021; Doellken et al. 2021; Schauer et al. 2022; Brennan et al. 2023), but identify potentially interesting trends that can be validated in further follow-up research with larger designer populations. Previous DfAM studies have explored various approaches to teaching DfAM and improving the design process; this study is unique in that it directly compares how using a CAD tool versus a worksheet tool impacts design decisions.

5. Method

Data were collected from students enrolled in a sophomore-level introductory design and manufacturing course at a large public university in the United States (with approval from the university’s Institutional Review Board). During the semester, all students used the worksheet and software tool as part of two different homework assignments. This paper describes the homework assignment that occurred first, eight weeks into the Fall 2018 semester. A total of 58 students gave their consent to participate in this study. In the remainder of the paper, these students are referred to as the study participants.

Assignment into experimental groups took the form of stratified randomization, aiming to assign half of the students to the software tool group and half to the worksheet tool group for the study, randomizing group assignment for the confounding effects of which sections students enrolled in. In each of the four laboratory sections, students were randomly assigned to teams of four to six students. (They were allowed to adjust the team membership, though few did.) Team assignments were made for a class project not relevant to this study, but for convenience, all members of a team were assigned to either the worksheet or software group, with the goals of ensuring a roughly equal number of total students and female students in each group and a roughly equal number of students in each laboratory section assigned to each group. The assignment to groups in teams also reduced the likelihood of participants consulting participants from the other tool group about the assigned problem.

5.1. Description of software and worksheet tools

We developed a new software tool and worksheet to explore the study hypotheses. The worksheet and software tool are described in detail (Budinoff 2019; Budinoff & McMains 2021), and the source code for the software tool is available on GitHub (Budinoff 2020), but both tools are introduced briefly here. Both tools addressed six specific manufacturability guidelines, which were chosen based on the frequency of occurrence of problems identified with student prints in the university makerspace (Table 1). These principles also agree well with crowd-sourced design principles derived from the analysis of popular 3D-printed artifacts (Perez et al. 2015). The worksheet (available in Appendix C of Budinoff 2019) was a two-page Portable Document Format (PDF) with generic pictorial examples of acceptable and potentially problematic geometry and principles for redesigning a part to avoid those problems. The flow of the worksheet was similar to that of the “Design for Additive Manufacturing Worksheet” (Booth et al. 2017) and the design guidelines used in a recent experimental DfAM study (Fillingim et al. 2020). Users read through each guideline and assess their part geometry to check for problems or guideline violations. The software tool was built as a MATrix LABoratory (MATLAB) Graphical User Interface (GUI). Users open the GUI in MATLAB and upload a stereolithography (STL) file representing the geometry they want to analyze. Geometry analysis is conducted, a process that takes about 60 seconds. The tool displays the part and highlights portions of the geometry that may be problematic or in violation of each guideline (example shown in Figure 1). The designer can click on an “Info” button for each guideline to open a new window with the same guideline descriptions as are shown in the worksheet. The displayed results are updated in near real time when the build orientation of the part is changed.

Table 1. Design principles described in both tools

Figure 1. Screenshot of the manufacturability software tool used in this study.

The key difference between the worksheet tool and the software tool was that the worksheet tool was static and not interactive, requiring designers to manually analyze their part themselves to determine how and if their geometry violated any of the described guidelines. The software tool was more interactive and responded immediately with updated results if the user changed the part’s orientation. If a designer changed the part geometry in an external CAD package, they needed to re-upload the geometry into the tool to assess whether the problems were resolved. The visualization and text used to present the manufacturability guidelines were consistent across both groups. We piloted the use of the worksheet and software with several novices before deploying them in this study and received feedback that they were easy to understand and use.

5.2. Design task

The design task was assigned as a course assignment, where students worked outside of class with no fixed time limit. Participants were asked to evaluate an angled pencil holder with an engraved logo, like that shown in Figure 2, for its manufacturability using a hobbyist FFF printer. The pencil holder is a variation of a design found on Thingiverse (Drato 2016). This object was selected because its desired functionality was familiar to students, it was feasible to print using a hobbyist FFF printer and it was not so complex that it would require hours of analysis. Other objects that have been studied in recent similar DfAM studies include a soap dish (Fillingim et al. 2020), a pencil case (Fillingim et al. 2020) and a phone stand (Prabhu et al. 2020a). This research focuses on a redesign to ensure that there were some manufacturability issues with the design that needed to be resolved and to make it possible to evaluate to what extent participants oversimplified the design, as has been observed in prior studies.

Figure 2. Study participants were asked to improve the manufacturability of a pencil holder with an engraved logo while maintaining cosmetic and functional constraints. The pencil holder is shown here with the class number.

Participants were given several design constraints (the logo must remain the same size on the front of the pencil holder and the front face and logo must be relatively smooth for esthetic reasons) and general size constraints. Prior research suggests that providing explicit constraints should result in more unique results: when a DfAM design prompt included explicit objectives and constraints, designers created solutions with greater uniqueness than designers who were given a vague prompt (Prabhu et al. 2020a).

Participants were asked to analyze the part using either the software or worksheet tool. Participants were only provided access to the tool they were assigned during the duration of the design task. Both tools were designed to be self-explanatory, with built-in instructions guiding the user using the tool, with no additional DfAM training. Participants were instructed to use the assigned tool to identify any problems that could interfere with the successful printing of the part. Then, they were asked to redesign the pencil holder with geometry changes that would mitigate the DfAM guideline violations of the original design. All participants were provided with a dimensioned engineering drawing of the part.

Participants were also asked to specify a build orientation for their redesigned part. Students submitted a visual representation of their redesign (i.e., CAD file, sketches or an engineering drawing) and a written description and justification of all the redesigns they had made. To ensure that the redesign problem was challenging and to avoid having a single obvious orientation that would eliminate all potential manufacturability issues, the part was designed such that there were several conflicting problems with the geometry. One of the problems was relatively straightforward though potentially difficult to notice: there were two very thin walls that would be prone to breaking. The other problems largely depended on orientation. The geometry featured many long, thin faces with sharp corners that could be prone to warping if printed on the build plate. These faces were not parallel, and so it was difficult to find a single orientation that simultaneously resulted in a good surface finish on all faces while also not requiring support material in internal cavities, which would be difficult to remove.

5.3. Study measurements

For H2, the goal was to understand participants’ perceptions of the design process for both the software and worksheet groups. To evaluate task load while redesigning the part, all participants were asked to rate the difficulty of the task according to the NASA Raw Task Load Index (NASA-RTLX), with indices such as mental demand and frustration (Hart & Staveland 1988). Each subscale and an unweighted average of the subscales, a presentation referred to as the raw TLX or RTLX (Hart 2006), are reported separately. Participants were also given instructions to record and report how long it took them to complete the study task. To minimize bias in the self-reported data, there was no grade associated with these reported data, a third party provided instructions for accessing the survey, and the data were not accessed until after the semester was over.

All reporting was done online after the completion of the study task. From each participant, in addition to collecting survey data on the measures described above, we collected the geometry of their redesign (drawing or CAD model) and a written explanation of all the redesigns they had made. After the completion of the study task, we provided participants with the opportunity to try the tool that they were not assigned to use (e.g., worksheet participants could try using the software too) and asked them to provide comments about both tools.

After designs were submitted, we evaluated each design and counted the number of features on each participant’s redesigned part that violated either a manufacturability guideline given in the problem statement or a DfAM guideline that was described by the worksheet and software tools, a metric we refer to as the number of manufacturability guideline violations. Many different novelty metrics have been proposed to measure novelty (Fiorineschi & Rotini 2021) and creativity (Miller et al. 2021). We adopted a version of the consensual assessment technique (CAT) to assess the creativity of participants’ designs. Specifically, we employed the frequently used taxonomy based on three ratings for product uniqueness (e.g., originality and novelty), product usefulness (e.g., utility and value) and product elegance (e.g., style and well-crafted) (Miller et al. 2021).

Raters were asked to rate each design using a scale from 1 to 6 (low to high) for the subclasses of uniqueness, usefulness and elegance. These three subclass ratings were then averaged to find the creativity score. This rating scheme was chosen because it has frequently been applied to assess engineering design artifacts and was built on significant prior research on assessing design creativity (Besemer 1998; Hennessey, Amabile, & Mueller 2011; Miller et al. 2021). Combining these three subclasses captures that while highly unique ideas may be novel, creativity tends to encompass a certain amount of quality as well (e.g., a highly novel but completely impractical and infeasible design would have high uniqueness but lower elegance and usefulness, and thus only moderate creativity, whereas a highly novel and well-scoped, practical solution would score high on all three subclasses).

Two raters scored all designs for creativity subscales and guideline violations. Both raters had degrees in mechanical engineering and significant 3D printing experience. To evaluate inter-rater reliability, Cohen’s kappa was used. The observed kappa with linear ratings across all subscales was found to be 0.63 for creativity, indicating substantial agreement, and 0.96 for the number of guideline violations, indicating almost perfect agreement (Landis & Koch 1977). The scores for each rater were averaged to find the mean rating for each category, which we report in the results section.

5.4. Statistical analysis

The Wilcoxon rank-sum test was used to determine differences in median ranks between the software and worksheet groups, as a nonparametric alternative to t-tests because not all of our data met the normality assumptions of t-tests. The effect size of the differences between groups is quantified using the absolute value of Cliff’s delta, $ d $ , a nonparametric alternative to Cohen’s $ d $ . Spearman’s rank correlation coefficient, $ \rho $ , was used to quantify correlations between study measurements. For 2x2 contingency tables with categorical variables, a two-tailed Fisher’s exact test was used to identify associations between the independent variable and dependent variables. Fisher’s exact test was chosen over a chi-square test because of our small sample size. A significance level of .05 was used throughout this study.

6. Results

6.1. Study participants

Several participants were excluded from further analysis after their redesigns were analyzed. One participant from the software group and one participant from the worksheet group did not attempt the design task. Seven participants (six from the software group and one from the worksheet group) were judged to not have consulted their assigned tool in their design process because they did not reference the provided guidelines and did not include any geometry changes related to any of the guidelines in their redesigns. The final sample consisted of 19 participants who used the software tool and 30 participants who used the worksheet tool, for a total of 49 participants. The unequal group size is because more software participants did not use their assigned tool and that group assignment took place before participants completed the design task and compliance with instructions could be assessed.

Before the study task, participants were asked to respond to the question, “What is your personal experience with additive manufacturing (colloquially, 3D printing)?” The experience level of the two groups was similar, with 68% of the software group (13 of 19) and 73% of the worksheet group (22 of 30) reporting that they had little or no 3D printing experience. Given the small number of experienced participants in both groups, we do not attempt to analyze interactions between designer experience and design outcomes in this study.

6.2. H1: Design outcomes

To evaluate H1 (designs produced with the support of software-based manufacturability assessment will have higher creativity and fewer manufacturing problems), we compared data from the software and worksheet groups for measures of design creativity, manufacturability and time to complete the design task. Median values for each variable are reported in Table 2 for both participant groups, along with the results of Wilcoxon rank-sum tests comparing the software group to the worksheet group. The effect size shows the magnitude of differences found, whereas statistical significance helps quantify how likely the findings are due to chance. Both are valuable in understanding the impact of the design tool on our study outcomes, and both quantities will be discussed in separate subsections for each study measure. The distribution of responses is shown in Figure 3 for both groups, with different colors used to represent different quartiles to compactly display differences in the distributions of data for software and worksheet participants. In summary, H1 was partially supported. Results for design outcomes of creativity (usefulness, uniqueness and elegance) and manufacturing violations will be discussed separately.

Table 2. Summary of median study measurements for student performance and perception

Note: $ W $ test statistic and $ p $ -values are from Wilcoxon rank-sum tests. Cliff’s $ d $ is a measure of effect size. Bold indicates statistical significance at the p < 0.05 level.

Figure 3. Software participants’ designs had (b) fewer guideline violations than worksheet participants. The (a) median creativity rating, (c) time taken to complete the design task and (d) RTLX scores did not have statistically significant differences between the two groups. Colors represent quartiles for (a), (c) and (d) and tertiles for (b) based on the combined groups’ data.

6.2.1. Creativity

The median creativity score for the software was 3.917, whereas the median score for the worksheet group was 3.417. The Wilcoxon rank-sum test did not identify a statistically significant difference between the creativity distributions, as summarized in Table 2 ( $ p=.254 $ ). However, more software participants had designs with high creativity compared with worksheet participants. Figure 3a shows the distribution of scores for both groups, with the percentage of participants in each quartile of all scores. Of the 19 software participants’ designs, 11 had high creativity (upper third of creativity score), compared with 8 of the 30 worksheet participants. The association between tool type and high creativity score was statistically significant ( $ p=.0385 $ from a two-tailed Fisher’s exact test).

Figure 4 shows each creativity sub-score separately. The sub-score bounds representing the bottom, middle and upper thirds are different from those used for the total creativity score because there was a larger range of values than for the total creativity score. For usefulness, the designs were primarily evaluated with respect to ease of pencil removal from the pencil holder and minimization of wasted space, especially on the bottom surface (this would make the pencil holder take up more space on a desk than necessary). As such, designs with vertical holes were given lower ratings than designs with slanted holes, since slanted holes make it easier for the user to remove pens while seated at a desk. Designs that included wasted space, such as diagonal backs when the holes were vertical, were also rated lower on usefulness due to wasted bottom surface area. For uniqueness, it was considered how many similar designs were found in the study, as well as how surprising the design was (i.e., was the design familiar or similar to generic pencil holder designs). Using a design very close to the original design did not receive high uniqueness ratings, as there were many submissions very similar to the original design. For product elegance, the main considerations were the design style and esthetic. It was mainly an instinctual reaction, although some consideration was given to wasted material in the design. High-level rating criteria were discussed and agreed upon by the two raters.

Figure 4. When each subscale is viewed separately, (a) a larger percentage of software participants had designs with high usefulness and elegance, while (b) a larger percentage of worksheet participants had high uniqueness. These differences are not statistically significant. Color categories were set based on the average of the groups’ quartiles.

As shown in Figure 4, a larger percentage of software participants had designs with high usefulness and elegance compared with worksheet participants. The distributions for uniqueness were similar, with the exception of three worksheet users whose designs had high uniqueness ( $ \ge 4.5 $ ). More software participants preserved more of the unique, slanted geometry, which was viewed by coders as functionally useful and elegant. Some participants in both groups greatly simplified the design to essentially a vertical box, which was not rated highly for elegance or usefulness.

Examples of participants’ designs and their averaged ratings are shown in Figure 5. Each design is shown in the build orientation chosen by the participant, with the vertical direction being the build direction. Some participants submitted designs that were largely indistinguishable from the original design (e.g., Figure 5c), with the only changes being to increase wall thicknesses or decrease the depth of holes. Another common design decision was to simplify the geometry by removing all slanted walls, similar to Figure 5a. A slight variation on the simplified, box-like design is shown in Figure 5f, with a box-like geometry but a change in build orientation so the logo faces upward. Designs with the highest creativity ratings tended to be similar to Figure 5b,e, where more of the original geometry was retained, but orientation was varied to eliminate manufacturability guideline violations. A few designs received moderately high uniqueness scores but low scores on other subscales, which utilized the simplified box-like design with vertical holes with additional angular features with no functional purpose that required more material than the box-like design (e.g., Figure 5d).

Figure 5. Designs (a) through (f) were submitted by different participants identified by participant IDs (e.g., P22) and assigned group (S, software; W, worksheet), with subscale scores for usefulness, uniqueness and elegance shown in brackets in that order.

To visualize commonalities between designs, we plotted each design on the axes of the creativity subscales of elegance and usefulness, with color representing uniqueness (Figure 6). There are three distinct clusters on this graph: designs with usefulness less than 4, designs with usefulness greater than four but low uniqueness and designs with high usefulness and high elegance with moderate uniqueness. Generally, designs that maintained more features of the original slanted geometry received high usefulness scores, while designs that removed the slanted access for pens or increased the overall size of the holder for no functional purpose received lower usefulness scores.

Figure 6. Design usefulness plotted against elegance with uniqueness shown as color (jitter is used to add random noise to usefulness values to prevent overlapping data from being obscured). Designs from Figure 5 are labeled by letter for reference.

Only 10 designs were scored with high usefulness ( $ >5 $ ) and high elegance ( $ >4 $ ), which are shown in the upper right of Figure 6. For all 10 designs, the build orientation was varied from the default. There was a large, statistically significant difference between the creativity score for participants who selected a non-default build orientation ( $ n=17 $ , $ MD=4.417 $ ) compared with the creativity score of participants who selected the default build orientation shown in Figure 2 ( $ n=32 $ , $ MD=3.333 $ ) using a Wilcoxon rank-sum test ( $ W=108 $ , $ p<.001 $ , $ d=1.0 $ ).

Approximately half of the software participants (9 of 19) chose an orientation other than the default orientation shown in the design task description (Figure 2), compared with about one-fourth of worksheet participants (8 of 30), but the association between tool type and choosing an orientation other than the default was not statistically significant ( $ p=.22 $ from a two-tailed Fisher’s exact test).

6.2.2. Manufacturability violations

The median number of manufacturability guideline violations for the software and worksheet groups differed ( $ p=.04 $ , refer to Table 2). Participants who used the software tool submitted designs that contained fewer manufacturability violations (Figure 3b). A smaller percentage of software participants had designs with problematic small features compared with worksheet participants (Figure 7). The most common guideline violations for both groups were related to build orientation. Although the design guidelines asked participants to prioritize the surface finish of the front face, 27 of 49 participants suggested an orientation that made the faces of the logo overhanging and required support material inside the logo.

Figure 7. A smaller percentage of (a) software participants had design guideline violations relating to orientation (i.e., warping and overhanging features) or small features compared with (b) worksheet participants.

As described in the creativity section, a common design change was to drastically simplify the design, to create a box-like body with vertical holes for pens. This design reduced the potential for stairstep error and warping compared with the original geometry, but participants often introduced new problems with their redesigns. Some participants suggested that the box-like design be printed with the logo facing up to ensure a better surface finish on the logo face. While this orientation would result in an improved logo appearance, it would require hard-to-remove support material inside the holes. Another common problem with box-like designs was thin features: many participants did not notice that reducing the thickness of the wall behind the logo would result in a thin-wall condition.

6.3. H2: Designer process

To evaluate H2 (designers who utilize software-based manufacturability assessment will report a lower subjective workload and will complete the design task in less time), we evaluated participants’ self-reported task load and the time taken on the design problem. Based on the data gathered from the software and worksheet groups, H2 was not supported, and both groups of participants faced similar difficulties as measured by task load and time taken to complete the design task. The time taken and task load are discussed separately below.

6.3.1. Time taken on design task

Participants who used the worksheet spent similar amounts of time as those who used the software (Figure 3c, Table 2). One outlier in the worksheet group who reported 600 minutes was excluded from time comparisons. The medians for the two groups were 120 minutes for the software group and 95 minutes for the worksheet group. For both groups, there was significant variation between participants in the amount of time spent on the design task, ranging from approximately 30 to 300 minutes. These time estimates include all time spent on the design task, so software students may have required more time to begin the design task as they had to open MATLAB and load the tool. However, the difference between groups in time taken was not statistically significant.

6.3.2. Task load

On average, software and worksheet participants reported similar task loads as measured by the NASA-RTLX (unweighted average of all subscales). Looking at each of the subscales separately, the subscales with the highest scores are effort, mental demand and frustration (Figure 8). In both groups, participants felt that they had to work hard to accomplish their level of performance (median effort score was 75 out of possible 100), indicating they felt it was a challenging design task. The mental demand and frustration sub-scores were slightly lower, both with medians of 60. Physical demand was low, as the only physical actions required were working with a computer or worksheet and sketching.

Figure 8. When each NASA-RTLX subscale is examined separately, a larger percentage of (a) software participants reported high frustration, while a larger percentage of (b) worksheet participants reported high mental demand, but these differences are not statistically significant. Color categories were set based on the average of the groups’ quartiles.

Although the three sub-scores with the highest loading are the same for both groups (effort, mental demand and frustration), there are some observable differences between the two participant groups. A smaller fraction of software participants reported high mental demand, but a larger fraction reported high frustration, when compared with the worksheet group (Figure 8). Software participants may have offset some of their mental demand from the design task to the software tool itself but felt more frustrated with interacting with an unfamiliar tool.

6.4. Additional observations from the design task

When both the software and worksheet participants are grouped together, there are some correlations between the studied metrics. There is a significant negative correlation between the creativity score and guideline violations ( $ \rho =-.453,p<.01 $ ), indicating that design outcomes that were more creative tended to also have fewer manufacturability problems.

Exploring build orientation seemed difficult for participants. Only about a third of participants suggested using a build orientation besides the default (17 of 49). Participants’ design descriptions often included screenshots of Cura (a 3D printing build preparation or slicing software) and SolidWorks to visualize what the part would look like at different orientations. The use of 3D software to visualize the part at different build orientations indicated that reorienting parts was difficult to do without the support of software.

The simultaneous nature of making changes to both the part geometry and build orientation seemed challenging. Some participants described making geometry and orientation decisions together, e.g., “There isn’t any other orientation which works with the way I set this up (mostly because I designed [the part] to be printed like this)” (P10, software). Other participants completed the geometry changes and then considered orientation separately. For example, P9 (worksheet) finalized geometry changes and then sketched the finalized geometry in several different orientations, crossing out orientations that were not preferred. P21 viewed orientation as purely a manufacturing decision to be considered after design, saying “for overhangs, this is usually dealt with at the printing stage.”

After the design task was over, participants were given the opportunity to try both tools and to provide comments. Participants who provided comments at this stage tended to be those who had used a 3D printer before this class ( $ p=.08 $ from a two-tailed Fisher’s exact test). One participant commented that the tools could be substituted by more hands-on experience. Participant 13 said, “Once you’ve internalized the techniques for FFF printing, the tools are essentially useless. After running a print or two, you should have enough experience to reason through the things presented by the tools without referencing them at all.”

Several participants commented that the tools were useful for different reasons. Participant 5 commented, “Good in conjunction, [the software] tool points out things, but worksheet does a better job explaining them.” Participants felt the software tool made identifying small issues easier. Participant 29 said, “The worksheet tool is full of tips that are easy to memorize for somebody who prints regularly. The [software] tool is more useful for finding small mistakes/flaws in design that could bypass a designer.” Similarly, Participant 39 said, “[The software tool is] not designed for the best user experience but it’s cool and easier than troubleshooting [without] a computer.”

7. Discussion

7.1. Study limitations

Based on analysis of the participants’ redesigns and associated descriptions, the dedication of participants varied. Some participants spent hours on their design, while others were more cursory, settling for the first combination of orientation and design changes that satisfied most of the guidelines. Differing levels of motivation are also reflected in the number of manufacturability guideline violations. The software tool would have flagged issues on participants’ redesigns, but most participants (16 of 19) did not explicitly describe re-uploading their parts into the tool. A similar finding of limited tool usage was reported by Booth et al. (2017), where few students consulted manufacturability guidelines before printing unless they were required to do so. There is variance in practicing engineers’ self-efficacy and motivation regarding the engineering design process (Carberry, Lee, & Ohland 2010), but the impact of motivation on design performance may be exacerbated in academic settings.

Another limitation is the self-reported nature of our data. Although the students were instructed not to collaborate, it is possible that some students discussed their designs, which may bias the results. Similarly, the self-reported data could result in inaccurate time estimates (e.g., participants might have under- or overreported their time due to embarrassment about spending either so much or so little time to finish the assignment). Incorporating a less biased timing mechanism in future work would be beneficial.

The supermajority of the participants were nonexperts with little prior experience with AM. This study’s findings might vary if conducted with participants who are employed as engineers in industry, rather than students. Follow-up experiments could also examine whether this study’s results are unique to students or can be replicated with more experienced designers. Prior work has found that DfAM tools are helpful even for industry professionals (Blösch-Paidosh & Shea 2022), but such findings may be more applicable to tools that support opportunistic, early design.

Another limitation was the sample size, which limited the statistical power of some tests. The study sample size was similar to or larger than most similar DfAM studies (Bracken Brennan et al. 2021; Doellken et al. 2021; Pearl & Meisel 2022; Brennan et al. 2023). The sample of participants was a convenience sample, which limited the number of participants assigned to each group. This limited statistical power impacted some of the results; for example, if the data were duplicated, the differences in creativity from the Wilcoxon rank-sum test would be statistically significant. In future work, this experiment could be replicated with more recruitment of participants and larger sample sizes.

Finally, this work focused on a redesign problem, rather than an original design problem. Redesigns represent an important part of engineering design work. While redesign is relevant for AM, where many products are transitioned to AM from other manufacturing processes, the outcomes of this study may not always generalize to a less constrained design problem. A follow-up experiment could be conducted with the same study measurements but an original design challenge.

7.2. Discussion of findings

7.2.1. H1: Design outcomes

Hypothesis 1 states that designs produced with the support of software-based manufacturability assessment will have higher creativity and fewer manufacturing problems. This hypothesis was partially supported, with significantly fewer manufacturing problems for the software group with a moderate effect size. The median creativity scores were not significantly different between the software and worksheet groups, with a similar number of participants having low creativity or average creativity designs. The lack of statistical significance may be due to our relatively small sample size, and since the effect size associated with differences in usefulness and elegance is moderate, this finding should be revisited in future work. Additionally, we identified an association between high creativity and software usage, indicating that software usage helped some portion of designers achieve higher creativity designs but did not necessarily eliminate low or average creativity designs.

The design task was created to have several conflicting objectives (i.e., ensuring low surface roughness and avoiding overhanging features), which created manufacturability problems at many of the obvious build orientations. However, the challenge of calculating manufacturability problems associated with overhanging features (e.g., calculating the angle between the normal vector of different faces and the build direction) seemed to encourage participants to prefer simple geometry and orthogonal features. This supports prior findings that students tend to simplify design geometry after DfAM training (Prabhu et al. 2020b; Schauer, Fillingim, & Fu 2022). This tendency for simplification presents a challenge: design tools focused on manufacturability should still support engineers’ creativity while promoting manufacturability-related design changes.

Based on the analysis of the submitted designs, exploring orientations seems key for creative solutions to satisfy conflicting objectives. Designs that were scored highly for creativity (especially subscales of elegance and usefulness) tended to have non-default orientations. Anecdotally, we observe (based on this design challenge and prior experience with students 3D printing for projects) that student designers tend to fixate on orientations that are functionally meaningful (i.e., the same orientation that the part is typically used in). Software DfAM tools may help novices explore more orientations and thereby promote more expert DfAM behavior: a prior experimental study focused on AM redesign suggests that experts tend to reorient more and create more novel designs (Fillingim et al. 2020).

In addition to simplified designs, many submitted designs retained most features from the original design. Although a concern with CAD tool usage is that it can cause fixation on the defined CAD geometry (Robertson & Radcliffe 2009; Edelman & Currano 2011), we did not find a significant difference in the uniqueness of the designs among worksheet and software participants – some users from both groups submitted designs that were only marginally different from the original design. In this design challenge, creativity was neither expressly encouraged nor discouraged. We found an association between high creativity and software usage, but did not find a statistically significant difference in the median ranks of all creativity scores between groups. These results could indicate that software is only helpful when applied carefully, helping promote medium-creativity to high-creativity designs, but providing less value with only cursory use. In industry, there may be design situations where large design changes are not desirable, whereas in early design, more exploration and more design changes are encouraged. A follow-up study with a more open-ended design challenge could be conducted to evaluate the effect of fixation on the initial geometry. However, based on this study’s data, it did not appear that software users were significantly more fixated on the initial geometry.

Using software tools could help designers consider a wider range of possible orientations and geometry than using design rules alone. In this study, only about a quarter of worksheet participants specified non-default build orientations. In a prior work, we found that spatial visualization skills were correlated with students’ confidence and reported difficulty in a DfAM task (Budinoff & McMains 2020). Worksheets do not appear to be as effective as software in encouraging designers to explore different orientations, even if exploring different orientations is encouraged, as it was in the worksheet used here. If a worksheet tool is preferred, it could be supplemented with easy-to-use software tools to help visualize parts in different orientations to ease the demand on designers’ spatial visualization skills and promote more creativity. Based on this study’s results and the body of research showing the vast range of individuals’ spatial visualization skills (e.g., Maeda & Yoon 2013), DfAM tools may be especially helpful when they help designers explore and visualize a wide range of build orientations.

In this design task, software use was significantly associated with high-creativity designs. This observation may be related to findings reported by Abdelall et al. (2018a), which suggest that designers fixate less on non-producible features when they interact with manufacturability analysis software. Possible explanations for the increased creativity include immediate, objective feedback, which might help motivate designers to make more changes to their geometry. Additionally, the ease of exploring and visualizing different orientations could help designers explore more of the design space. Further study on how CAD-based manufacturability tools could help ease design fixation, a frequent problem among designers during mechanical design (Ullman, Stauffer, & Dietterich 1986), is needed.

7.2.2. H2: Designer process

Hypothesis 2 states that designers who utilize software-based manufacturability assessment will report a lower subjective workload and will complete the design task in less time. This hypothesis was not supported. Participants’ overall task load and time to complete the task were equivalent.

Some nuances between the groups of participants were observed, such as RTLX subscale averages, although these differences were not statistically significant. Software participants tended to have lower mental demand, possibly because the software tool helped make each guideline visible and reduced the need for participants to assess their geometry’s compliance with guidelines at a given orientation. However, the software tool did fully resolve one of the main demands on working memory that a design problem such as this presents: resolving many conflicting objectives at once. The difficulty in balancing multiple design objectives is one reason that computational tools to support the optimization of build orientation in AM remain the focus of so much research: it is impossible to manually try every combination of geometry and build orientation (Di Angelo et al. 2020; Bushra & Budinoff 2021).

Participant comments and higher NASA-RTLX sub-scores for frustration for software participants indicate some potential usability challenges with the software interface. One participant said the worksheet was “better at explaining” than the software tool, which is an interesting comment given that the worksheet and software tool had identical guideline visualization and descriptions. This comment may indicate that the all-at-once guideline presentation of the worksheet was easier to interpret for novices, rather than the one-at-a-time guideline-checking format used in the software tool. One key limitation of manufacturability analysis systems is that they take more time to get used to than a simple list of principles. While designers can achieve better performance using CAD-based manufacturability analysis systems, they must first overcome challenges with usability and training.

7.3. Implications

The analysis of the results indicates several useful findings and highlights many areas for future research. Participants who used the software tool had fewer manufacturability guideline violations. The highlighting of problematic geometry in the software helped more participants identify or visualize the problem areas than self-evaluation alone, demonstrating the promise of the effectiveness of geometry analysis for improving designs. Based on evidence from these results and those of similar studies (Barnawal et al. 2017; Mehta et al. 2019), software can help designers evaluate the manufacturability of their designs more effectively, which has the potential to make concurrent engineering design easier. Barnawal et al. (2017) argued that novice designers would benefit especially from manufacturability analysis systems because, as summarized by Ahmed, Wallace, & Blessing (2003), they rely primarily on trial and error in their design methodology. Because the participants in this study were predominantly nonexperts, we cannot directly evaluate the effect of experience on design tool usage, but the results provide support that novices do benefit from manufacturability software.

Manufacturability analysis software is often utilized somewhat late in the design process, during detail or embodiment design. Heuristics and principles can be used throughout the design process, but most recent work has explored the benefit of heuristics early in the design process (e.g., Blösch-Paidosh & Shea 2022). Further research could examine how timing influences the relative efficacy of different design tools. The use of early CAD-based design exploration is an emerging topic (Nourimand & Olechowski 2020; Budinoff & Kramer 2022), and CAD-based manufacturability analysis systems could be used to help designers explore the feasibility of different manufacturing processes with a rough CAD model. Such systems could also encourage designers to explore drastically different geometric forms that are more compatible with one manufacturing process or another (e.g., machining versus sheet forming). This study’s results indicate that early exploration of manufacturability of a rough CAD model may promote more design exploration than the use of design rules alone. The use of lists of principles may be most appropriate very early in the design process, but they could be used in tandem with or be replaced by software-based systems as soon as rough assemblies and parts have been designed.

Another consideration in the choice of design tool is the development time and effort and ease of use of each tool. The effort involved in developing a software tool is significantly more than developing a worksheet. The tradeoffs between development time and cost must be weighed against the benefits of promoting highly creative designs and reducing manufacturability problems in design outcomes. We found that more participants from the software group did not refer to the tool during the redesign (six in the software group compared with one in the worksheet group), which may indicate the higher overhead of installing software compared with downloading a worksheet, or that nonexperts are more comfortable with using worksheets rather than computational tools. For some design challenges (e.g., designs with highly complex geometry), the benefits of improved visualization of build orientations and problematic geometry will justify the increased complexity of the software.

Based on participant comments, different participants trusted the recommendations of the system to varying degrees, with some preferring to rely on their prior experiences rather than the recommendations of the tool. This trend may indicate that some participants trusted their own judgment over design tool feedback. Future research could explore how to increase the trustworthiness of manufacturability analysis software.

Our work has several implications for DfAM education. Using a software tool could help prevent unnecessary printing of poorly designed parts, which could be beneficial for design projects and makerspaces to reduce waste and lead time and improve the outcomes of design projects. DfAM training in academic contexts is often limited to lecture-based modalities given the time and cost constraints associated with manufacturing, but interactive DfAM training has been found to be related to larger increases in DfAM self-efficacy when compared with lecture (Prabhu et al. 2021), so interactive training is desirable. Emerging research suggests that virtual training for AM technologies is as effective as in-person training (Mathur et al. 2021). We hypothesize that virtual prototyping activities, like the design task described here, could help promote self-efficacy and involvement in prototyping activities by providing students with low-stakes opportunities to gain competence and confidence in DfAM. In this work, participants’ retention of the guidelines and a comparison of long-term learning between the software and worksheet groups were not evaluated. To determine the long-term impact of tool type on student understanding of DfAM constraints, more research is needed. Long-term retention of guidelines could vary significantly between worksheet and software groups, which could have implications for their usage in educational contexts.

8. Conclusion

In this study, participants who used a software-based DfAM tool created designs with fewer manufacturability violations than participants who relied on a list of written principles. We also identified an association between high creativity and the use of the software tool, although the median ranks of creativity in the software group and worksheet group were not statistically different. The time taken to complete the design challenge and the reported task load were equivalent between groups. We also identified a statistically significant difference with a large effect size in the creativity of designs that used a non-naive build orientation, suggesting that for DfAM, the exploration of build orientation is necessary and should be supported in design tool use. The use of DfAM software seems to be valuable for visualizing problematic geometry and exploring alternative build orientations, tasks that participants seemed to struggle to do when relying on written principles alone. This study provided further support for the effectiveness of software-based manufacturability analysis tools to identify geometry associated with manufacturability problems when compared with written principles. More work is needed to explore which DfAM tool types are most effective at each stage of the design process and how to promote design tool usability and trustworthiness. The results also highlighted a novel potential benefit of software-based DfAM to increase design space exploration and support student DfAM training.