Idea generation, development and selection: a study of mechanical engineering students’ natural approaches and the impact of hybrid learning blocks

Abstract

Developing effective design solutions requires successful idea generation, development and selection. Studies have demonstrated that engineering students face challenges in these idea phases and may struggle to implement recommended practices, hindering the potential for an innovative outcome. The first part of the study investigated student practices in idea generation, development and selection through think-aloud experimental sessions and post-session interviews. Data analysis from mechanical engineering students’ sessions, with think-aloud and interview data, revealed that students focused on existing ideas, assumed requirements that constrained their divergence, limited their development of ideas and did not engage much in idea selection. Then, in the second phase of the study, we implemented a learning intervention that leveraged research-based education practices to examine student adoption of recommended practices. After engaging with the learning blocks, students generated unconventional ideas, abstained from requirement assumptions early in ideation, generated a larger quantity of ideas, developed ideas intentionally and used more rigorous idea selection methods. These outcomes demonstrated that a relatively short and targeted intervention can support students in leveraging recommended approaches to idea generation, development and selection.

1. Introduction

Numerous reports have called for engineering students to develop the ability to design innovative solutions to complex problems in our world (e.g., Duderstadt 2008). Successful solutions to these problems require engineers to successfully implement idea generation, development and selection practices. If recommended practices are not followed in these idea phases, engineers may pursue conventional ideas that are only small modifications of existing ideas (Cross 2001), and potentially great ideas are not considered. Ideally, engineers need to generate a diverse number of novel concepts in the initial stages of design to create innovative solutions (Zenios et al. 2009). These initial ideas need to be developed to have the potential to succeed; thus, engineers need to combine and iterate on these early ideas by adding new features and transforming aspects of their design ideas (Kim & Wilemon 2002). After rounds of development, ideas can be evaluated according to important criteria of the problem and context, and a narrowed collection of ideas can be further refined and evaluated until engineers arrive at their final solution (Rietzschel, Nijstad & Stroebe 2006).

While these phases of idea generation, development and selection are crucial to successful innovation, studies demonstrate numerous challenges faced by students and practitioners in their idea generation (Cross 2001; Ahmed, Wallace & Blessing 2003), idea development (Crismond & Adams 2012) and idea selection practices (Toh & Miller 2015). While some challenges within these idea phases are known, much of the idea generation literature is focused on specific elements or tools (Daly et al. 2016; Zheng & Miller 2016; Lee et al. 2018a), rather than the implementation of a collection of recommended practices. Additionally, much of the research examined students’ and practitioners’ outcomes (Shah, Kulkarni & Vargas-Hernandez 2000; Hernandez, Schmidt & Okudan 2013; Lee et al. 2020), and fewer studies investigated engineers’ thought processes during design tasks.

To fill these gaps, this study used think-aloud sessions during idea phases paired with semi-structured interviews to investigate how engineering students generated, developed and selected design ideas. The think-aloud approach allowed us to investigate students’ thought processes as they engaged in the tasks. Afterward, students completed a set of hybrid learning blocks focused on recommended practices in idea generation, development and selection. Then, they engaged in a similar series of ideation tasks, allowing us to investigate changes in ideation behaviors.

2. Related work

2.1. Idea generation approaches

Recommended practices in idea generation encourage that multiple, diverse concepts are generated and considered (Zenios et al. 2009). By creating a large number of diverse ideas, engineers are more likely to generate nonobvious solutions (Zenios et al. 2009). Additionally, diverse ideas support broader perspectives on solutions and support deeper consideration of the real problem (Dorst & Cross 2001). Diverse ideas can include unconventional ideas, and these ideas can stimulate novel solutions that have not been previously considered (Kelley & Littman 2001). Idea generation recommended practices also encourage limiting early evaluation and documenting any new idea even if it seems impractical, as a “wild” idea could be transformed into a successful solution and inspire other ideas (Kelley & Littman 2001).

Both novice and experienced engineers have been shown to struggle to implement recommended design practices. Novice engineers have difficulty generating and considering multiple ideas; novices often limit the diversity of ideas by focusing on a particular concept or variations of similar ideas, a term called fixation (Purcell & Gero 1996). In addition to fixating on a particular idea type, novices can fixate on early ideas even when they realize that these ideas have major flaws (Ball, Evans & Dennis 1994). Even expert engineers have been shown to struggle to break away from existing, well-known solutions (Linsey et al. 2010) and evaluate ideas too early (Kelley & Littman 2001).

To support engineering designers in achieving recommended practices in idea generation, the use of ideation structures and tools is recommended. For example, brainstorming “rules” provide a structure for how groups should collect ideas, by building off other suggestions and not limiting the types of ideas (Osborn 1963). Ideation tools have been shown to promote quantity, creativity, diversity and elaboration of ideas generated (Linsey, Wood & Markman 2008; Hernandez et al. 2013; Daly et al. 2016; Lee et al. 2018a, b). Examples of tools include Brainwriting (Heslin 2009), Design Heuristics (Daly et al. 2012b), IDEO cards (IDEO 2002), Morphological Analysis (Allen 1962), TRIZ (Altshuller 1997) and Wordtree Design-by-Analogy (Linsey et al. 2008). Some tools may be better suited for achieving particular goals (i.e., some tools may best limit fixation while others improve the number of ideas generated). Structures and tools are sometimes specifically meant for group ideation (i.e., Brainwriting) while others support individual ideation. Group idea generation can benefit ideation, but individual ideation is recommended before group ideation (Diehl & Stroebe 1987). While studies have examined the impacts of specific structures and tools, few studies have focused on the extent to which students aim to employ recommended practices in their approaches and which tools are used during idea generation.

2.2. Idea development approaches

Engineers employing recommended practices in idea development iterate on early ideas to improve their potential. This iteration includes elaborating on existing ideas, building new ideas inspired by existing ones and generating new types of ideas based on gaps identified within existing ideas (McMahon et al. 2016). Engineers often iterate to modify ideas to address inconsistencies or errors, improve solutions to optimize certain characteristics and integrate multiple ideas to develop new ideas (Adams & Atman 1999). Furthermore, engineers may ask for feedback from their stakeholders to inform where ideas need further iteration (Sanders & Stappers 2008).

In practice, novice engineers have been shown to limit idea development and focus on evaluating and selecting an idea for pursuit (Crismond & Adams 2012). If they do engage in some development, they focus on developing a single idea by refining the same solution and adjusting the details of that solution and thus do not consider other options (Cross 2008). Novices engage in minimal iteration of ideas as compared to experts (Atman et al. 1999) and solve design problems as a linear process that can be done only once (Crismond & Adams 2012), leaving very little room to explore beyond their initial ideas.

There are few strategies and tools for idea development discussed in design literature, as design methods emphasize idea generation and selection (Dubberly 2004; Cross 2008). One existing strategy that has been demonstrated to support idea development is Brainstorming in small groups (McMahon et al. 2016), which encourages building on initial ideas without early evaluation. Other group members can use the initial ideas to develop more complete ideas and combine features of multiple ideas. Additionally, some idea generation tools have been explored as idea development tools. For example, Design Heuristics was shown to support students in elaborating or further specifying their design ideas (Christian et al. 2012; Kramer et al. 2015; Clancy et al. 2023). Also, Design Heuristics helped students to consider additional features and transform their previous ideas to further develop their ideas. C-Sketch in a group setting has been shown to support idea development as group members add modifications to previous ideas produced by other group members (Shah et al. 2001). Although research has demonstrated the usefulness of specific tools in supporting idea development, students’ natural idea development practices have been underexplored in the research literature.

2.3. Idea selection approaches

During idea selection, engineers evaluate numerous ideas and select promising ideas (Kudrowitz & Wallace 2013). Recommended practices encourage designers to appropriately evaluate and select ideas by balancing benefits and trade-offs by articulating both the positive features as well as drawbacks (Crismond & Adams 2012). Recommended practices also suggest ideas to be selected after employing back-of-the-envelope estimated calculations to ensure that their concepts meet functional requirements (Brand 1995).

While various idea generation and development tools can help in exploring a design solution space, innovative ideas are often filtered out during idea selection (Rietzschel et al. 2006). Both novice and expert designers who select poor concepts have large costs associated with redesign, while designers who select high-quality concepts increase their likelihood of product success (Huang et al. 2013). Expert engineers often select concepts that are conventional or have shown success in the past instead of novel ones (Ford & Gioia 2000). Also, Toh & Miller (2015) found that novice engineers focused on technical feasibility and effectiveness at the cost of originality. Inherent bias against unconventional ideas exists due to the risk and uncertainties of unconventional ideas (Rubenson & Runco 1995). Although innovation is emphasized in idea generation, both novice and expert engineers often filter out innovative ideas during concept selection to minimize risk.

To support designers in idea selection, various formalized methods have been developed, including the Analytical Hierarchy Process (Marsh 1993), Pugh’s evaluation method (Pugh 1991) and Utility Theory (Pahl & Beitz 1991). These methods assign attribute values to compare the characteristics of design options to find an optimal solution. Studies have examined the usefulness and effectiveness of idea selection tools (Starkey, Gosnell & Miller 2015; Zheng & Miller 2016; Zheng, Ritter & Miller 2018). Studies compared how the Tool for Assessing Semantic Creativity (TASC) and Concept Selection Matrix (CSM) influenced students’ decision-making process during idea selection and identified that students are more likely to select ideas that are ranked highly with the CSM method (Zheng & Miller 2016; Zheng et al. 2018). Another study examined the effects of TASC and the Shah Vargas-Hernandez and Smith (SVS) method and concluded that TASC may be a means to remove biases as the group evaluated the creativity of ideas (Starkey et al. 2015). However, few studies have examined engineering students’ natural idea selection practices and their thought processes.

2.4. Development of hybrid learning blocks to support idea generation, development and selection

Teaching idea generation, development and selection that supports innovation is challenging for educators (Grasso et al. 2010). Often, developing skills in generating and pursuing innovative concepts are left to students to figure out, rather than implementing specific techniques taught in class (Dym et al. 2005). Also, instructors who desire to provide more explicit instructions have indicated challenges because of the amount of effort involved in providing the education their students require (Richards & Carlson-Skalak 1997). The curriculum is slow to change and restructuring an existing course or adding a new course to focus on specific design skills can be a long process that requires significant investment from faculty members.

To provide support for some of these challenges in design education, the Center for Socially Engaged Design developed Hybrid Learning Blocks that provide content, assessment and practice with specific design skills. The Center for Socially Engaged Design defines its design approach as a contextually rich integration of human, cultural, economic and environmental factors within the processes of designing technology. The Center has a significant focus on educating students in comprehensive skillsets required for this type of engineering problem-solving, including creativity, stakeholder engagement and contextual integration. The Hybrid Learning Blocks were an education resource developed by the Center in response to educational challenges engineering students faced in tending to the social and creative aspects of engineering work (Young et al. 2017). They were developed to be used by cocurricular student organizations as well as by instructors within their courses. Hybrid Learning Block topics include, for example, conducting design interviews, performing observations, eliciting user requirements, developing engineering specifications, generating concepts, developing concepts and selecting concepts.

The Hybrid Learning Blocks integrate asynchronous online learning with personalized coaching and feedback. The asynchronous learning blocks were built to facilitate access to learning materials without time and place constraints. Since students working on design projects can be at different stages of their work, providing asynchronous learning modules allows them to access specific modules as needed. They are also easily adapted to synchronous versions, however, their development as asynchronous resources offers flexibility to users. Each learning block leverages the same overall structure, consisting of five key elements (as shown in Figure 1): (1) Prior Knowledge Review gauges students’ preconceptions and existing knowledge on the topic; (2) Core Content provides recommended practices on a particular design topic using readings and videos; (3) Knowledge Check uses a combination of closed- and open-ended problems to evaluate students’ learning; (4) Application prompts students to apply the concepts from the learning block to a real-life scenario and receive feedback from a coach to evaluate their skills and (5) Block Reflection allows students to reflect on their learning experience and challenges students on their pre-existing ideas about the topic.

Figure 1. The hybrid learning block model.

3. Phase 1 research design

This first phase of the study investigated students’ natural idea generation, development and selection practices. We aimed to examine students’ initial ideation practices, how they refined their concepts and how they chose a final solution. The research aimed to gather information about the following research question:

How do mechanical engineering students approach idea generation, development and selection?

3.1. Recruitment

Twenty-one undergraduate mechanical engineering students were recruited for the study, which resulted in over 30 hours of think-aloud and interview data. This number of student participants is appropriate for an in-depth qualitative study (Creswell 2013) and is similar to other qualitative design studies (Kim et al. 2016; Lauff, Kotys-Schwartz & Rentschler 2018). Student participants were recruited on a rolling basis and the researchers documented consistent behaviors with 21 students, indicating saturation – defined as no additional themes emerging as additional participant data are added and further data collection may not be necessary (Creswell 2013) – had been achieved.

The student participants were recruited through targeted emails to undergraduate mechanical engineering students at a large Midwestern university. All student participants had taken at least one design-related college course where they gained experience in idea generation, development and selection. Also, many student participants had design-related internships or cocurricular design activities. Thus, they had multiple exposures to design and had the opportunity to develop strategies to employ in idea generation, development and selection. Student participants’ background information with anonymized names is included in Table 1.

Table 1. Participant demographics

3.2. Data collection

Student participants completed a design task without being provided any suggested strategies to leverage and were interviewed afterward. The design task asked students to develop solutions to a given problem statement and select a final solution at the end. Student participants completed this task using whatever approaches they wanted, in order to investigate their natural tendencies in a nonguided setting. They were asked to spend a minimum of an hour working on the design task using any resources they needed. Student participants often requested to use a computer or phone with the internet.

Student participants were asked to think aloud throughout the session as they wrote and completed the design task. The think-aloud data were recorded using a Livescribe Echo pen. The think-aloud protocol asks participants to verbalize their thought processes during a problem-solving task (van Someren, Barnard & Sandberg 1994). Think-aloud approaches capture processes and ideas in a person’s working memory rather than their long-term memory (Ericsson & Simon 1980). Working memory provides a more accurate representation of their processes as they engage in a task as compared to recalling information after completing a problem-solving event.

The problems for the design task were developed based on several criteria. Solutions to these problems needed to be product-oriented since we planned the experiment with mechanical engineering students. The problems were developed to minimize the expertise needed in a particular context to ensure that students did not need extensive knowledge to generate ideas. We modified three existing tasks used in other studies that had similar criteria (Rechkemmer et al. 2017) and we conducted two rounds of pilot tests to refine the language. After the pilot tests, the two tasks we selected were: (1) the low-skill snow transporter problem that asks students to design a personal tool for people who lack ski and snowboard experience, and (2) the one-handed opener for lidded food containers problem that asks students to develop a way for people with limited use of one upper extremity to open a lidded food container. The full problem descriptions are included in the Supplementary Material.

After the design task, the student participants were interviewed using a semi-structured interview protocol. Interviews allow for the exploration of perceptions and opinions and enable probing for more information, which helps ensure the validity of the data because it allows for clarification of responses (Hutchinson & Wilson 1992) and more complete information (Bailey 1994). The interview questions were developed through multiple iterations. Open-ended questions were constructed, following recommended practices in interview protocols (Jacob & Furgerson 2012), to understand students’ idea generation, development and selection practices, and questions were framed neutrally to avoid expressing personal opinions and leading interviewees, consistent with recommended practices (Patton 2015). Examples of questions included: How did you generate ideas to address the problem? How, if at all, did you iterate on any of your ideas? Can you tell me about how you selected your final idea? Prior to using the protocol for data collection, one pilot interview was conducted to ensure clarity. Each interview was audio-recorded for analysis.

3.3. Data analysis

The think-aloud and interview data were transcribed for analysis, and student participants’ sketched data were matched with think-aloud data. The think-aloud data were analyzed to uncover student participants’ approaches by first considering the data with a list of deductive codes for each idea phase developed based on previously documented behaviors in idea generation, development and selection such as “focused on existing solutions to the problem” for the idea generation phase and “balancing benefits/trade-offs” for the selecting ideas phase (Crismond & Adams 2012). The interview data was considered additional information to support the think-aloud data as student participants often elaborated on their processes.

After labeling behaviors according to our pre-defined set, inductive codes were added based on recurring trends in the data to form the complete set of codes to describe student participants’ behaviors for each idea phase. For example, an inductive code of “assumed additional requirements” was added to the codebook as a recurring pattern to describe the approach of adding additional requirements not stated in the problem. Table 2 includes the list of codes to describe student behaviors during Phase 1. After the codebook was finalized, a third coder independently coded the interviews and think-aloud sessions and compared codes to one of the other coders. Inter-rater reliability for behavior codes was calculated as 75%. Values greater than 70% are typically acceptable for inter-rater reliability (Osborne 2008). The coders discussed all discrepancies and reached a full agreement prior to finalizing the findings.

Table 2. Frequency of student participants’ idea generation, development and selection behaviors during Phase 1

^a Added codes through inductive coding.

4. Phase 1 results

The findings represent patterns in student participants’ idea generation, development and selection approaches throughout the study during Phase 1. We summarize these patterns across idea phases in Table 2. Many of these patterns without intervention align with existing literature on the novice designer’s challenges in idea generation, development and selection (Crismond & Adams 2012).

4.1. Phase 1 idea generation approaches

When student participants were given the freedom to approach ideation how they chose, student participants (1) assumed additional requirements that were not explicitly described in the problem statement that limited divergence in ideation, (2) immediately evaluated and/or eliminated ideas after generation, (3) looked for existing solutions to the problem, (4) did not utilize any idea generation strategies and (5) focused on the practicality of ideas as a goal.

When student participants were given a design problem, they used the stated constraints from the design problem as a guide and assumed additional requirements that were not part of the problem statement. For example, Henry was working on a one-handed opener problem and indicated an additional requirement that was not stated in the problem:

I’ll call this design requirement, container must be fixated without the use of arm.

By creating an additional requirement, Henry only came up with ideas where the container was constrained to open with one hand.

Similarly, Isaac came up with an assumed requirement not included in the problem statement. He was tasked with the low-skill snow transport problem that prompted him to design a personal transportation method on snow, but he added:

The whole thing must be able to stand on the snow and move…A device that pushes into the snow to increase resistance.

By focusing on a device that can stand on the snow, he focused on devices that emphasize on balancing the user.

Student participants immediately evaluated and eliminated possible solutions during early concept generation. For example, Isaac was working on the snow transportation problem. He initially thought of an idea to use dogs to pull a sled to transport people on snow. However, he considered the weaknesses of the idea and immediately discarded it:

I did Iditarod racing and using dogs to pull a sled, but I’m not really sure if this solution needs to be just independent, not really needing a bunch of dogs to use… Now I’m thinking of those sled dogs pulling them and now I’m just thinking of some way you can get pulled up a hill without using animals because that is a big variable, sounds expensive, and a lot of maintenance. I don’t like it.

Victoria was working on the snow transportation problem and quickly discarded a solution that may be pricey and time-consuming to learn for the user, which led her to not consider an idea in the early stage:

Snowmobiles are an existing solution but they’re pricey and may require some time to learn. We probably want something similar to a motorized bike or an electric scooter but made for the snow…

By evaluating ideas immediately after generation, student participants did not document their ideas on paper and did not consider them as a possible solution.

Student participants also relied on current, existing solutions and minimally diverged, limiting their consideration of many alternatives. Since student participants were allowed to use any resource they needed, they searched on Google for existing solutions to the problem, which led to conventional solutions. For example, Andrea who was working on the one-handed container opener said:

I Googled one hand opener to see if there [were] any off-the-shelf products that [are] out there. And I found some, and I borrowed some ideas from like current products, that [are] like online.

Similarly, Steve focused on solutions he identified from an online search:

I’m drawing an example on my sheet, but I just found a picture on Google so I’m just going off of that.

None of the student participants used any specific ideation strategies to support them in creating diverse alternatives. In the interviews, several student participants indicated that they were aware of ideation techniques but did not apply them to support ideation. For example, in an interview, Brian said:

I remember we talked about a lot of different ways that it’s possible to ideate solutions and a few of those… Remember talking about one method… the TRIZ method. And then, I remember the acronym SCAMPER. I’m not sure if I remember it correctly.

Similarly, another student participant mentioned the knowledge of idea generation tools learned from previous exposure to design but did not implement them during the task:

There’s the TRIZ method, we learned about that. Morphological chart, SCAMPER. Those are all various things that I memorized from my class. (Jeffrey)

Student participants had access to the internet and they could have leveraged online resources; student participants knew of idea generation techniques but did not apply them.

Student participants also emphasized coming up with feasible and existing solutions that meet all their requirements as a goal for idea generation. For example:

For concept generation I think it’s just come up with an idea that would hit all your objectives but also looks good, and is feasible. (Daniel)

Coming up with a feasible idea that satisfies the goals and ultimately something that can be implemented. If you come up with an idea, and it’s a great idea, but you can’t actually make that idea come into fruition, then it’s not successful. (Paul)

Student participants focused on coming up with existing and feasible solutions at the sacrifice of diversity, which can lead to conventional ideas. Recommended practices in idea generation encourage designers to come up with novel, unconventional ideas that can be used to inspire new ideas. Focusing on the feasibility of ideas early in concept generation can reduce the diversity of concepts considered.

4.2. Phase 1 idea development approach

Participants showed minimal signs of developing ideas further than their initial generation. For example, student participants indicated that he did not expand on his initial ideas to make improvements:

I didn’t really expand on them too much, or I came up with things that I thought were problematic about them but I didn’t do too much to change my design to make them better. (Brian)

I guess these were all my brainstorming thoughts, initial thoughts … This was the first phase, this was the design phase, and then the design comparison phase. (Tim)

Student participants indicated that after they generated their ideas, they moved directly to finalizing their solutions.

4.3. Phase 1 idea selection approaches

Student participants frequently did not have to engage in idea selection as they focused on one idea throughout their design session. For example, Olivia, who was working on the one-handed opener problem, indicated that she had one idea for her solution and focused on the details during the majority of her design task:

I had this idea and went straight into, “How would I design this? What material should I use?”

She focused on fine-tuning the one solution by considering the material she would use and figuring out the detailed dimensions of each component. Since she only considered one solution, she did not engage in concept selection.

Student participants who generated multiple concepts lacked structures for how they selected the most promising idea. They showed favoritism in evaluating ideas and used inconsistent criteria to compare ideas. For example, student participants focused on their favorite idea and neglected other possible solutions:

I mean, what’s going on in my head, pretty much right now, is I very much prefer my first idea with the rubber bands… I’m just going to neglect the second idea. (Daniel)

I think it was ’cause I really like this idea and I started thinking about issues with some of the other methods that could be used. (Brian)

Student participants used inconsistent evaluation criteria to compare ideas during Phase 1. For example, Henry was working on the personal transportation problem. Henry emphasized that his idea with treads would be good for recreation and quadcopter would be safer:

I really like number one, the treads and number four, the quadcopter. I think both of these have a lot of strengths and uses and I think more use for different types of things. The treads are more for recreation and the quadcopter’s more for safety. It depends on what you’re using them for, but if I have to say which one is the best solution with the design prompt in mind and saying that this is for personal use and skiing and snowboarding are given as examples. I think the treads are the best one for this.

Henry did not use consistent criteria to compare all his ideas. He considered the quadcopter as a safe design, but he did not consider the safety of the tread idea. Although he listed some benefits of his ideas, he ultimately picked his tread idea for its convenience in personal use. Henry did not use a clear structure in his concept selection.

5. Phase 2 study

After documenting student participants’ natural idea generation, development and selection practices, we hypothesized that explicit instructions on recommended practices would support students in changing their approaches in these design phases. After student participants engaged with the Hybrid Learning Blocks, we studied the impact on their approaches, reflected in this guiding research question:

How do the Hybrid Learning Blocks impact students’ idea generation, development and selection practices?

5.1. Participants

All student participants who completed the previous design task using their natural approaches were invited to complete the learning intervention, the Hybrid Learning Blocks. Ten mechanical engineering students, from Phase 1 completed them as well as a second design task and interview. Participants were compensated 200 USD for approximately 18 hours of their time.

Student participants were instructed to complete three Hybrid Learning Blocks created by the Center for Socially Engaged Design in the following sequence: “Idea Generation,” “Concept Development” and “Concept Selection” within a 3–4 week time frame (Center for Socially Engaged Design n.d.). Each learning block took approximately 5–7 hours to complete. Each block had specific learning objectives aligned with recommended practices in the particular idea phase. The “Idea Generation” block focused on applying divergent thinking to conduct idea generation and exploring the solution space using various ideation techniques. The “Concept Development” block emphasized iterating on the ideas from idea generation and drawing out novelty in design solutions. Last, the “Concept Selection” block’s learning outcomes include filtering potential solutions through objective comparisons against needs specifications and using recommended techniques to evaluate concepts.

Once the student participants completed the Hybrid Learning Blocks, they did a post-block design task. This time, they developed ideas for the problem statement that they had not completed during the Phase 1 design task. The Phase 2 study structure was identical to the Phase 1 structure, except the interview protocol included a few additional questions related to student participants’ Hybrid Learning Block experiences.

5.2. Data analysis

After student participants completed the Phase 2 design task, their behaviors were analyzed and built on the existing codebook from Table 2. Their behaviors were compared to student participant behaviors from Phase 1. Additionally, we examined the sketched and think-aloud data to measure outcomes of idea generation, development and selection for Phases 1 and 2, including the quantity of total ideas, variety of total ideas, quantity of ideas developed, number of criteria used in selection and prioritization of criteria in selection. The metrics are summarized in Table 3 and described in more detail in the following paragraphs.

Table 3. Student participants’ idea generation, development and selection outcomes and measurement methods

5.3. Quantity of total ideas generated in all idea phases

To measure quantity, we leveraged practices used in prior research (Shah et al. 2000; Linsey et al. 2005). A single product solution was defined in two different ways: (1) student participants clearly indicated an idea by having a sketch with descriptions of an idea or (2) student participants only described an idea in words but the idea covered two or more functions of the design. When student participants came up with single components of ideas using idea generation techniques, we did not count them as individual ideas. For example, a student participant used the Mind Map to come up with various different ways to power a snow transporter such as wind power, motor and solar power, and we did not count these individual components as an idea. When the same student participant used two or more components from his Mind Map to build possible solutions and sketched out the details, we counted them as ideas. Inter-rater reliability between two coders using this approach was 94%. The coders discussed all discrepancies and reached a full agreement prior to finalizing the findings. After quantifying the total ideas generated, we conducted a paired t-test with the error rate of alpha = 0.05 using the participants who completed both phases.

5.4. Quantity of ideas developed

In quantifying the ideas developed, we followed the same procedure as quantifying the total number of ideas with additional criteria. Two coders only counted ideas that were explicitly indicated by student participants as (1) combining components of previous ideas, (2) building on previous ideas and (3) developing ideas after initial generation. The inter-rater reliability was 83% and the coders discussed all discrepancies until they reached full agreement prior to finalizing the findings. Both the think-aloud and interview data were used to analyze the quantity of ideas developed. After agreement with the coders, a paired t-test with the error rate of alpha = 0.05 was used for participants who completed both phases.

5.5. Variety of ideas

The variety of ideas was measured based on two different approaches: (1) ideas were grouped based on key features of the design to capture different types of ideas generated and (2) ideas were broken down into various functions or “bins” to analyze different subfunctions of ideas that student participants’ considered.

Each idea was classified by solution type based on the key features of the design, similar to approaches used in other studies measuring variety (Daly et al. 2012a; Jablokow et al. 2015). For example, in the one-handed opener problem, all concepts that focused on using a handheld tool to pry open the container were classified as a type of solution that occurred several times among many participants. A coding scheme was created that consisted of exclusive categories differentiating “obvious” ideas from unexpected ideas. For the low-skill snow transporter problem, eight different codes were created (1-ATV, 2-snowmobile, 3-snowboard, 4-snowshoes, 5-ski, 6-scooter, 7-motorcycle, 8-other). For the one-handed opener problem, five different codes were created (1-base/lid restraint, 2-machine (twist), 3-handheld tool (puncture), 4-handheld tool (pry), 5-other). For both problems, the “other” category represented combinations of features from multiple categories and ideas that did not fit into the above categories. For example, in the low-skill snow transporter problem, any flying objects such as drones were placed in the “other” category. Using two coders, inter-rater reliability was 78%. The coders discussed all discrepancies and reached a full agreement prior to finalizing the findings.

For the second way of measuring the variety of ideas, we created categories based on various functions or “bins” of ideas (Linsey et al. 2010). For example, in the low-skilled snow transporter problem, participants came up with a variety of ways to power their transporter including solar energy, wind energy, battery, magnetic force, and so forth. Each method of powering the snow transporter would be considered a bin. Based on all the bins, we counted how many bins were considered unique, meaning they were used by a limited number of student participants in this study. We counted bins that were only used by 1, 2 or 3 student participants. Then we compared how many of those unique bins were used by student participants during the Phase 1 and Phase 2 design tasks. Using two coders, inter-rater reliability was 71%. The coders discussed all discrepancies and reached full agreement prior to finalizing the findings. After coding, we used a paired t-test with only the participants who completed both phases with the error rate of alpha = 0.05 to conduct statistical analysis.

5.6. Number of criteria considered and prioritization of criteria in idea selection

Two coders counted the number of criteria that student participants considered during their idea selection and created a binary system: (1) an approach that used only one criterion in selecting ideas and (2) an approach that considered multiple criteria. Also, we examined if student participants prioritized their evaluation criteria in selecting ideas. Student participants who prioritized their criteria either ranked criteria or assigned different weighing values to each criterion to indicate their importance. We did not evaluate the specific idea selected by student participants because the focus of the work was to characterize students’ idea generation, development and selection processes.

6. Phase 2 results

There was a notable change in approaches leveraged by student participants in the Phase 2 design task compared to the Phase 1 approaches. We summarize these patterns in Table 4 and discuss approach and outcome patterns for each idea phase in the following subsections.

Table 4. Student participants’ behavior shifts from Phase 1 to Phase 2

^a Added codes through inductive coding.

6.1. Phase 2 idea generation approaches

After completion of the Hybrid Learning Blocks, student participants minimized assuming requirements in the early phase of idea generation, emphasized and generated unconventional solutions, utilized one or two idea generation tools and focused on increasing the quantity of ideas. The combination of these approaches led to student participants coming up with a larger quantity of diverse ideas.

While many student participants in Phase 1 generated assumed requirements, student participants in Phase 2 emphasized the importance of not limiting ideas early in idea generation and did not show signs of coming up with assumed requirements. For example:

[Idea generation] is coming up with solutions and sort of taking a question and using it to inspire solutions and not limiting your solutions. It’s like an initial dump of all of your ideas, just to get those all out there. (Brian)

Similar to the Phase 1 design task, student participants started idea generation with existing ideas, however, they then intentionally looked for unconventional ideas to help them diverge in idea generation. After coming up with several existing ideas, Cathy, who worked on the one-handed container opener problem, looked for unconventional ways to open a jar for her seventh idea:

What is the coolest way you could open a jar? Well, my go-to answer for that is to smash it, and I’m not supposed to limit myself during idea generation, something tells me that smashing it isn’t a good idea. Maybe if it was a controlled smash. Is there a way to control [it]… can you puncture a jar without getting stuff in your food… Now, we are going to just cut the top off.

Unlike Phase 1, student participants utilized at least one idea generation technique, including Design Heuristics, Mind Mapping, Morphological Matrix and SCAMPER. Using idea generation tools often helped student participants to approach idea generation in a structured way. For example, Brian was working on the snow transporter problem and used a Mind Map to generate ideas. His Mind Map incorporated central nodes that described the characteristics of his design such as power, snow movement, control direction and braking. Then he created components for each central node. For example, he thought of different ways to power a snow transporter such as wind power, solar power, turbine, jet snow propulsion, and so forth, as shown in Figure 2a.

Figure 2. (a) An example Mind Map used to generate ideas and (b) ideas synthesized from combining ideas in the Mind Map.

After coming up with various different functions within Mind Map, Brian combined multiple functions to create ideas. As seen in Figure 2b, one idea used wind power and a smooth surface to create a snow sail. The user can control direction by turning the sail and brake by moving the sail away from the wind. Another idea used jet propulsion and a smooth surface to create a snowmobile with a jet engine. By combining various functions from ideation techniques, student participants generated a number of different ideas.

In Phase 2, student participants articulated that coming up with a large quantity is important in idea generation. They focused on generating a lot of ideas that may be wild and unconventional, which is considered a recommended practice in idea generation. For example:

It’s coming out with a large quantity of ideas, no matter how ridiculous. (Henry)

Student participants aimed to diverge to generate a large quantity of ideas and also gave themselves a target number of ideas to generate. In Phase 2, student participants articulated a clear quantity goal. For example:

Let’s say I want at least 10 ideas before I move onto the next phase. (Ethan)

By setting a clear quantity goal in idea generation, student participants generated a large number of ideas to ensure that they consider multiple ideas before evaluating them.

6.2. Phase 2 idea development approaches

In Phase 2, student participants intentionally developed ideas and they separated idea generation and development as two distinct phases in design. For example:

Let’s breakdown the process beforehand. And divide it in terms of the blocks. Idea generation. Concept development. Concept selection. Okay, so we’re on idea generation. (Ethan)

By articulating idea development as a phase, student participants set aside time to build on their previous ideas. Participants said that after coming up with initial ideas, they used idea development strategies such as Design Heuristics to help them build on their initial ideas. For example:

Once I feel like I was slowing down, I think I started switching over to development and that’s when I used the design heuristic cards and shuffled them. That’s when I came up with, I think, 11 to 22 [concepts]. (Isaac)

By having explicit idea generation and development phases, participants built on their initial ideas to ultimately have a larger quantity of ideas.

6.3. Phase 2 idea selection approaches

In the Phase 2 design task, student participants systematically organized their ideas into groups and used idea selection methods such as a decision matrix to select their final idea. After generating a large quantity of ideas, participants grouped their ideas based on similarities. For many student participants, the initial grouping of ideas helped them discard similar ideas before using a decision matrix. Afterward, student participants listed important criteria or requirements for their ideas. Then, student participants often assigned weighing values for each criterion with minimal justification. For example:

I think, well, the number one is probably going to be like ease of use and I’m going to weigh that as a solid five. And then I’m going to say cost because I feel like they’re going to buy a lot of them. It’s also important. That’s a four. And then let’s just like feasibility. Then that also a four and then …we’ll just say what else is important. Storage ability is important. So, ease of use for the twist and pry. I think that one will get a solid … I think that one gets a one because it is … I mean, it’s automatic but you still have to get the jar and all that to line up and that might take a little bit of difficulty. (Henry)

Henry came up with weighing values based on what was believed to be important.

After coming up with criteria and weighing values, student participants attempted to be objective in evaluating ideas based on how well each idea meets the criteria. Student participants compared ideas and depending on the comparison, they assigned appropriate values. For example:

I think I’ll … have an objective voice. I’m just not arbitrarily picking something to do. I can just go through and say, “This is why I did it that way.” (Isaac)

Overall, in the Phase 2 design task, student participants attempted to limit their biases by using idea selection methods but their process was influenced by their perception of what criteria were important in their final idea. While they showed improvement in using a structured approach, student participants struggled with coming up with fair evaluation criteria.

7. Comparing outcomes for Phase 1 and Phase 2

In addition to behaviors that shifted from Phase 1 and Phase 2, we also analyzed differences in outcomes, which are summarized in Table 5.

Table 5. Concept generation, development and selection outcomes

7.1. Idea generation outcomes

There were some notable differences in the Phase 1 and 2 quantity of ideas. Student participants generated an average of 5.3 (SE 0.66) and 14.3 ideas (SE 1.47) in Phase 1 and 2, respectively, with a p-value of 0.00006 (Figure 3).

Figure 3. The average quantity of ideas generated in the design task during Phases 1 and 2.

With regards to differences in variety, for the first variety metric we applied – idea type – student participants generated fewer expected concept types in Phase 2 as compared to Phase 1. For the one-hand container opener problem, 70% of ideas in Phase 1 and 26% of ideas in Phase 2 involved either the base or lid being restrained before opening the container, representing the most obvious idea. The “other” category represents ideas that were not similar to other concepts in the pool of all concepts, making it a unique category itself. Thus, ideas in the “other” category did not fit into the other categories, comprising 13% in Phase 1 versus 63% in Phase 2 for the one-hand container opener problem. These comparisons are represented in Figure 4a. In the snow transporter problem, 38% and 50% of the ideas were categorized as “other” in Phases 1 and 2, respectively (represented in Figure 4b). Figure 4 demonstrates a visual representation of different types of ideas generated by the student participants to showcase the dominant and nondominant ideas considered.

Figure 4. (a) Percent of ideas grouped by solution types for Phases 1 and 2 for the one-handed container opener problem and (b) percent of ideas grouped by solution types for Phases 1 and 2 for the snow transporter problem.

We saw a similar trend of more unconventional concepts generated for Phase 2 when using the second variety metric we applied – the frequency of unusual features. By sorting ideas into bins of similar ideas, we found that on average, 0.7 (SE 0.27) and 1.2 (SE 0.46) bins were occupied by only one student participant in Phases 1 and 2, respectively, with a p-value of 0.45. This indicates that on average, student participants generated less than one (0.7) unique feature during Phase 1 while student participants came up with more than one (1.2) unique subfeature in Phase 2. On average, 1.2 (SE 0.25) and 2.5 (SE 0.5) bins were occupied by two or fewer student participants in Phase 1 and 2, respectively, with a p-value of 0.08. On average, 1.6 (SE 0.31) and 3.2 (SE 0.5) bins were occupied by three or fewer students in Phases 1 and 2, respectively, with a p-value of 0.03 (Figure 5); in other words, student participants in Phase 1 came up with on 1.6 features unique to three or fewer student participants while student participants in Phase 2 generated on average 3.2 subfeatures unique to three or fewer student participants. This analysis shows that students came up with more unique features of ideas in Phase 2.

Figure 5. Number of bins occupied by 1, 1–2 or 1–3 student participants.

7.2. Idea development outcomes

There were differences in the number of ideas developed. We counted concepts to be developed if student participants (1) combined components of previous ideas, (2) built on previous ideas and (3) came back to initial ideas to further develop them. In general, student participants in Phase 1 did not develop their initial ideas. Student participants developed an average of 0.7 (SE 0.2) and 4.6 concepts (SE 0.9) for Phases 1 and 2, respectively, with a p-value of 0.0006. These results are represented in Figure 6.

Figure 6. Quantity of developed ideas.

7.3. Idea selection outcomes

During Phase 1, student participants focused on a single idea throughout the task or showed minimal evaluation to select a single idea. For example, David was working on the one-hand container opener problem. He selected an idea based on practicality without considering or prioritizing other criteria:

Alright, so my winner is the first idea, because I think that would probably actually work. Granted a person is, you know, strong enough to open a jar. That’s the big constraint here. (David)

After going through the learning blocks, student participants considered multiple, consistent criteria and compared their ideas before choosing their final one. Also, student participants prioritized multiple criteria by ranking criteria or providing a weighing value for each criterion. Student participants heavily relied on using a decision matrix to help them compare ideas and placed numerical values depending on their perceived quality of each idea. In the end, student participants added up all the values and became focused on picking the idea with the highest rating:

All right. To total these up, taking the sum of the product of the weight and the scores, idea number one gets five, ten, 12 points. Number two gets … nine, 11. Then number three gets 14. So objectively here, number three is the winner. (Cathy)

Although Cathy’s three ideas came out to be similar in the total value (12, 11 and 14 units) in her decision matrix, Cathy picked idea 3, which had 14 units without further questioning or reasoning her choice. Although student participants balanced benefits and trade-offs, they became fixated on picking the solution with the highest quantitative value, which may be a limiting behavior that does not align with recommended practices in idea selection.

8. Discussion

Across all three phases of idea generation, development and selection, mechanical engineering students demonstrated novice approaches in Phase 1, representing how they would naturally approach these tasks. After completing the Hybrid Learning Blocks, we saw substantial differences in their outcomes.

During Phase 1 idea generation, students limited the alternatives they considered, created additional assumed requirements and relied on existing solutions. These findings build on previous research documenting challenges designers encounter in generating a large quantity of ideas (Cross 2001) that deviate from existing solutions (Linsey et al. 2010). Using their natural approaches, students did not leverage any idea generation strategies to support them in coming up with alternatives. Relying on existing solutions and limited use of idea generation strategies directed students to focus on variations of similar concepts, aligning with previous findings from other researchers (Jansson & Smith 1991; Purcell & Gero 1996). After completing the Hybrid Learning Blocks, students adopted some of the recommended practices in idea generation. Students adopted some of the ideation techniques documented in the literature such as Mind Map, Morphological Analysis and Design Heuristics to help them generate ideas. Additionally, students articulated goals in idea generation such as generating a large quantity of ideas, setting a target number of ideas to consider and minimizing early evaluation. By equipping students with idea generation techniques and teaching them recommended practices, students generated a greater number of ideas and came up with varying types of ideas as well, which are considered recommended practices (Brophy 2001; Zenios et al. 2009). Our results mirror previous studies demonstrating the benefits of systematically applying idea generation techniques to support the quantity and quality of ideas created (White, Wood & Jensen 2012; Daly et al. 2016; Lee et al. 2018a). Furthermore, we demonstrated that providing a comprehensive learning intervention supported students in leveraging the recommended practices in idea generation.

In students’ natural idea development, students placed minimal emphasis on developing their initial concepts and showed minimal improvements from their previous ideas. Students approached ideation as a linear path with little to no iteration, similar to findings from other research (Crismond & Adams 2012). After completing the learning blocks, students intentionally built on their initial ideas to further develop their ideas. Students separated idea generation and development into two distinct phases; thus, students intentionally spent time building on previous ideas and combining different features of multiple ideas to create new ideas. While students made improvements, the literature describes that experts iterate often and go through the idea generation and development phases multiple times (Brophy 2001; Crismond & Adams 2012). Although students were intentional in setting aside time to develop ideas, students did not engage in multiple cycles of development. Research has demonstrated that experienced designers engage in multiple iterations and look for new perspectives to build on previous ideas (Gerber 2008; Crismond & Adams 2012).

As students naturally selected ideas, they did not have to engage in comparing different ideas as they focused on developing a single concept throughout their task. Students who considered multiple concepts used intuition and picked a favorite idea. Students showed signs of fixation throughout the design task, similar to previous studies (Jansson & Smith 1991; Purcell & Gero 1996) and our study demonstrated that signs of fixation persisted through idea selection. In Phase 2, students used a decision matrix and sought more objective evaluation, which has been a common method to support idea selection demonstrated in the literature (Pugh 1991). Students in Phase 2 articulated important criteria and balanced benefits and trade-offs in selecting their idea, which are important characteristics described in the literature (Crismond & Adams 2012). However, they often arbitrarily assigned numerical values to design criteria and picked the idea with the highest rating at the end, which may be a limitation of using a decision matrix. In addition to balancing benefits and trade-offs, research has demonstrated that experts use analytical methods (McKenna, Linsenmeier & Glucksberg 2008), back-of-the-envelope calculations (Linder & Flowers 2001) and prototyping in selecting ideas (Lauff et al. 2018). Overall, we observed shifts in idea selection behavior from arbitrarily picking an idea to evaluating multiple criteria before selection.

8.1. Limitations

This study examined students from a single large institution in the U.S., and findings in other institutions may differ. Further, our study was limited by the relative lack of diversity across our participants with regards to gender and race and ethnicity. A more diverse group of participants may have revealed additional approaches to idea generation, development and selection compared to what we observed in our data. The study was designed to gain an in-depth understanding of students’ idea generation, development and selection practices. Instead of claiming generalizability, qualitative studies emphasize the transferability of the results, allowing the reader to make connections between this study and their situation (Creswell 2013; Patton 2015).

In this study, the design tasks created an artificial environment for students to engage in idea generation, development and selection to capture specific behaviors within these phases instead of examining a holistic design process. More specifically, students were asked to work individually and complete the task in one sitting. In practice, engineers often work on design tasks for longer periods and they often have opportunities to work in teams and engage with stakeholders to gain feedback throughout their tasks. Also, we did not measure students’ prior exposure and understanding of these topics before engaging in the study. Thus, some students may have been exposed to the learning content for the first time while others may have received a refresher on the content they have learned in the past. Additionally, providing compensation may have exerted influence in motivating students to participate and complete the study as large incentives can induce greater participation.

8.2. Implications

Our findings articulate students’ natural idea generation, development and selection behaviors when they are given the freedom to approach design tasks. Previous studies examined the effects of specific tools within idea generation, development and selection approaches (Hernandez et al. 2013; Starkey et al. 2015; Zheng & Miller 2016; Lee et al. 2018a; Zheng et al. 2018) and students’ natural behaviors have been underexplored. Design educators should benefit from understanding students’ behaviors that may limit them in achieving success at these design phases. By being aware of these shortcomings, educators can plan to provide explicit instructions on how to approach idea generation, development and selection.

Although student participants had multiple design experiences through both classes and cocurricular activities, students had not adopted recommended practices in idea phases. This lack of adopting recommended practices indicates that providing instructions can facilitate an uptake of appropriate strategies to support idea generation, development and selection. The Hybrid Learning Blocks through the Center for Socially Engaged Design are a tool to provide support as students engage in design, and can be readily implemented in design courses to support design instruction (csed.engin.umich.edu/online-learning). The on-demand option to learn design skills in any order may be particularly supportive for students since students in design projects will need to develop specific skills when they need them.

Overall, the Hybrid Learning Blocks showed evidence in supporting students to adopt evidenced-based design practices in lessons that last 5–7 hours. In addition to this study that supported idea generation, development and selection, another study has demonstrated the benefits of the Hybrid Learning Blocks in aiding interview practices to engage stakeholders (Young et al. 2017). The Hybrid Learning Blocks will continue to evolve to include recommended practices in idea generation, development and selection based on new research findings. By providing regular updates, we can ensure that students are learning and adopting the most up-to-date practices.

A flexible learning model that breaks down learning objectives into different phases of a design process can support design education. For students, open-ended design experiences through cocurricular activities and internships are not sufficient to teach recommended practices in idea generation, development and selection because these activities may lack explicit instruction. Many design activities emphasize achieving success at the end and may lack scaffolding to support designers through each phase of a design process. Thus, there is value in articulating clear goals within each phase and emphasizing reflection to ensure that designers are meeting their goals during each phase of their design. Breaking down design phases to provide support can help engineering designers to achieve success in their overall design projects; as demonstrated in the literature, implementing recommended practices is particularly important in the front-end, which includes problem definition and idea generation, because the front-end activities set the trajectory for the rest of design (Pahl & Beitz 1991; Brophy 2001).

In teaching design courses, instructors can leverage a flexible learning model to present materials to students when they need them. In a typical one- or two-semester design course, students work on large projects and move through their projects at a different pace, making it challenging to provide relevant material at the right time for all students. By leveraging flexible learning modules that emphasize different phases of a design process, students can learn and implement relevant design practices when they need them. Additionally, since an asynchronous learning intervention has been demonstrated to support engineering students, similar learning interventions may be used to aid early practitioners to adopt recommended practices in different design phases.

9. Conclusions

This study examined students’ natural approaches to idea generation, development and selection that they had developed through their prior design experiences. These approaches demonstrated novice behaviors as students created assumed requirements that limited their divergent thinking, relied on existing solutions, generated a few ideas and did not engage in idea selection. After completing the Hybrid Learning Blocks, students adopted some of the evidence-based design practices documented in the literature. After the intervention, students minimized early evaluation, generated unconventional ideas and focused on generating a large quantity of ideas. Also, students set aside time to iterate, combine and build on existing ideas. Afterward, students used idea selection methods to balance the benefits and trade-offs of ideas before finalizing their idea. This study demonstrated that providing concrete lessons using the Hybrid Learning Blocks can support students to develop clear approaches and goals in each phase of idea generation, development and selection. By supporting students’ design practices, we can equip students to develop innovative solutions to solve complex, open-ended design problems.