A social science mixed-methods approach to stimulating and measuring creativity in the design classroom

Abstract

This study aimed to evaluate the effectiveness of using a social science mixed-methods approach to stimulate creativity and improve the attainment of creative outcomes in teaching design. In a focused study framed through a design collaborative experiment within a classroom context, sketches produced by a sample of 351 students were analyzed and the impact of stimuli was categorized by visual, physical, quantitative and contextual information on creative processes and outcomes in product design. Sixteen combinations of these stimuli were integrated as parameters of design briefs (DBs) given to the participating students. This research was augmented with a survey to understand participants’ perceptions and reactions and was rated by expert judges. The results demonstrate that certain combinations of quantitative and qualitative stimuli have a positive impact on creative processes and outcomes. These findings will inform new techniques for engaging and inspiring students in design studies.

1. Introduction

Defining and exploring “creativity” is a complex and daunting task. It manifests differently across disciplines: as a process, a personal trait or a product. Creativity is not confined to a single field or practice. It is complex in its multilayered manifestation affecting individual performance, groups and product outcomes. There is thus a great incentive to innovate and use mixed methods to stimulate or simulate and measure the creative process. In social science disciplines, mixed methods are defined by studies mixing various qualitative and quantitative research techniques (Johnson, Onwuegbuzie, & Turner 2007; Onwuegbuzie & Collins 2007). To explore the range of potential methods to motivate creativity, we studied the effects of various stimuli on participatory design among students. This exploration was specifically anchored in observing the effects of selected control stimuli on creativity, assessed through a social science mixed-methods approach including quantitative and qualitative measures of creativity. These stimuli were delivered in the form of design briefs (DBs) within a participatory design session. We tested the impact on creative ideation by varying the stimuli in DBs, gathered participant perceptions and ascertained product outcomes in the form of creativity scores assigned by expert judges. The participants of this controlled study were all students undergoing an introductory course in design. The classroom context is a thematically fitting space for the study, as it embodies the study’s significance.

The aims of this study were to test stimuli that potentially impact creative processes and outcomes and assess the effectiveness of mixed approaches in measuring and stimulating creativity using the framework of the “design brief,” within the context of design education. The results of the study are discussed, and the implications for design education are detailed. From this research, knowledge of a cohesive approach tackling creativity as a tangible concept is shared involving social science methods.

2. Theoretical background

There is plenty of literature, which proposes the requirement of creativity to foster innovation in educational and professional settings (Amabile 1996; Csikszentmihalyi 1996; Craft, Jeffrey, & Leibling 2001; McIntyre 2011). Research shows that definitions of creativity have evolved in three distinct phases: notably the first as a personality construct, the second as a mental process and the latest as a sociocultural system (Sawyer 2011). Runco & Jaeger (2012) trace the evolution of definitions of creativity, with some conceptualizations of creativity dating as far back as 1839. Within Ma’s (2009) investigation of the creative components of problem definition and solution generation, he defines numerous aspects of creativity: as the creative side of an individual (person), as a problem-solving tool (process) and as an outcome (product). According to Sternberg & Lubart (1999), creativity is measured via the creation of work that is novel and appropriate.

A creative product is defined as “anything that produces ‘effective surprise’ in the observer, in addition to a ‘shock of recognition’ that the product of response, although novel, is entirely appropriate” (Amabile 1983, pp. 358–359; Bruner 1962). Similarly, other studies discern that creativity can be devised in novelty and usability (Sarkar & Chakrabarti 2007). Dean et al. (2006) examined 90 empirical studies on creativity and narrowed down the definition of “creativity” to four main constructs or dimensions: novelty, workability, relevance and thoroughness. In their research, Runco & Jaeger (2012) state that there is a wide-ranging consensus that originality and usefulness are essential components of creativity. From an engineering perspective, creativity entails producing original and possibly practicable ideas to solve a problem (Bourgeois-Bougrine et al. 2017). While there are many differing perspectives on defining the nebulous concept of creativity, it is evident that creativity is a process and a multifaceted phenomenon, which is not focused on just one product.

Ma’s research primarily focuses on creativity within the design domain (2019). His work identifies five stages involving an exploration of the creative process within the realm of problem-solving, offering a structured framework to effectively approach challenges and generate innovative solutions. The first is looking at problem definition; the second, seeking retrieval of problem-related knowledge; the third, looking at solution generation; the fourth, identifying criteria for solution evaluation; and the final, selecting a solution for problem resolution and implementing it (Figure 1). In the context of this analysis, the understanding of the creative process undergoes a significant transformation. It is reconceptualized as a “reorganization of knowledge,” a concept articulated by Ma (2009, p. 31).

Figure 1. Ma’s stages of the problem-solving process.

The interest in such creativity-driven processes in design, especially within educational institutions, is amplified due to the challenges designers face during idea generation. These challenges, highlighted by Genco, Hölttä‐Otto, & Seepersad (2012), include dealing with constraints that intermittently restrict their creative process. To address these challenges, research has delved into the use of design heuristics in concept generation, particularly among mechanical engineers and industrial designers. The results, as demonstrated by Yilmaz et al. (2015), highlight that design experts employ heuristics, showcasing their positive impact on fostering creativity within distinct design disciplines. Moreover, the pivotal role of collaborative interactions among designers in group design settings has gained prominence (Sauder & Jin 2016). These interactions have been shown to be instrumental in enhancing creativity-related thinking processes. By sharing design elements and posing questions, creativity is stimulated, underscoring the influence of collaboration on the creative endeavors of designers.

In seeking solutions, some categorize the inherently social and collaborative aspects of creativity research (Littleton, Rojas-Drummond, & Miell 2008). One such manifestation of this is the practice of brainstorming. While brainstorming usually demands that people openly talk about novel solutions to a problem, this method is ineffective for developing design solutions (White, Wood, & Jensen 2012). Classic brainstorming has been shown to lack the desired collaborative effect, as groups do not produce better solutions in the brainstorming environment than individuals working alone (Mullen, Johnson, & Salas 1991). One resolution of this issue is the utilization of stimuli to remove mental blocks while facilitating the creation of new ideas (Howard, Dekoninck, & Culley 2010).

Much research has gone into understanding the dearth of research on external and intrapersonal factors on observable creativity in the field of social psychology (Amabile 1996; Amabile and Pillemer 2012; Hennessey 2003). Means of quantifying and measuring creativity have been formulated as a result. One such “gold standard” of creativity assessment is the Consensual Assessment Technique (CAT) (Carson 2006), though it is limited to research settings (Baer & McKool 2009). Overall, all this has advanced the understanding of creativity, as well as causes and methods of stimulation or obstacles for creative outcomes such as competition, diversity of partners and technological collaboration (Nieto & Santamaría 2007).

During the initial stages of the design process, often considered the most creative phase, the utilization of diverse stimuli can be exceptionally advantageous (Liikkanen & Perttula 2008). Indeed, the approach adopted by student designers when addressing design challenges and generating creative solutions may vary based on how tasks are exemplified, the structure of DBs and exposure to informational stimuli (Sarkar & Chakrabarti 2008). Furthermore, it is worth noting that the formulation and delivery of DBs, as well as the combination of relevant information to address specific design tasks, wield significant influence over the creative outcomes of design solutions (Koronis, Casakin, & Silva 2021; Koronis et al. 2022). In this context, the incorporation of analogical sources within DBs, as a means of imparting inspiring information, emerges as a promising avenue for enhancing creative design outcomes (Casakin, Koronis, & Silva 2022; Koronis, Casakin, & Silva 2023). Furthermore, it is evident that collaborative elements of creativity are important and can yield innovative outcomes (see Koronis et al. 2019).

Similarly, another example of one such significant factor is the environment. A classroom climate is considered by some as a favorable environment for creativity and stimulation, especially if it presents high competition with low friction (Ma 2009). The meta-analysis found that creativity scores were higher when pupils were allowed to manipulate materials, engage in discussions, participate in self-evaluation and learn via self-initiative (Lopez, Esquivel, & Houtz 1993; Ma 2009). Existing conventions and formats applied in structuring information should be evaluated, such as “DBs” within design contexts.

In the context of education, educators who aim to nurture students’ creativity are particularly interested in crafting DBs that encourage creative project outcomes across various undergraduate levels, as highlighted by Sadowska & Laffy (2017) and Raghunath, Koronis, & Silva (2020). Therefore, the investigation of various conditions in which the DB and external stimuli, such as physical objects, may influence the creativity of students’ design outputs within a studio environment becomes a key area of focus.

When employing DBs to provide this essential information, several critical factors should be considered like the level of detail included in written instructions and the potential inclusion of visual prompts. Striking the right balance between clarity and ambiguity becomes paramount to achieving the desired outcomes. While some level of ambiguity can foster innovation and creativity, it is essential to avoid an excessive lack of clarity that might lead to confusion and frustration among students, as cautioned by Sawyer (2017). Ultimately, such frustration can hinder the generation of truly creative and impactful results.

2.1. The design brief

In a design project, DBs are typically presented at the beginning of the process (Camburn et al. 2017) to set expectations and the outcome, thereby setting the pitch for the project and guiding its process (Koronis et al. 2019). The DB is an essential agreement between the creator and the receiver – often a business or client – in accordance with the scope and purpose of the project, as well as the parameters of the product that is to be created. A DB is also known as an innovation brief, a creative brief, a job ticket, a marketing brief or a project brief, in other disciplines (Phillips 2004). A well-written DB, says Phillips (2004), is a road map of sorts, or a contractual agreement between interested parties, and may be seen as a business plan, as well as a project tracking tool, attached with various guidelines, specifications or parameters.

Across different fields, research has found that DBs help to focus conceptual thinking (Fleer 2000), spur on questioning (Fleer 2000) and can be a medium of public engagement (Payne et al. 2015). This results in an emotional relationship being created between the designer and the end user (McDonagh-Philp & Lebbon 2000). Much research has explored the use of DBs as important to the design process (Phillips 2004; Lawson 2006; Cross 2007; Goldschmidt & Rodgers 2013; Haug 2015). Prior research from Dorst & Cross (2001) found that stimulating the DB by providing varying information may spur creative concepts. Some studies revealed specific preferences, such as designers preferring visual representations as stimuli when pursuing inspiration (Vasconcelos et al. 2017).

Visual stimuli attached to the DB, including artwork, industrial artifacts and natural elements, frequently serve as sources of inspiration by sparking the generation of ideas. Consequently, these visual representations can expand the range of known solutions (Gonçalves, Cardoso, & Badke-Schaub 2016) and expedite the creative problem-solving process, leading to time and effort savings (Cai, Yi-Luen Do, & Zimring 2010).

3. Aims and contributions

In the pedagogy of design education, helping students ascertain and develop ideas that will generate creative solutions is an area of importance for tertiary institutions, particularly in design disciplines. Creativity in the form of designed outcomes can be stimulated when designers are exposed to stimuli attached to a DB.

We believe that a participatory and collaborative design process can stimulate new creative design solutions. This proposition stems from the notion that creativity can be cultivated and that effectively achieving design solutions can be accomplished by thoughtfully incorporating a diverse range of stimuli into DBs. We thus empirically manipulated the structure of the DB. This manipulation involved providing student designers with stimuli in the form of visual and textual displays, along with detailed written instructions for executing the design task. As such, the first hypothesis in our study was as follows:

H1: By introducing various stimuli in design briefs, we anticipate observing differences in creativity levels among students who receive different and distinct sets of instructions. Consequently, our experiment aims to establish statistically significant differences when comparing groups of students who received different design briefs.

The second area of focus was whether the DB would effectively stimulate students. Prior literature suggests that the briefing process is important to the design outcomes (Nutt 1993); however, the exact measures of what constitutes a creativity-stimulating DB are insufficiently covered in the existing literature. As such, in this study, we intended to gain a comprehensive understanding of how participants perceived DBs and triangulate this with how students performed with the briefs they received. Consequently, the second set of hypotheses was as follows:

H2a: Students receiving only the baseline design brief will find it unclear (too vague) to help them conceptualize product ideas.

H2b: Students receiving the design brief with one or more stimuli or parameters will find it more suitable to help them conceptualize product ideas.

A comprehensive approach to assessing creativity is necessary (Feldhusen & Goh 1995; Cropley 2000; Plucker & Makel 2010; Park, Chun, & Lee 2016). A study (Park et al. 2016) discussed the merits of using a triangulation of subjective and objective assessment methods for creativity research. They define subjective evaluation methods as qualitative, which involve analyzing data from direct fieldwork observations, surveys, interviews and written documents. Furthermore, they define objective evaluation methods as comprising quantitative analysis that concerns systematic empirical research through quantitative approaches. Triangulation has been generally defined as a permutation of methodologies to study the same or similar phenomenon (Denzin 1978; Jick 1979). The proposition of the above studies inspired our study – as we aimed to employ a nuanced amalgamation of creativity assessment metrics – to achieve a triangulated understanding of creativity stimulation and measurement.

4. Methods

Social science mixed methods with qualitative and quantitative measures were used to stimulate and measure creativity, as exhibited in Figure 2. We adopted a concurrent triangulated approach to answering a research question (Creswell & Clark 2018). In a similar vein, we collected and analyzed various sets of data and then merged all the results into an overall interpretation. The purpose of this study was to try and ascertain the DB and the most efficient stimulus or parameters that interact most effectively to create the highest creativity scores.

Figure 2. An overview of the model developed for the stimulation and measurement of creativity.

Within this context, we purposefully integrated a modified 6-3-5/Collaborative Sketch (C-Sketch) methodology into our research. It is worth clarifying that the use of the modified collaborative method was not intended as a manipulable experimental variable. Instead, it functioned as the conduit through which we captured the design output generated by each team, thereby serving as our designated data collection approach.

In the first component of our study, students were shown the DB and stimuli and explained what was expected of them. The second component of our study was the combined 6-3-5/C-Sketch techniques that were aimed at encouraging concept generation (see Otto & Wood 2001). Students participating in the study were introduced to the C-Sketch ideation method as it is effective in the idea generation process, especially in providing feedback on proposed ideas (Shah et al. 2001). The 6-3-5 brainwriting method is frequently employed and has been praised for reducing individual dominance in idea generation (King & Sivaloganathan 1999). Developed by Bernd Rohrbach, the 6-3-5 method is a brainstorming method, using written words (Rohrbach 1969), while the C-Sketch (Shah 1993; Shah et al. 2001) uses sketching as the primary form of communication. In trying to stimulate creativity without verbal aids, C-Sketch was also a useful tool to teach our students to generate ideas using drawings as the main procedure of communication, with additions by peers serving as nonverbal modes of idea generation or refinement (see Shah et al. 2001). This combined 6-3-5/C-Sketch methodology does not provide contextual information such as participant perception or familiarity. To address this limitation, we paired the 6-3-5/C-Sketch method with a post-sketch survey. This information helped us contextualize the sketches and gather insight into the DB perceptions.

The sketches were presented to the judges who marked them based on a rubric, which aimed to score creativity on the aforementioned measures: novelty, appropriateness and usability, which were decided to be critical factors for measuring creativity.

Furthermore, the assessment was conducted with the use of the aforementioned CAT, by Amabile (1996), which measures creativity via constructs of novelty, appropriateness and usability. CAT was used as it is a tool used in multiple studies for assessing creativity based on the evaluations of professionals and academic experts (Amabile 1979; Stubbs & Amabile 1979; Berglas, Amabile, & Handel 1981; Stubbs 1981; Amabile 1982; Hennessey 1982; Amabile, Goldfarb, & Brackfield 1990) and is also validated by Cheng (2015). This technique is also validated in various other contexts and disciplines, and it was developed initially for researching artistic creativity in children (Amabile 1996), in pedagogical studies (Baer 1993; Baer 2003), in gender and ethnicity studies (Kaufman, Baer, & Gentile 2004) and in comparing domain-specific and domain-general models of creativity (Hennessey 1994). CAT is not connected to any specific theory of creativity, and its validity does not change with any construct of creativity (Baer & McKool 2009).

We also used the metrics of novelty and appropriateness as described by Kampylis & Valtanen (2010) with the addition of usability. We postulate that the incorporation of usability overlaps with workability (“An idea is workable/feasible if it can be easily implemented and does not violate known constraints”) and relevance in the way (Dean et al. 2006, p. 650) described it. An idea is relevant if it “applies to the stated problem and will be effective at solving the problem” (Dean et al. 2006, p. 663). Usability was thus included as a third metric, defined in the international standard, ISO 9241-11, as “the extent to which specified users can use a product to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO 2018).

4.1. Participants

Students in this experiment followed an “introduction to design” course, mandatory for all freshmen (first-year undergraduate students). The course introduces essential skills and mindset of innovation, entrepreneurship and design methodologies, including teamwork and workflow organization and various computational techniques. Participants ranged between 18 and 21 years old and were in classes representing a mix of genders, nationalities, educational backgrounds and academic scores. Each group of five to six people was formed freely, and most participants remained in groups they had formed earlier in the semester. A total of 351 students participated in this study, producing a total of 1,003 sketches. Overall, 74 workgroups were created. The participants were told that the completed sketches would not impact their academic standing or performance in any manner.

4.2. Experimental procedure

A full-factorial design was employed via the design of experiments (DOEs) described by Jiju (2003) to lay out the varying conditions. DBs were presented to the students either in succinct form or along with one of four stimuli (quantitative, visual, contextual or physical) to re-design an orange squeezer. In our experimental context, “stimulus” pertains to the input parameters contained within the DB. All participants received design instructions, which covered the basic principles of the project.

The baseline (BL) DB requested students to “design a device to extract orange juice from fresh oranges at home” and detailed that it should be low cost, easy to manufacture, machine washable and have a small footprint. The only other information given in the BL brief was not to use a blender or blender-type machine as a base for the design. No quantifiable requirements were given in the BL, such as the exact cost or the size. Detailed examples of each DB can be found in Supplementary Appendix A for reference.

The first stimulus was quantitative requirements (Q) (quantitative here refers to the type of information in the DB), which provided participants with specifications for the product. These included a target cost of $20 (which the research team decided was reasonable), a limit of less than two manufacturing processes, machine washability, whereby no damage should be seen after 100 washes, and its volume quantified by 15 × 15 × 15 cm. The second stimulus was a visual example (V), which was delivered in the form of a video demonstrating the use of an existing orange squeezer. The third stimulus was a physical example (P), whereby participants received an actual orange squeezer to interact with during the ideation session. Orange squeezers were picked as they are commonly used in product design classes (see Koronis et al. 2019). The last stimulus provided was contextual (C), which presented the participants with details regarding the pain points users had with existing orange squeezers and reasons for using an orange squeezer in the setting. Sixteen cohorts were created for the varied DBs, including one cohort that received only the BL brief. The remaining 15 cohorts were given DBs paired with one or several stimuli, as illustrated in Figure 3. Groups exposed to different stimuli were each conducted in separate classrooms, and the experiment was carried out in distinct sessions.

Figure 3. Cohort (Co.) breakdown by design brief and stimuli.

4.3. The 6-3-5/C-sketch method

As described earlier, our research paired the 6-3-5 with the C-Sketch method, such that the format of 6-3-5 remained. Our experiment is in line with the study of Gonçalves et al. (2016), but with the use of sketching as the primary form of communication, following previous research work by Linsey et al. (2005) where ideas are exemplified in the form of sketches and text.

Accordingly, groups were tasked with creating three sketches. Students were asked to ponder about this problem for a few minutes, as per the retrieval of problem-related knowledge. Next, students embarked on the sketch exercise, which sought to generate solutions. Students then pass their C-Sketches around the table for their peers to respond and improve, thus evaluating the solution. The first round takes 15 minutes, and participants then pass their sketches to their adjacent team members for 10 minutes before passing them on again, all the way around the group until they return to the original creator in a total of five rounds.

Each group member uses an assigned color pen, and when the creator receives their sketch back at the end of the exercise, they can assess what was added to their sketch and in what order. The notion is that teammates may draw inspiration from one another and add creatively, in esthetics or functions, to their peers’ sketches (Goldschmidt 1991; Gu et al. 2018).

Our study applied this method so that students were first exposed to the DB allocated to their class and then asked to participate in the 6-3-5/C-Sketch method. They were tasked with creating three sketches before passing them clockwise through the rest of their group until they received their original sketch back. Group sizes averaged five people; the 6-3-5/C-Sketch method is illustrated in Figure 4.

Figure 4. Illustration of the 6-3-5/C-sketch.

4.4. Judges’ rating

The assessment was conducted by expert judges who were not told the study’s hypotheses and experimental conditions, inspecting creativity as a product based on the sketches submitted by students. Judges were selected based on their level of experience and background. The judging panel included faculty members and researchers specializing in either engineering product development or architecture and having at least 10 years of proficiency in architecture, design education and product development. Thus, they qualified to be part of an expert panel of judges (Kaufman & Kaufman 2007).

The sketches were evaluated in accordance with Amabile’s (1996) well-established CAT (see Supplementary Appendix B) to quantitatively assess and derive creativity scores for each sketch. Sketches were all scanned and anonymized before being circulated to respective judges. The sketches were grouped by cohort to ensure grading for each cohort aligned with the specific DB received. Judges were instructed to go through the sketches and assign a numerical criterion ranging from 1 to 5 for each of the three components (namely novelty, appropriateness and usability).

The novelty was measured in consideration of how the design was different (or unfamiliar) from the usual way of extracting the juice and quantified along a Likert scale where 1 indicated the sketch to be entirely like an existing method and 5 indicated a real surprise. Appropriateness was measured in consideration of the extent to which the design aligned to the brief’s requirements, where 1 indicated the sketch not to be aligned and 5 indicated complete alignment. Lastly, usability was measured as the ability of the design to extract efficiently and effectively the most amount of juice with nominal effort and discomfort, where 1 denoted a poor fit and 5 denoted an excellent fit.

Three judges were selected for the assessment of the sketches to ensure inter-rater reliability. These judges independently assessed the sketches and graded each sketch for novelty, appropriateness and usability. The scores were checked for inter-rater reliability. Any set of scores with a standard deviation that indicated the presence of an outlier was carefully examined for errors in data entry or re-evaluated by all judges to achieve consensus.

While assessing sketches, it is noteworthy that a few instances revealed a notable standard deviation among the scores provided by the judges. However, it is imperative to emphasize that the overarching reliability values remained within the “good-to-fair” spectrum. Despite affording judges an ample amount of time for the meticulous evaluation of the design outcomes, it is plausible that certain variations in scoring arose due to factors such as fatigue or the potential for human error. Indeed, the task of evaluating numerous drawings while considering multiple metrics and design requisites can, at times, introduce inherent challenges that may affect the precision of judgment.

4.5. Post-sketch survey

A post-sketch survey was administered (see Supplementary Appendix C) to ascertain participants’ genders, perceptions of their received DBs and familiarity with orange squeezers. The survey was administered to all participants after they completed the 6-3-5/C-Sketch method to gather feedback on the different briefs received across various cohorts. The questions provided had optional answers tied to scales ranging from 1 to 3 (Supplementary Appendix C). Only those who completed the post-survey were included in the analysis, resulting in a sample of 278. Only those who willingly completed the survey, allowing their scores to be matched with the design outcomes, were included in the analysis. Participants who chose not to participate or left survey items incomplete were not considered in the analysis.

To generate a consolidated score, we computed the average of the individual contributions made by each student. This resulted in each student’s dataset being consolidated to yield a mean score for their sketches, as opposed to the prior practice of having three separate scores per creativity metric, corresponding to each of the three sketches. As a robustness check, we compared the statistical analysis of the sketches (N = 278) to the statistical results of the individual contributions (N = 1003). We found that the results were very comparable. With this aggregation, though, the data are no longer normally distributed (p < 0.05 for the Shapiro–Wilk and Kolmogorov–Smirnov tests). However, the variances were homogeneous (p > 0.24 for Levene’s test); as such, the one-way analysis of variance (ANOVA) was considered still proper for the analysis.

5. Results

5.1. Statistical analysis of creative outcomes

IBM Statistical Package for the Social Sciences (SPSS) version 25 was employed to calculate the intra-class correlation coefficient (ICC) estimates based on an average-measures, absolute-agreement, two-way mixed-effects model. The inter-rater reliability – a measurement of consistency between multiple raters – determines the agreement’s level across all judges (Heale & Twycross 2015). The ICC estimates fared in the good-to-fair range, with novelty scores of ICC = 0.66, appropriateness scores of ICC = 0.56 and usability scores of ICC = 0.55. The mean score of the three judges’ scores was taken for each metric to ensure that a single score would be attributed to these individual metrics of creativity.

According to Shapiro–Wilk’s and Levene’s tests, distributions of creativity scores were not normal or homogeneous. Thus, the Kruskal–Wallis test, the non-parametric equivalent to the standard ANOVA test, was employed for statistical analysis. Statistical analysis was conducted at the sketch level, treating each design solution as an individual and distinct observation. The test revealed statistically significant differences across conditions. The mean was calculated for each sketch from the three judges’ scores. The mean was extrapolated for each group within each DB type (see Table 1).

Table 1. Mean scores and standard deviation (SD) per brief

Note: “Groups” signifies the group of students per condition. “Students” counts the individual participants per condition. “Sketches” shows the number of design sketches produced per condition.

In examining the differences between various briefs using the Kruskal–Wallis test, we observed a significant impact on novelty scores related to the various stimuli, H (15, n = 1003) = 51.76, p < 0.05. The highest-performing group for novelty was the quantitative group (Brief Q, μ = 2.99, SD = 0.8). Conversely, the worst-performing group for novelty was exposed to the paired quantitative and physical stimuli (Brief QP, μ = 2.47, SD = 0.8).

The magnitude of the adjusted statistical significance of the average scores indicated that appropriateness scores were also significantly affected by providing different kinds of stimuli, H (15, n = 1003) = 59.38, p < 0.05. For appropriateness scores, the quantitative–visual group achieved the highest performance (Brief QV, μ = 3.17, SD = 0.66), whereas the worst-performing group was exposed to no stimuli (Brief BL, μ = 2.57, SD = 0.8).

Finally, the Kruskal–Wallis test revealed strong evidence of different conditions in usability scores, H (15, n = 1003) = 73.34, p < 0. According to mean scores, participants exposed to physical and contextual (PC) stimuli marked the highest scores in this metric (Brief PC, μ = 3.17 SD = 0.66). Lastly, the worst-performing group was tied between those exposed to the physical (Brief P, μ = 2.63, SD = 0.6) and those exposed to the quantitative (Brief Q, μ = 2.65, SD = 0.8).

These findings combined suggest the confirmation of hypothesis (H1). They indicate that different briefs do indeed stimulate differing dimensions of creativity encompassing novelty, usability and appropriateness in the context of the design outcomes.

5.2. Post-experimental survey statistics analysis

A total of 278 participants completed the post-sketch survey, and 36% of participants were female. The descriptive data can be found in Supplementary Appendix D. The results showed that 68% of participants who answered regarding the adequacy of the DB (DB1) felt that the DB was just right (compared with 12% who found it too specific, and 20% who found it too vague). Interestingly, in survey item DB2, which focused on the influence of the DB on their initial ideas, 33% of participants across all cohorts who received different DBs said that the DB did not influence their initial ideas. In contrast, 24% indicated that the DB helped them broaden their initial ideas, and 43% found it helpful to narrow down their initial ideas.

A one-way within-group ANOVA was conducted to assess the experimental condition’s impact on the perception of the DB. The ANOVA findings are displayed in Supplementary Appendix E. There was a statistically significant difference at the p < 0.05 level in DB perception (DB1) for the experimental conditions: F (15,262) = 1.93, p = 0.02. The effect size, calculated using eta squared, was 0.09. No significant correlations existed between DB influence and DB2 and the creativity metrics (novelty, appropriateness and usability). A summary of significant findings is displayed in Table 2.

Table 2. Key findings from mean scores and statistical analysis

Survey results revealed that those in the PC condition felt most that the DB was just right. In particular, among the 38 students who took the survey within this condition, 23 of them (61%) expressed this sentiment. However, it is worth noting that this perception did not necessarily lead to highly novel, appropriate or usable ideas. Those in the quantitative, visual and physical (QVP) condition scored the lowest mean, thus inferring that their brief was seen as too specific (by six students) but leaning slightly toward just right (10 students). The quantitative and physical (QP) condition had the most centered mean score, inferring that students with this condition were more prone to perceiving the DB as being just right.

Within the conditions, those in the PC (QPC) scored the highest mean in DB1 (seven participants), thus inferring that they found their DB the most vague out of all the groups. This result showed that having a higher level of detailed information assisted all brief variations expected from the BL brief.

We hypothesized that those in the BL group would be more likely to find the DB too vague (H2a). However, 60% of the BL group participants (13 out of 21) found it to be just right, 15% found it too specific and 25% found it too vague, which disproves H2a.

Our survey shows that briefs were generally perceived as just right by most participants, but some degree of ambiguity is desirable to foster innovation and creativity. Instructors should create the necessary environment that not only encourages this but also ensures that students do not become overly frustrated or confused in the process (Sawyer 2012, 2017). On an added note, perhaps, additional information may fixate the designer; thus, they tend to produce fewer novel solutions (Jansson & Smith 1991); however, this trend was not observed in our sample.

We further hypothesized that those in conditions with one or more stimuli would find it just right (H2b). Our survey supported this notion as 70% of participants across all conditions – exclusive of BL – found the DB just right, while 11% found it too specific and 19% found it too vague. Interestingly, the statistical analysis unveiled a noteworthy pattern. Participants in both the QVP condition and the PC condition not only regarded the brief as too specific but also failed to produce highly rated novel, appropriate or usable ideas. In contrast, those who scored high or low on creativity metrics had insignificantly different DB perceptions about the specificity of the parameters provided in the DB.

Therefore, the results affirm H2b. The inclusion of stimuli in the DB likely facilitated a better understanding of how the device functions and aided students in envisioning the practical implementation of their design solution. Nonetheless, this did not necessarily result in achieving top quartile scores, although it did demonstrate an enhancement over the mean score.

6. Discussion

6.1. Initial stimulation: design brief

This research aimed to highlight the different outcomes that emerged from the inclusion of DBs in teaching design, combined with a range of stimuli, including quantitative elements (the granular requirements specified in the DB), visual (the accompanying video of an orange squeezer), physical (a physical orange squeezer) and contextual (detailed script exhibiting users’ frustrations with existing orange squeezers and users’ need for orange squeezers) stimuli. The study investigated how these various elements affected outcomes, with a particular focus on how outcomes varied based on the inclusion of these components.

In trying to contribute to this research area focused on effective DB crafting for education, we found that pairing briefs with various inspirational stimuli increases groups’ performance across different metrics. Conversely, we found that what helps in one creativity metric can be a limiting parameter for another metric. Also, combining many stimuli all together was not effective in improving novelty scores. In our sample, novelty is enhanced when single stimuli are present (Brief BL or Q), but those in turn drastically decrease appropriateness and usability. Maybe, significantly improving these metrics requires more information and interaction with artifacts. It underscores the multifaceted nature of improving design outcomes and emphasizes the idea that a singular approach may not be universally applicable to all the diverse metrics under consideration.

While statistical analysis did not reveal significant correlations, our analysis aimed to uncover qualitative insights and potential trends that may inform future research or provide a richer context for understanding the complex relationship between experimental conditions and creativity scores. This approach thus aligns with the understanding that creativity is influenced by a multitude of factors that extend beyond simple correlations. As such, our approach in the subsequent section aimed to explore potential patterns or trends that might not be immediately apparent through traditional statistical correlations.

6.2. Analysis and discussion of visual data and the generation of creative design solutions

6.2.1. Novelty

Groups using the control brief recorded a higher mean score than those using other briefs. The control brief was nominated as “just right” by 75% of the participants (n = 15). Figure 5 shows an exemplar of a highly novel solution to the orange squeezer design. The solution was seen, to some extent, as surprising as it changed existing designs. It consists of a sophisticated machine utilizing a rotatable Archimedes screw system that grabs the orange and blends the inside of one orange. The gadget is holding the orange in place using a clamp, and a syringe spinner extracts the juice through an inlet. It is noteworthy that although no extra stimuli were provided to the design team, the generated solution scored 4.33 for novelty. One plausible but straightforward motive could be that the information in the BL brief was sufficient (just right) for most of this group – and gave impetus to their imagination. At the same time, it did not constrain or fixate them on the example solutions.

Figure 5. Sketch from brief BL (baseline) that scored high in novelty.

The condition where most solutions scored low on novelty was Brief QP, as this condition was the least favorable for novelty as realized from the mean scores. Five students in this group (50%) found the brief narrowing down their ideas, so that it could also play a role in lowering this group’s overall performance. In Figure 6, an example of a solution for this situation is illustrated. Remarkably, although no external visual examples were provided to the design team, the design outcome was almost a replica of the standard prototype for the manual orange squeezer on the market. The standard prototype is probably common knowledge to most designers and, combined with the constraining information, may have introduced to the participants of this group some level of fixation, thus affecting the novelty of their design solution. Only slight variations were observed, such as adding a filter to the base of the squeezer, which was not found in the visual example. On this line, additional studies also showed that exposure to visual examples constrains ideation (Koronis et al., 2020; Vasconcelos et al. 2017; Viswanathan and Linsey 2012). As a result, the Figure 3 solution scored only 1.00 point for novelty, and the brief itself was perceived by 50% of the students, the one here included, as a restricting one.

Figure 6. Sketch of an orange squeezer from Brief QP with a low novelty score.

6.2.2. Appropriateness

This metric’s highest scores are found in groups provided with Brief QV, where students were shown the BL brief supplemented by a QV example. Eleven participants (61%) found this brief suitable and possibly helped them conceptualize appropriate ideas. Figure 7 shows an illustration of a characteristic solution for this condition considered a satisfactory response to the brief requirements (i.e., manufacturability, washability and low cost). The supplementary quantitative information likely served to gain an insightful understanding of the design requirement and consequently enabled students to pay particular attention to BL briefs’ needs. However, the video exemplar probably helped us understand how such requirements can be fulfilled in practice. The solution was rated as highly appropriate and in line with the brief requests; thus, it scored a mean score of 4.67 out of 5. If the primary goal is to increase performance and user-friendliness, it is arguable that providing a visual or video example is useful for appropriateness (see Koronis et al., 2020). This assertion can be supported by Fu, Cagan, & Kotovsky (2010), who claimed that good examples can help design teams generate high-quality ideas that accomplish the brief’s requirements. Accordingly, students can build on solutions that are known to be effective.

Figure 7. Sketch from BL brief QV with a high appropriateness score.

In the state where most solutions scored low in appropriateness, the BL brief without any exemplar was provided to the students. The survey found that participants designated this brief as highly suitable (13 out of 21), while about half (11 in total) found it helpful in narrowing their ideas down. Figure 8 illustrates an example portrayed by a complex mechanism composed of many parts, making it difficult to manufacture, clean and largely exceed target costs, which were indicated. As a result, this solution did not address the design requirements. It is judicious to cogitate that the lack of outside help such as quantitative requirements with specific numerical values or a visual prototype has been the leading cause of these low scores (see Koronis et al., 2020).

Figure 8. Sketch from brief BL that fared among the lowest scores in appropriateness.

6.2.3. Usability

When participants were provided with the BL brief paired with all stimuli, namely Brief QVPC, it was evidenced that most solutions scored highly for usability. Not surprisingly, this group found the brief not narrowing down their ideas while it was perceived by 13 students (72%) as a just-right brief. An example of a knowledge solution created in this condition is seen in Figure 9. The solution consisted of a practical device comprised of an orange crusher to squeeze out the juice. A juice collector is mounted at the base of the squeezer, making the extraction of juice feasible from more than one orange at once. The available quantitative and contextual information likely helped students generate efficient design outcomes that did not violate the brief constraints.

Figure 9. Sketch from brief QVPC that ranked high in usability.

Furthermore, the instructional video example possibly helped to gain insight into the devices’ function and how they can be effortlessly executed and created in practice. The solution was seen as very useful and scored 4.33. As the literature shows, good examples can help design teams generate high-quality ideas that fulfill the brief’s requirements (Fu et al. 2010). On another note, contextual briefs help inspire designers by communicating the higher management’s vision for the design project, aiding designers to commiserate with user needs or difficulties while considering the broader societal benefit of their product or design (Guay, Vallerand, & Blanchard 2000).

When students were exposed to a physical example along with the brief (P), most design outcomes were scored low in the usability metric. Half of the group participants (10 students) found it restricting their ideas, while four (20%) nominated it as too vague. Figure 10 depicts an ideal solution that did not satisfy functional needs. The solution comprised a detachable sharpened straw, which could be incorporated into the fruit. The lower part of the straw had a rough surface when turned around and broke through the orange skin. The idea was that the device would allow users to drink the juice directly from the orange. However, this was deliberated as impractical and scored 1 for this metric. The results show that when quantitative information is absent, usability seems to be lowered, and thus, the physical example alone was not effective in conveying how the device could be used in real life.

Figure 10. Sketch from brief P that scored low in usability.

6.3. Post-sketch survey assessment

The inclusion of a post-sketch survey served to validate our hypotheses regarding participants’ perceptions of the DB. In the future, this survey can be expanded to include personality data and deeper personal insights. This information, paired with the DBs, was a thoughtful way to assess the success of the DBs administered to varying groups.

Interestingly, those who felt the DB was too specific were referring to the quantitative, video and physical conditions, while those who thought it was too vague were referring to the PC conditions. Those who found the DB just right were referring to the quantitative and physical conditions, which is interesting as it infers providing potential designers with an interactive object, such as the physical orange squeezer, and the parameters within which to create are most liked by designers. Lastly, those referring to the quantitative, video, physical and contextual conditions found the DB to influence their ideas the least, while those with the quantitative, video and physical conditions were most likely to find the DB to either narrow down or broaden their ideas. We have noticed that contextual briefs, in general, assist in motivating designers to empathize with user needs or difficulties and see the broader societal benefits of their work (Guay et al. 2000). However, this was not evidenced in our samples.

In conclusion, our findings suggest that the use of stimuli in conjunction with a BL DB can indeed impact creative outcomes and influence participants’ perceptions of the design process. Importantly, the strength and significance of these patterns varied across different experimental conditions, underscoring the need for further exploration in future research.

6.4. Judge’s rating assessment

Judges were selected carefully for this study to ensure that reliability in scoring was attained. Amabile (1996) places great emphasis on the selection of appropriate judges for assessment. The need for expert judges was conclusively validated by Dollinger & Shafran (2005) and later by Kaufman et al. (2008), whose studies found that significant differences existed between expert and non-expert judges’ creativity assessment scores.

Hennessey, Amabile, & Mueller (2011) addressed the key limitations of the CAT tool, which include the time required to select appropriate judges for evaluation and employ the assessment tool. Additionally, the CAT is not tied to a particular stimulus in methodology, and its use of the paired 6-3-5/C-Sketch method is relatively novel and worthy of exploration. Kaufman (2016, p. 78) argues that since experts do not discuss the rating process with one another, this becomes counterintuitive but essential. Hekkert & Van Wieringen (1996) found that consensus usually only showed the original opinion of one (likely vocal) expert when experts discussed their opinions. We had our judges go through a training session to understand the evaluation rubric and process. All raters evaluated 18% of the data to provide sufficient inter-rater reliability until a good agreement in rating was obtained. We conducted the training of our judges in a cyclical fashion, which involved ongoing discussions and feedback sessions to calibrate their assessments. This iterative process aimed to enhance inter-rater reliability and ensure a consistent understanding of the evaluation rubric and criteria among all judges.

7. Limitations and significance of results

We created a mixed-methods model of creativity stimulation and measurement; it is apparent that there are limitations in the current experimental design, leading to a low level of correlation between variables. It is also evident that refining this model is crucial, particularly concerning the validation of creativity scores. The motivation to add usability as a metric was based on our expert judges’ feedback from our earlier pilot study. In that study, judges felt that the appropriateness metric – which was evaluated in relation to the DB’s requirements – did not sufficiently account for the idea’s effectiveness in satisfying the end user’s goals. We acknowledge the difficulty in assessing user–product interactions unless a usability test is run (Sonderegger & Sauer 2009). Nevertheless, we initially expected that the sketches would provide sufficient insight into the product’s performance and that our expert judges would evaluate them accordingly. However, it is apparent that further refinement of our model and the introduction of a secondary measure of creativity are warranted to bolster the overall validity and comprehensiveness of our study.

We acknowledge the fruitfulness of using a peer evaluation of the sketches by student designers. Secondly, our research was limited in identifying the reasons behind attaining creativity scores via our observation-based assessment. This can be further expanded by having a more investigative post-sketch survey to flesh out more information on our student’s cognitive process when they receive briefs paired with various stimuli. Lastly, our research is limited to providing participant profiles to better understand our sample in terms of BL creativity scores, whether creative personalities emerge from the sample, and their perceptions before and after the sketch exercise.

An additional implication stemming from the collaborative nature of this design task is the potential dependence of findings on the specific idea generation method that was employed. In the context of the C-Sketch/365 process, it is important to note that sketches were not created independently but by a team of student designers. However, it is reasonable to regard the final collective outcome as a distinct and independent observation, initiated by one student and further developed by other team members. Consequently, we acknowledge that our research findings do not explain whether implicit factors inherent in the entire creative process may either enhance or reduce the impacts on individual creative outcomes.

The stimulation and measurement of creativity have been attempted within this study through the use of existing but disparate design tools and frameworks: from DBs paired with different stimuli to the 6-3-5/C-Sketch methodology, post-sketch survey and sketch assessments for the stimulation and measurement of creativity. This mixed application of methodologies was assessed via a post-sketch survey and judgment of creativity based on a rubric-based system. Despite the inherent limitations, this paper introduces a novel and controlled method for investigating and quantifying creativity within research and design contexts. This method is aimed at refining our understanding of creativity and its promotion across various fields and workplaces, with particular relevance to educational settings.

8. Conclusions

This study aimed to answer two research questions, which focused on the identification of stimuli that augmented the creativity metrics and tested how various DBs were seen by the participants. In evaluating the stimuli that enhance creativity, our study identified quantitative information and the BL brief (Q) as positively influencing novelty scores. However, representations that enhanced the appropriateness of design outcomes were the pairing of QV, while usability increased when all effects were combined (QVPC). The statistical analysis indicated that those who scored high or low on creativity indices had insignificantly different perceptions of the DB. No significant correlations existed between DB perceptions and creativity metrics.

In our study, we introduced four distinct stimuli to augment the DB, with the aim of stimulating creativity. We meticulously analyzed the advantages and limitations associated with each stimulus, contributing valuable insights to the ongoing discourse on the composition of effective DBs. However, our contributions extend beyond the confines of educational design settings, carrying implications that are particularly relevant to boosting creativity in other institutional settings.

We firmly advocate for a broader and more concerted endeavor in the realm of controlled research, specifically directed toward optimizing DBs to unleash creativity. We encourage educators, researchers and practitioners to recognize the significance of nurturing creativity within educational contexts. The insights garnered from this study can inform educational practices, fostering an environment where creativity is not just encouraged but actively cultivated.