Idea evaluation as a design process: understanding how experts develop ideas and manage fixations

Abstract

Idea evaluation is used to identify and select ideas for development as future innovations. However, approaching idea evaluation as a decision gate can limit the role of the person evaluating ideas, create fixation bias, and underutilise the person’s creative potential. Although studies show that during evaluation experts are able to engage in design activities, it is still not clear how they design and develop ideas. The aim of this study was to understand how experts develop ideas during evaluation. Using the think-aloud technique, we identify different ways in which experts develop ideas. Specifically, we show how experts transform initial idea concepts using iterative steps of elaboration and transformation of different idea components. Then, relying on concept-knowledge theory (C-K theory), we identify six types of reasoning that the experts use during idea evaluation. This helps us to distinguish between three different roles that experts can move between during evaluation: gatekeeper, designer managing fixation, and designer managing defixation. These findings suggest that there is value in viewing idea evaluation as a design process because it allows us to identify and leverage the experts’ knowledge and creativity to a fuller extent.

1. Introduction

Idea evaluation plays an important role in the innovation process. Its purpose is to identify promising ideas for innovations and to filter out ideas that do not present value for the company (Hammedi, van Riel, & Sasovova 2011; Eling, Langerak, & Griffin 2015; Sukhov 2018). Idea evaluation has been historically conceptualised as a decision gate, whereby experts estimate the likelihood of an idea being successful with the aim of reducing uncertainty and risk related to idea implementation (Cooper 1990; Cooper & Sommer 2016). Consequently, this has led to an impression that idea evaluation can be optimised and outsourced, and that good ideas can be found when specific criteria are applied (e.g., Magnusson, Netz, & Wästlund 2014; Eling et al. 2015), which has influenced managers to look for other ways of optimising idea evaluation as a decision gate. However, this search for the “best” ideas during a period of high uncertainty has created suitable conditions for fixation bias, whereby individuals who are tasked with making decisions regarding idea quality are locked into a single frame of reasoning and the specific role of a gatekeeper. This type of fixation in relation to idea evaluation, whereby uncertainty leads to avoidance of creativity, has been found to result in the dismissal of new and original ideas (Licuanan, Dailey, & Mumford 2007; Mueller, Melwani, & Goncalo 2012), which are crucial for innovation.

Previous studies have suggested a different conceptualisation of idea evaluation whereby experts are not limited to idea evaluation, but rather are encouraged to envision a potential future and forecast what the ideas they are evaluating can become (Hatchuel & Weil 2009; Florén & Frishammar 2012). Furthermore, research suggests that individuals can alter their behavior during idea evaluation and approach idea evaluation as a creative process (e.g., Harvey & Kou 2013; Sukhov et al., 2018). Moreover, encouragement of creativity during evaluation has been found to enhance the generation and selection of ideas (Gillier & Bayus 2022). This approach highlights the importance of creative idea development activities during evaluation such as elaboration and transformation of ideas because it reveals why ideas are considered to be good and how they can be even better (Sukhov et al. 2021). Furthermore, previous studies have suggested that ideas emerge from fragments of the problem and solution elements of an idea (Maher, Poon, & Boulanger 1996; Dorst & Cross 2001), and that idea generation, development, and evaluation are better viewed as a holistic and iterative design process (Dorst & Cross 2001; Sosa 2019). Recent studies have also found that a combination of generative and evaluative activities during idea evaluation yields better task performance because it improves the ability to recognise creative and original ideas and the ability to better discriminate between feasible and unfeasible ideas (van Broekhoven et al., 2021). Furthermore, some studies have pointed to the notion that experts can adopt different roles during idea evaluation. For instance, experts might fixate on the initial idea and use their knowledge to assess the potential of the initial idea concept, but they could also act as designers, recognising the idea’s hidden potential, reconceptualising it and finding a better path for the idea’s development (Sukhov et al. 2021; Hua, Harvey, & Rietzschel 2022). However, at present, we have little understanding of how experts actually engage with ideas during evaluation, the extent of fixations during evaluation, and whether they are able to act as designers in a consistent manner.

The aim of this study was to better understand how experts engage in creative idea development activities (i.e., design activities) during the idea evaluation process. More particularly, we focus on understanding the extent of experts’ fixation during idea evaluation and on their capacity to act as designers in an effort to manage fixations. By doing so, we aim to expand our conceptualisation of idea evaluation as a holistic and generative design process. Thus, we undertook an explorative study wherein we collected data from five industry professionals tasked with evaluating a range of ideas for new product and service innovations using the think-aloud method (Ericsson & Simon 1980; Van Someren, Barnard, & Sandberg 1994). This enabled us to gain a better understanding of how different experts engage with ideas in terms of idea evaluation and idea development, and to expand our perspective regarding the design activities and the roles of experts during idea evaluation.

Our study makes three main contributions to the literature. First, we show that experts perceive ideas in different ways, in terms of both how they evaluate and how they develop ideas. This suggests that early ideas require engagement, conversation, and discussion to enable experts to realise their potential, understand their limitations, and integrate the experts’ knowledge in a complementary manner. Second, we find that ideas have a dual purpose, acting as both an object for evaluation and a trigger for the reconceptualisation and creation of new ideas. Specifically, experts can fixate on ideas by evaluating and further elaborating on the initial idea concept, or they can defixate from the initial idea concept and envision an alternative development path for the idea. This changes our understanding of the role of idea evaluators from one of gatekeepers to one of the potential designers who are more involved in the co-creation of ideas. Third, we identify specific design patterns that can be used by experts for developing ideas further. This helps us to advance our understanding of the roles of experts during idea evaluation and show how ideas can evolve in specific directions. Hence, we show how different strategies can be used to develop ideas further during evaluation. Overall, the findings of our study provide value to innovation managers and design scholars and practitioners by illuminating the intricate details of experts’ designerly behavior and their contributions to creative idea development during evaluation.

The rest of the paper is organised as follows. First, we present a theoretical background explaining the main concepts underlying our investigation. Next, we present a description of the method used and our results. Finally, we discuss the implications of our findings in light of our theoretical framework and outline recommendations for managers and for future research.

2. Theoretical background

2.1. Idea evaluation

Idea evaluation is generally understood to be a process aimed at identifying ideas that have the highest potential for the organisation and filtering out ideas that do not present sufficient value (Hammedi et al. 2011). The challenges associated with idea evaluation include the high degree of uncertainty that the evaluators experience, partly because of the incompleteness of early versions of underdeveloped ideas (Chang, Chen, & Wey 2007; Frishammar, Florén, & Wincent 2010), and partly because of the ideas’ originality (Moreau, Markman, & Lehmann 2001; Deichmann & Ende 2014). Thus, people tasked with idea evaluation may have difficulty understanding the ideas, which can result in underappreciation of their quality (Sukhov 2018), adopt a negative attitude toward ideas that do not fit into a specific framework (Licuanan et al. 2007), or avoid creative ideas because of the high level of uncertainty associated with their future development (Mueller et al. 2012).

Idea evaluation is often explained in terms of a rational and systematic approach that can involve specific criteria for determining idea quality (e.g., Balachandra & Friar 1997; Carbonell-Foulquié, Munuera-Alemán, & Rodrıguez-Escudero 2004; Frederiksen & Knudsen 2017). The use of these criteria is aimed at directing and anchoring the evaluators’ reasoning and helping them to make a more informed decision regarding idea quality (Kudrowitz & Wallace 2013; Magnusson et al. 2014). However, a more holistic and intuitive decision-making approach to idea evaluation has also been portrayed as useful for experts in determining idea quality because of the need to quickly sort through large quantities of ideas while relying on the individuals’ domain knowledge and expertise (Dane & Pratt 2007; Dayan & Di Benedetto 2011; Magnusson et al. 2014). Moreover, research suggests that a combination of a rational analytic approach and an intuitive holistic approach helps to increase the overall efficiency and effectiveness of idea evaluation (Eling et al. 2015; Dziallas 2020; Sukhov et al. 2021).

Given recent developments in research, an increasing number of studies are pointing toward the notion that idea evaluation should be approached as a generative task to take greater advantage of the evaluating person’s knowledge and creative capacity (Sosa 2019; Sukhov et al. 2021; van Broekhoven et al., 2021). This is explained by the need to understand what the ideas can become, which requires a forward-looking perspective and active engagement with the ideas during evaluation. However, the extent of the experts’ engagement with ideas during evaluation and an understanding of what is happening to ideas during this process remain unclear (Sosa 2019).

2.2. Ideas for innovation

In innovation research, ideas often refer to creative products or descriptions of solutions to problems whereby people engage in ideation activities to generate early ideas on how to solve a specific problem (Osborn 1957; Dorst & Cross 2001; Hua et al. 2022). Despite the reference to ideas as solutions to problems, the definition of what is a problem and what is a solution is rarely elaborated upon, making it difficult to understand what makes a good idea, or even when an idea is sufficiently developed to be considered complete (Sukhov 2018; Sosa 2019). Hence, it is important to clarify that a problem is something that describes “the disharmony between reality and a person’s preferences for the reality” (Smith 1988, p. 1491), and can be triggered by someone’s dissatisfaction with a situation in the hope of exchanging that situation for a better one (Sukhov, Magnusson, & Netz 2019). A solution refers to an understanding of a problem and a suggestion as to how that problem can be solved, and what is required to solve the problem (Suh 2001; Sukhov et al. 2019). Furthermore, ideas are contextual—they are conceived and communicated in specific surroundings, and this context may also play an important role in relation to perceptions of an idea’s intentions and quality. Previous studies have found that clarifying and elaborating on these components during the process of idea description and communication helps to concretise an idea, which increases the comprehension of the idea by external evaluators and leads to a higher level of perception of the idea’s quality (Sukhov 2018).

Based on the classical understanding of idea evaluation as a decision gate, the role of people involved in evaluation is to determine whether the solution to a specific problem is appropriate, and whether the idea should be selected for further development and implementation (Cooper 1990, 2014). However, in practice, early ideas submitted for evaluation are seldom clear and complete, and require active interpretation by filling in the gaps in idea description using subjective knowledge (Sukhov 2018). This makes the notion of objective idea evaluation difficult because interpretation of early ideas by different people can be highly subjective in light of their unique experiences, differences in knowledge, and personal values affecting their sensemaking process (Sukhov et al., 2018; Ulrich & Nielsen 2020; Sukhov et al. 2021).

Research has also found that ideas have different characteristics. They can act as mental representations that are often unelaborated and transmittable, but can also act as physical representations that are interpretable and communicable through action (Hua et al. 2022). This means that ideas need to be both conceived and communicated, and ideas that are communicated by the idea creator can trigger further intellectualisation by person evaluating the idea, which can be continuous and emergent, and lead to the creation of new ideas or idea enhancements by the individual (Sosa 2019; Ulrich & Nielsen 2020; Sukhov et al. 2021; Hua et al. 2022). This conceptualisation challenges the understanding of idea evaluation as a decision gate, and points to the importance of elaboration, interpretation, clarification, and the creation of new ideas during evaluation. Furthermore, it emphasises the need for idea development during the idea evaluation process and the need to understand the potential differences in how ideas are perceived by different experts. This leads to the following proposition:

Proposition 1: Early ideas require interpretation and elaboration, and are perceived differently by different experts during evaluation.

2.3. Idea development and design process

To understand how individuals come up with new and appropriate solutions to problems, it is important to model the creative process using a theory that can explain creative thinking during idea development. The design theory that is best suited to modeling this type of creative thinking is the concept-knowledge (C-K) theory presented by Hatchuel & Weil (2003). Specifically, C-K design theory is a recognised lens for understanding the interactions between concepts and knowledge that enable the creation of new ideas (Choulier, Coatane’a, & Forest 2010; Hatchuel, Masson, & Weil 2017). In C-K theory, new propositions are generated through the interaction of two spaces, the concept space and the knowledge space, which can extend and interact with each other using four operators, two allowing extension within the same space (C- > C and K- > K) and two others enabling movement from one space to the other (C- > K and K- > C). For a more detailed explanation of C-K theory, see Hatchuel & Weil (2003, 2009).

The knowledge space characterises propositions that have a logical status (K) (i.e., the individual knows that something is true or false). The knowledge space then interacts with the concept space where propositions do not have a logical status (C) (i.e., something can be imagined but the individual cannot say whether it is true or false). Therefore, for any generated concept, or a problem-solving element of an idea, it is possible to identify the knowledge that was used for its generation. For instance, in the illustrative example of the Mg/CO₂ engines used for Mars missions presented in their study, Hatchuel and Weil identified that knowledge of Mars’ environment (such as frequent storms) led to the new concept of rapid refueling of CO₂ for unplanned moves (Hatchuel & Weil 2009). The concept space is then organised in the form of a tree, with the concepts following the branch belonging to the same set of knowledge. To create a new partition (branch), the expert must call upon different knowledge to identify a new property that allows the generation of a different concept (Hatchuel & Weil 2003). These qualities enable researchers to use C-K theory for analysing design processes, and make explicit the links between knowledge and concepts that are important for the development and creation of ideas (e.g., Ondrus & Pigneur 2009; Zedin et al. 2017). Thus, the development and transformation of ideas during evaluation can be made explicit by tracking the knowledge mobilised and the concepts generated.

Proposition 2: Experts are able to use their knowledge and imagination for designing and developing ideas during evaluation.

2.4. Fixation and defixation

C-K theory also enables the identification of specific subprocesses that can occur during the design process. For instance, C-K theory can help to identify fixation, which is one of the major cognitive biases hindering creativity (Jansson & Smith 1991; Agogué et al. 2013). Fixation means that individuals rely on the same knowledge base to generate concepts, resulting in fixation on a limited set of concepts and inability to access a different knowledge base for associating new and different sets of concepts (Purcell & Gero 1996; Hatchuel et al. 2017). Therefore, the idea generator’s fixation can lead to them generating only similar and unoriginal ideas, while the evaluator’s fixation can lead to bias in their evaluation by following a single frame of reasoning and not taking into account other ways in which ideas could have greater potential. Thus, when evaluating ideas, experts can experience fixation by focusing on the evaluation or elaboration of the initial idea concept, but they can also experience defixation by, for example, interpreting the meaning of the idea drawing on their past experience, or even reinterpreting the task (Sukhov et al. 2021). Indeed, there are mechanisms that can help to overcome these fixation effects. One study found that the use of examples corresponding to creative solutions that are beyond the individual’s initial fixation can be a means of overcoming fixation effects and achieving defixation (Agogué et al. 2013). However, other studies have found that the use of examples can have an adverse effect on creativity and can even reinforce fixation (e.g., Smith, Ward, & Schumacher 1993). Other tools that help to defixate and result in generating more original and appropriate ideas involve negating the category of solutions included in the fixation (Ezzat et al. 2020) or using minimal directive feedback (Ezzat et al. 2017), because they help to prompt the individual to look for different ways of problem-solving.

Furthermore, experts can exhibit different types of behavior during idea evaluation. Previous studies have found that experts can act in a more proactive and open way, trying to actively engage in idea development, or they can be more reserved and conservative during evaluation (Sukhov 2018). In a similar vein, by engaging in actions that aim to foster creativity, experts evaluating ideas can be understood as creative leaders (Mainemelis, Kark, & Epitropaki 2015) who actively envision how ideas can be developed and what needs to be done to achieve creative outcomes. Hence, when experts evaluate ideas, they can adopt different behaviors in terms of fixation, either reinforcing it or trying to avoid fixation effects. To understand how this happens, it is important to understand and analyze idea evaluation as a design process. The knowledge that experts possess can either constrain them to thinking in a specific way or act as a trigger for creating a new partition from the initial idea, and thus defixate the initial idea intention by changing its framing (Dorst & Cross 2001). However, how this process unfolds and what types of fixations or defixations occur remain to be understood.

Proposition 3: Experts exhibit fixation and defixation when developing ideas during evaluation.

Against this theoretical background, in the next section, we explain our methodological procedure in an effort to further understand how experts engage in creative idea development during idea evaluation.

3. Method

3.1. Data collection

To study how experts engage in idea development during evaluation, we used a combination of a questionnaire survey together with the think-aloud protocol to collect data (Ericsson & Simon 1980). The think-aloud protocol focuses on construction rather than recall of information (Nisbett & Wilson 1977; Russo, Johnson, & Stephens 1989; Ericsson & Simon 1993) allowing researchers to document individuals’ thought processes as they occur, and is often used in design studies (Payne 1994; Atman & Bursic 1998; Dorst & Cross 2001; Coley, Houseman, & Roy 2007).

The data consisted of individual idea evaluations by five experts from a global vehicle manufacturing company located in France (the ideas were documented and evaluated in English). These experts worked in the same company, had managerial positions, and possessed a high level of technical competence and an understanding of their company’s business. Our study followed the company’s own innovation process, wherein ideas were initially generated in a global online idea “jam,” with employees suggesting improvements to the company’s operational and business functions.

The experts were individually presented with 31 ideas that had been created by other employees, and tasked with evaluating these ideas. All ideas were presented in a uniform format containing a textual description of the idea, ranging between a few sentences to several paragraphs in length. The ideas were labelled with a number and a name that were used to keep track of the ideas in the internal idea management system. The experts were instructed to make a “rough screening” of incoming ideas and speak aloud their thoughts during this process using the think-aloud approach. Idea evaluation took between 60 and 90 minutes, wherein ideas were rated on a scale between 0 and 100 based on two criteria: how good the idea was in the short term (ST), and how good the idea was in the long term (LT). Applying these criteria allowed the experts not only to adopt a holistic perspective regarding the idea’s potential, but also to assess the idea’s value to the company using different temporal perspectives. If an idea received a high rating (above 50 on the 100-point scale) it was considered a promising idea from a short- and/or long-term perspective. If the idea received a low rating (below 50) it was considered to be of lower quality.

The ratings for all ideas were collected, but because of practical limitations, the think-aloud protocols were only collected for 64 of the 155 idea evaluations. During the idea evaluation session, the experts performed think-aloud protocols for about an hour and then continued their evaluations of the remaining ideas without the think-aloud protocol. The think-aloud protocol cases were audio-recorded and transcribed verbatim.

3.2. Data analysis

The data were analyzed in three steps (see Figure 1). First, to understand how different experts evaluated ideas, we performed a statistical test using Cronbach’s alpha wherein we compared the experts’ ratings of all of the ideas. This allowed us to observe whether there was agreement or disagreement in terms of how the experts perceived idea quality from a short- and long-term perspective.

Figure 1. Research process.

Second, to gain a better understanding of what happens to ideas and how they are developed during evaluation, we used a model of an idea for innovation (see Sukhov 2018; Sukhov et al. 2019) to trace idea-development activities from the think-aloud protocols. This enabled us to gain a qualitative understanding of the activities that are relevant to idea development, and we identified and coded these activities based on the five components of an idea: i) Context, that is, where/when does the idea occur?; ii) Situation, that is, what is the idea about?; iii) Importance/dissatisfaction, that is, why and to whom is the idea important?; iv) Procedure, that is, how can the idea work?; and v) Resources, that is, what is needed for the idea to work? Inspired by Dorst & Cross (2001), Sukhov et al. (2019), and Hua et al. (2022), we characterised these activities in one of two ways, either as an elaboration activity (elaborating on and completing the initial idea), or as a transformation activity (changing the idea into a new and different idea). This enabled us to code the content of the initial ideas and trace all of the additional idea fragments that were generated by the experts, resulting in a detailed examination of 456 idea development activities.

Third, to understand how the idea development process unfolds, we used C-K theory to identify all of the 456 idea development activities and classify them in terms of concepts and knowledge (Kroll, Le Masson, & Weil 2013), relying on the definitions of the two spaces included in C-K theory. Thus, any activity with a logical status for the expert evaluator (true or false) belonged to the knowledge space and any activity without a logical status for the expert belonged to the concept space. We organised the knowledge space by grouping knowledge into sets based on related themes and the concept space by shaping it in the form of a tree. The evaluated idea corresponded to C0 in our C-K representation (the initial concept to start the design reasoning), and it could be either at the top of the tree with all of the concepts generated by the expert derived from it, or on a branch of the tree if the expert decided to go beyond the initial idea, and thus created another partition to generate concepts that were not derived from the original concept.

During the analysis, we noticed the presence of negative conjunctions, meaning that some of the concepts generated by the experts, or even the C0, were considered unfeasible and rejected as alternatives. Moreover, for each partition created in the C space, we identified the knowledge required for the generation of the concepts (including whether this knowledge was implicit, in which case we made it explicit in the C-K representation). This allowed us to identify two types of paths that could occur during the development of an idea: an Easy path and a Difficult path.

Easy paths can be characterised by mobilising knowledge that is directly related to the idea, or that is easy to access for the expert. Following an easy path can also be related to a fixating behavior, since the knowledge related to this type of a path is restrictive with respect to the idea (Hatchuel et al. 2017), and does not allow the expert to explore other paths. Thus, in cases when the concepts were generated from knowledge of existing solutions in the same domain as the initial idea, or when the knowledge allowed the expert to think that the concepts were easily implementable, the expert exhibited a fixating behavior by following an easy path of reasoning.

Difficult paths occurred when the expert mobilised different knowledge with respect to the idea, which resembled an expansive process (Hatchuel et al. 2017). Following a difficult path resembled a defixating behavior, where the expert used knowledge from different knowledge domains in relation to the initial idea, generated technical solutions that did not exist, or identified complexities in implementation of the concept. Thus, in cases when the generated concepts were not derived from knowledge of existing (or easily accessible) solutions or when the expert mobilised knowledge that did not belong to the same domain, the expert exhibited a defixating behavior by following a difficult path of reasoning. Depending on the knowledge that was mobilised to generate the concepts, we were able to differentiate each partition of the concept tree into either an easy path or a difficult path.

To limit bias during the coding of ideas based on C-K theory, all ideas were coded independently by two researchers. Initially, the independent coding resulted in a 70 percent agreement between both researchers, which was followed by a discussion focusing on cases containing disagreement. After a review and a discussion of the coding differences the coders reached a mutual agreement on all cases.

These procedures enabled us to trace the experts’ generative progression and creation of new partitions and concepts based on their knowledge. This enabled us to compare the idea development activities and design patterns and to outline the different approaches the experts used when engaging with ideas.

4. Results

4.1. How are ideas evaluated?

To determine whether the experts agreed or disagreed on their ratings of an idea’s quality, we performed a Cronbach’s alpha test on all ideas evaluated by all experts. The results were .04 for the short-term (ST) and .43 for the long-term (LT) ratings. This indicated significant levels of disagreement among the experts, and thus differences in how they perceived idea quality. These results were in line with our first proposition that ideas are perceived differently by different experts, but pointed toward a need for closer qualitative examination of how ideas are interpreted and elaborated by experts during idea evaluation.

4.2. What is happening to ideas?

To understand what happens to ideas when experts evaluate them, we applied the idea model (see Sukhov et al. 2019) in an effort to identify changes to the idea in terms of its various components (context, situation, importance/dissatisfaction, procedure, resources). This made it easier to trace idea development activities and distinguish between elaborations of the initial idea or transformations of its various elements. For the sake of brevity, we illustrate the results of our analysis using the example of Idea 2 (Customer Services—Remote diagnostic information). Additional examples are provided in a Supplementary Material.

“Idea 2: Customer services—remote diagnostic information

Provide the customer support and service by using the remote tools and virtual mechnics/technicans to: Diagnose the vehicle and inform customer of the health of the vehicle. The intention would be to investigate enhanced diagnostic solution in order to provide as early as possible information on vehicle status to Customer Services. The goal is for [the company] to know about any anomaly before the customers knows and thus become a pro-active service provider.”

Table 1 shows the analysis of the initial idea (upper part of the table), the breakdown of the idea into its five elements, and a partial extract of idea development activities provided by Expert 3 (lower part of the table). It can be seen from Table 1 that idea development activities iterate between different components of the idea, forming an additional layer of information provided by the expert.

Table 1. Breakdown of Idea 2 and idea development activities provided by Expert 3

This visualisation shows that the expert quickly saw a problem in relation to the initial idea’s proposed solution, namely, that the initial solution of “applying remote tools and performing diagnostics” was insufficient, and saw that the real problem was actually “knowing how to diagnose the vehicle.” The expert elaborates that the new solution might lie in “asking good questions to the customer”, which shifts the initial idea into a new area by focusing on finding new ways of understanding the customer’s needs.

When comparing the number and type of idea development activities of all of the experts using this template, we also found that they undertook a different range of activities, and perceived ideas in different ways.

Figure 2 shows that three of the experts varied in terms of their number of idea development activities in the case of Idea 2. They also differed in terms of the extent to which they were able to transform the idea. The experts also rated the short- and long-term value of this idea differently. Further comparisons of all of the experts’ engagement with the ideas showed that they undertook between two and 17 activities per idea (with an average of eight activities per idea, see Figure 3 for an example of activities by Expert 1), and engaged in anywhere between zero and four idea transformations (with an average of one transformation per idea).

Figure 2. Number of idea development and transformation activities of Expert 1, Expert 3, and Expert 4 in relation to Idea 2 and the idea’s evaluation scores.

Figure 3. The range of idea development and transformation activities for different ideas provided by Expert 1.

These findings provided further support for our first proposition, confirming that ideas require elaboration, and that they are perceived and evaluated differently by different experts. Furthermore, in line with our second proposition, we could see that experts are prone to developing and transforming ideas, but that the extent of their development and transformation activities varied depending on the ideas with which they were engaging.

4.3. What does the development process look like?

To further understand the process of idea development during evaluation, we coded all of the ideas based on C-K methodology, as explained in the Method section above. For additional clarity, Figure 4 shows an example of our coding illustrating the development of Idea 2 by Expert 3. On the left-hand side, in the think-aloud extracts from Expert 3, we have annotated the various elements that allowed us to apply the coding based on C-K theory that appears on the right-hand side. In the extracts, knowledge is presented in bold type and concepts are presented in blue and underlined. The verbatim comments indicating the rejection of the evaluated idea are presented in red and the indications that help to explain the new partition that the expert creates when generating new concepts are presented in italics. In the C-K coded representation, concepts are represented as a tree on the left and knowledge is grouped by theme on the right.

Figure 4. Development of Idea 2 by Expert 3 based on C-K theory.

To trace the expert’s reasoning, first we identified the initial concept (C0) as the initial idea to be evaluated. For the evaluation, the expert started by appealing to their previous experience, and found that the idea was not new. The expert then relied on his knowledge to claim that the initial idea was focusing on the wrong area, that is, “The problem is that when you have the […] codes it’s not enough to diagnose.” Therefore, his knowledge and experience allowed the expert to generate a concept that differed from the initial idea, that is, “asking good questions to the customer”. As shown in the dotted oval in the C space in Figure 4, this new concept involves the customer in the solution, whereas the initial concept did not. Thus, the expert has created a new partition whose C0 is not the same as the idea he is evaluating. Therefore, we formulated a new C0*, which is the origin of the concept tree.

After adding some knowledge about the technology, the expert rejects the initial idea (the initial C0), which is illustrated with a negative conjunction and presented in red. Despite this rejection of the initial idea, the expert continues to engage with the idea, connecting the idea to knowledge about other existing solutions and experiments related to the subject. This allows the expert to generate a new concept, that of “remote diagnosis.” This concept is placed in the concept tree above the previously generated concept because it is less specific. Finally, the expert concludes the evaluation by adding knowledge about complexity and confirms that the idea to be considered is much more advanced than the one proposed. The generated concepts are not based on knowledge of existing solutions, and the related knowledge indicates complexity in implementation, which corresponds to a difficult path in the expert evaluator’s concept tree.

By mapping all 64 cases and systematically comparing them for differences and similarities, we were able to identify six different types of reasoning in relation to the experts’ idea evaluation, as shown in Figure 5.

Figure 5. Types of reasoning used in the experts’ idea evaluation.

The previous example of Idea 2 that was developed by Expert 3 corresponds to Type B reasoning “Discovering a new path”, which produces a negative conjunction of the initial concept and then proposes an alternative path to be explored in an effort to generate more value for the company. The various types of reasoning are explained as follows.

1. 1. Type O “Being a gate-keeper” corresponds to behavior where only knowledge is mobilised and no additional concepts are generated during the evaluation.

2. 2. Type N “Dismissing alternative paths” corresponds to behavior whereby the experts reject their own generated concepts (negative conjunction).

3. 3. Type A “Elaborating on the initial concept” corresponds to behavior whereby only easy paths are explored by experts, and the concepts that are generated are directly related to the initial idea on an easy path, referring to existing or easily implementable solutions. In this type of reasoning, it is possible that some solutions are generated using a difficult path. However, these solutions always result in a negative conjunction.

4. 4. Type B “Discovering a new path” has a unique structure in that the initial concept (C0) does not remain at the top of the concept tree and corresponds to a negative conjunction. This means that the expert reject the initial idea and prefer to generate alternative concepts by approaching the problem from a new angle, which requires different knowledge.

5. 5. Type C “Searching for other alternatives” is the mirror opposite of type A. Here, the expert evaluators explore only difficult paths, with no referral to existing solutions but an acknowledgment of the complexity of implementation. If concepts on an easy path are generated, they are immediately rejected.

6. 6. Type D “Exploring all possibilities” involves an approach whereby both easy and difficult paths are explored. The unique feature of this type of reasoning is broad exploration without any negative conjunction. The C0 can be either at the top of the tree or on a branch of the easy path, which can create departitions on the difficult path in response to the need for a broader exploration of different ideas.

By adopting these different types of reasoning, the experts could either close, explore, or even open new paths by which the idea could be improved. Furthermore, we observed that the experts did not limit themselves to one type of reasoning, but adopted different types of reasoning depending on the idea they were evaluating. The distribution of the types of reasoning used by the experts is shown in Table 2.

Table 2. Distribution of different types of reasoning used by the expert evaluators

We were able to further distinguish between different types of behavior the experts exhibited based on the think-aloud protocols. Decision-making behavior, wherein the focus was on understanding and assessing the initial idea concept C0 (Types O and N) occurred in 25% of the cases, while Designer behavior, wherein more emphasis was placed on exploring different alternatives for idea development (Types A, B, C, and D), occurred in 75% of the cases. We observed that the experts were able to apply at least two different types of reasoning, but we saw no evidence of consistent expert behavior, suggesting that they engaged in whatever type of reasoning was appropriate for the idea they were evaluating. Given the high number of cases where the experts engaged in design activities during idea evaluation, this provides additional support for our second proposition.

In the final step of our analysis, we were able to distinguish between different roles that the experts played during their engagement with the ideas (see Table 3). When the experts acted as decision-makers or gatekeepers (applying reasoning types O or N), they closed all alternative paths of idea development, which meant that they did not leave an option for a designer to continue exploration of their idea after the evaluation. In doing so, they left no degree of freedom in relation to how the idea could be developed. However, when the experts acted as designers in and of themselves, they could fixate on the initial idea concept (C0) and pursue an easy path using Type A or D reasoning, which involved elaborating on the idea with less design effort, and little use of different knowledge domains to reframe the problem. In doing so, they left a degree of freedom on an easy path and searched for a means of rapid implementation. They could also defixate from the initial idea and open up a difficult path using type B, C, or D reasoning, which required a significant design effort to develop the idea in a creative way. In doing so, they tried to either begin with defixation by changing paths using type B reasoning (strongly constrained defixation), or promote defixation by continuing the exploration on difficult paths using type C or D reasoning (weakly constrained or unconstrained defixation, respectively), leaving a degree of freedom for difficult paths. These findings provide further support for our third proposition that experts can not only promote both fixation and defixation when developing ideas during evaluation but also contribute to a much more nuanced understanding of how this happens.

Table 3. The different types of reasoning used during idea evaluation

We also found that the experts tended to evaluate the ideas in a certain way based on how they engaged with the design and the degrees of freedom they left for the idea. For instance, they could have a strong opinion about the high short-term and low long-term value of an idea, which meant that they believed that the idea could be implemented quickly, but that it required development along the easy path, which corresponded to type A “Elaborating on the initial concept” reasoning. They could also conclude that the initial idea was bad from both the short- and long-term perspectives, and point to the necessity of defixation and pursuing a different path, which corresponded to type B “Discovering a new path” reasoning. They could also see the high long-term value of the idea and encourage its development along the difficult path, expressing the need for defixation, which corresponded to type C “Searching for other alternatives” reasoning. Finally, they could experience a high degree of freedom in terms of idea development, realising that the idea had value from both the short- and long-term perspectives, and that it could be developed in different ways but could also give rise to new ideas, which corresponded to type D “Exploring all possibilities” reasoning.

Based on this analysis, we observed that some experts acted as decision makers, but all of the experts were able to adopt either defixating or fixating behaviors depending on the ideas they evaluated. Thus, it was possible for all of the experts to defixate themselves from idea evaluation as a decision gate and move toward idea evaluation as a design process. Additionally, type D “Exploring all possibilities” reasoning revealed that it was also possible for experts to switch between defixating and fixating behavior in relation to the same idea.

5. Discussion

The aim of this study was to better understand how experts engage in creative idea development activities during the idea evaluation process. Our findings revealed that experts perceive ideas in different ways, are frequently engaging in idea development activities during evaluation, and can exhibit different types of reasoning and intentions in identifying ways in which ideas can be developed further. In particular, we identified three different roles that the experts moved between during evaluation: decision-maker, designer managing fixation, and designer managing defixation. This means that experts are able to move beyond their role as gatekeeper during idea evaluation and act as designers who can either focus on the development of the initial idea or see beyond the initial idea concept. This is consistent with studies showing that engagement in creative idea development activities can help the evaluator to understand the full potential of an idea (Sukhov et al. 2021; van Broekhoven et al., 2021).

We also observed that during the evaluation process, experts could embrace fixation to the initial idea and suggest incremental improvements, but they could also transform the idea by gradually changing some of its components and realising new opportunities for idea development. This is in line with the findings of Purcell & Gero (1996), who suggested that fixation is linked to the specific knowledge possessed by experts. The way in which experts activate this knowledge allows them to either design improvements while remaining fixated on the initial idea concept, or to go further than the initial idea by generating new alternatives and opening up new paths through the mobilisation of new knowledge. Hence, the introduction of new knowledge during evaluation can help with defixation from the idea concept and enhancing its creativity.

Our findings also provide evidence supporting Sosa’s (2019) accretion view of ideation, whereby ideas emerge from interactions with different ideas or idea fragments. Specifically, because of the significant differences in idea evaluation ratings among the experts, we were able to confirm Sosa’s (2019) notion that early ideas can mean different things to different experts, despite them having similar expertise. When comparing the different types of reasoning used during evaluation, it became clear that early ideas might hold different meanings for a single designer. Hence, we further suggest that understanding idea evaluation as a design process can help in focusing on the fluency and flexibility of this process, rather than fixating on the need to find the best ideas.

Thus, viewing idea evaluation as a design process shifts our understanding of the role of experts from one of decision-makers to one of designers managing fixation and defixation. Additionally, given the proactive behavior exhibited during evaluation, the experts can also be viewed as creative leaders (Mainemelis et al. 2015) who engage with ideas and envision how they can be developed to achieve a creative outcome. This means that the types of reasoning used by experts during evaluation and their intentions can be understood in a new light. Fixation type A “Elaborating on the initial concept” reasoning shows the willingness of the experts to launch a particular project quickly without wasting additional time on exploring other paths. Strongly Constrained Defixating type B “Discovering a new path” reasoning corresponds to an orientation toward future exploration of the paths chosen by the expert (which do not have to be the fixation paths and can be specific defixation paths). Weakly Constrained Defixating type C “Searching for other alternatives” reasoning corresponds to a willingness to continue exploration, which encourages defixation without setting a particular path to follow. Finally, Unconstrained Defixating type D “Exploring all possibilities” reasoning leaves the future exploration of the idea free in terms of both the fixation and defixation paths. This could indicate that experts acting as leaders would search for a portfolio of projects with different time horizons rather than focusing on a particular project in relation to the current idea.

This study has revealed several key insights into the experts’ behavior and their roles during idea evaluation. To reduce fixation bias during idea evaluation and engage experts in creative idea development, it is important to view idea evaluation as a design process. This can help to increase the level of engagement with ideas and better utilise the experts’ knowledge, expertise, and creativity if those objectives are considered important in the organisational context. Additionally, by making their reasoning explicit during evaluation, new paths for exploration might emerge. Furthermore, by identifying the different types of reasoning used during idea evaluation and outlining different types of behaviors, the experts can be shown the various ways in which they can engage with ideas, and managers can foster an environment in which experts can explore different perspectives. Engaging with ideas using fixating or defixating behavior can help people evaluating ideas to increase creativity during idea development and selection, which is also in line with existing research (e.g., Agogué et al. 2013; Gillier & Bayus 2022). Finally, given the substantial differences in how experts perceive ideas, it is important to make idea evaluation more explicit in order to better capture the experts’ knowledge and share their unique insights with others. The introduction of new knowledge and different perspectives can help to create new ideas that are valuable to the organisation.

This study provided a basis for combining different theoretical perspectives and analytical steps to better understand how ideas emerge through the gradual development of different idea fragments. By combining the framework of an idea for innovation (Sukhov et al. 2019) with C-K theory (Hatchuel & Weil 2003, 2009), we were able to trace the emergence of new ideas and understand how experts navigate during idea evaluation. Although this was a rather consuming analytical work, we believe that our study was able to shed light on a complex empirical phenomenon and explain how idea fragments are combined into layers of different narratives, and how experts are able to elaborate and transform ideas. A potential research implication is to further expand this framework to make tracing of idea development through its fragments more straightforward.

6. Limitations and future research

In this study, we identified six types of design evaluation reasoning using a small sample of participants and a limited number of ideas. We also chose a particular theoretical framework that guided our analysis. Although this study has some limitations in terms of the generalizability and robustness of the results, we argue that undertaking this study has provided us with a unique opportunity to understand idea evaluation as a design process in a real-life setting while performing an in-depth analysis of the process as it unfolds. Thus, future research could investigate the recurrence of these types of reasoning using a larger sample and different study settings to obtain a deeper understanding of the process of idea development during evaluation. Additionally, while our data related to individual experts evaluating ideas, organisations using different approaches and forms of leadership might have several people interacting during a joint idea-evaluation process. Therefore, future research could explore the dynamics of multiple evaluators and whether the results obtained in such a scenario complement our findings. Finally, our findings could lead to further research on how to undertake idea evaluation as a design process, and an investigation of the role of different instructions on participants’ behavior during evaluation.