Supporting designers: moving from method menagerie to method ecosystem

Abstract

Supporting designers is one of the main motivations for design research. However, there is an ongoing debate about the ability of design research to transfer its results, which are often provided in form of design methods, into practice. This article takes the position that the transfer of design methods alone is not an appropriate indicator for assessing the impact of design research by discussing alternative pathways for impacting design practice. Impact is created by different means – first of all through the students that are trained based on the research results including design methods and tools and by the systematic way of thinking they acquired that comes along with being involved with research in this area. Despite having a considerable impact on practice, this article takes the position that the transfer of methods can be improved by moving from cultivating method menageries to facilitating the evolution of method ecosystems. It explains what is understood by a method ecosystem and discusses implications for developing future design methods and for improving existing methods. This paper takes the position that efforts on improving and maturing existing design methods should be raised to satisfy the needs of designers and to truly support them.

1. Introduction

One of the stated purposes of design research is to provide support for industry for designing better products in a more effective and efficient way, for example, by studying designers, teams, organizations, or users as well as technologies, products or systems (Horvath 2004; Blessing & Chakrabarti 2009; Tomiyama et al. 2009; Reich 2010; Andreasen 2011; Braha et al. 2013; Cross 2018).

While there are many examples of such support in form of methods, tools, guidelines or processes that has had a direct and significant impact on individual companies, it has proven to be more challenging to evidence the contribution of design methods and other types of design support to industry at large (Daalhuizen 2014; Jagtap et al. 2014; Gericke, Kramer & Roschuni 2016; Cross 2018).

Much research on methods has not yet had a lasting effect on the wider industry (Wallace 2011). One of the vehicles of affecting industry beyond a specific collaboration with partners is through the development of methods that industry can follow. Industry does work undeniably in a systematic way (Daalhuizen 2014), but more evidence is required to show that they follow methods proposed by the academic engineering design research community (Tomiyama et al. 2009). Once companies have adopted a method, they often use it for a long time, but sometimes for purposes that it is not intended for (Gericke et al. 2016).

This position paper argues that one of the roots of the problem is that universities develop a plethora of methods in isolation, rather than offering industry methods and tools that they can adopt and adapt to their context and fit into their existing set of methods.

Universities and businesses have different goals and operate at different time scales. Universities traditionally aim to excel through rigorous research, which contributes to knowledge and are tied into academic funding and degree cycles, whereas businesses aim for productivity to assure a profit through the timely delivery of high-quality products. However, academia is increasingly assessed through the impact of their research on business (primarily) or society. Therefore, the question of how method development can make a contribution to industry is becoming more pertinent.

The title of the paper alludes to the current state of method development through academic engineering design research. This development is akin to the menageries before modern zoo keeping was developed. Exotic animals like lions or rhinoceroses were kept in unsuitable cages and displayed to the royal visitors or the general public. Many methods are similarly displayed at conferences or in publications and forgotten soon afterwards. Instead, what we should aim for is an ecosystem of methods that coevolves in an impactful way within the industrial application environment. Some methodologies that have been developed over the years (such as Pugh 1991; Roozenburg & Eekels 1995; French 1999; Andreasen & Hein 2000; Frey & Dym 2006; Pahl et al. 2007; Cross 2008; Ullman 2010; Vajna 2014 and others) describe a consistent set of methods and thus could be considered as an ecosystem. It should be noted that the term design methodology is differently used within the design research community. Roozenburg & Eekels (1995) discuss the differences. The main difference is between understanding design methodology as ‘the study of methods used in a particular discipline’ compared to understanding used here (compare Table 1) – ‘a specification of an overarching approach to producing an artefact that specifies what the different activities are, what methods should be used to perform them, how to sequence them, what their information outputs should be, and (frequently) how to describe the information produced at each stage’ (Gericke, Eckert & Stacey 2017). However, design methodologies assume an idealized environment suitable to support design education, but do not (and cannot) describe the large variety of industrial contexts. These works represent artificial ecosystems. In this paper, a method ecosystem is understood as a set of methods that can coexist in a self-sustaining way. We distinguish between such ecosystems that can be found in practice, which often evolved over time and ecosystems that are used primarily for teaching. We refer to the latter ones as artificial ecosystems. Some are based on observation of good practice (i.e., descriptive approaches) while others are prescriptive and provide usually generalized guidance for designers and students (Blessing 1996; Gericke & Blessing 2011, 2012; Gericke, Qureshi & Blessing 2013; Wynn & Clarkson 2018). As a conclusion, all design methodologies are ecosystems but not all ecosystems are necessarily design methodologies.

Table 1. Examples of different types of design support from Gericke et al. (2017)

This critique does not want to diminish the usefulness of these works but aims to motivate the further development of the field of design research.

Design methods are embedded in a complex environment. They influence and are influenced, for example, by the task, the design process, the individual prerequisites of the designer, prerequisites of the group and external conditions (Frankenberger & Badke-Schaub 1998). The integration of methods into the process is twofold, into the sequence of tasks in which they are used and with each other. This implies a common vocabulary or at least an explicit vocabulary, which allows a translation and models that can be linked up. The need for the integration, or at least the possibility of integration, of methods might be greater than ever before, as products are becoming more complex and interdisciplinary, companies aim to deliver holistic user experiences based on new business models and the development processes are becoming more global (Nguyen, Müller & Stark 2013; Kimita et al. 2015; Eckert et al. 2019; Wichmann, Eisenbart & Gericke 2019). For example, Industry 4.0 will require a greater integration between software, hardware and electronics. Sensors on the product and its production lines as well as more powerful simulations will generate unprecedented volumes of data. Principles of circularity and sustainability will become commercial and moral imperative (Tukker 2015; Man & Strandhagen 2017; Cong, Zhao & Sutherland 2019; Wichmann et al. 2019). All this requires greater transparency in the ways of working and greater collaboration with other fields; and therefore, engagement with their tools and methods and ideally a coordination across different disciplines to assure that gaps and incompatibilities do not cause inefficiencies and product failures. Hence, the design research community needs to be ready to engage and gain an understanding of how to support industry through methods that are fit for the evolving industrial practice. The current problems and future challenges raise the question whether the gap between engineering design practice and engineering design research is actually increasing rather than narrowing at a time when the integration of disciplines in products require coherence within disciplines, that is, the disciplines can collaborate in a logical and consistent way using mutually understood concepts and representations. We therefore postulate that the community needs to urgently step up its efforts.

Many design methods are developed as part of PhDs or time-limited research projects. As such, they play an import role in training the next generation of engineering design academics and industry experts (National Academy of Engineering 2004; Eder 2007; Blessing & Chakrabarti 2009; Tomiyama et al. 2009; Cross 2018). Publications on methods play an important part in building our community through discussion at conferences and in journals. When time and funding runs out, many of these methods or tools are not developed to a point, where they can be picked up by industry or other academics (Gericke, Meißner & Paetzold 2013). Industry can benefit greatly from being part of these research efforts and from hiring people with expertise in methods.

However, for our research community to have a greater impact in industry, we argue that we need to create systems of usable methods that engineering designers in industry can use. This does not imply that all methods need to be connected or that the research community needs to reach a consensus on one common way of looking at design research. Rather the paper argues that methods need to be developed to a sufficient degree that industry can use them in conjunction with existing methods without the creators of the methods being actively involved; and that academic publications explicitly discuss how proposed methods can be used in conjunction with other methods. We should strive to improve and extend existing methods to give industry continuity, rather than researchers addressing limitations by starting from scratch to develop a ‘new’ method. Similar arguments apply to the need to create joint up and compatible tools, however this paper will focus on methods.

This position paper is a collective effort by members of the Design Process Special Interest Group (formerly known as Modelling and Management of Engineering Processes – MMEP) of the Design Society. It has arisen from a series of workshops at the International Design Conference DESIGN’16 (>30 participants) and DESIGN’18 (>40 participants) as well as two specially convened workshops in October 2016 (15 participants) and in November 2018 (12 participants).

The authors of this position paper represent a diverse community – representing different nationalities (Germany, UK, Canada, Sweden, Italy, Greece), different communities (Design Society and ASME with backgrounds in Engineering Design, Systems Engineering, Design Optimization, Computational Design, Design Theory) and different career paths (one currently in industry, four with industrial background and currently in academia and six in academia). Most of the authors are serving as editors or reviewers for leading academic journals, conferences and funding bodies. All the authors have been engaged in developing methods for up to 25 years and have had both successes and failures in introducing their own methods into industry and have had numerous discussions with industry experts on the barriers of introduction. From this experience, the paper is making claims about the challenges of introducing methods in industry, that have resonated with all the authors.

The authors developed the position of this paper together starting with 2 days of discussions at the workshop in November 2018. The main insights were summarized in this paper, which was improved through several rounds of comments and rewriting. This position paper is not a review paper. As a position paper, it is intended to stimulate discussions about the addressed topic, related challenges and of course about the position presented here.

2. Design methods

2.1. What is a method anyway

Building on Blessing & Chakrabarti (2009), Gericke et al. (2017) set out to clarify some of the central concepts of design research (see Table 1) with which the term method is often mixed up.

A design methodology is an approach that combines methods, guidelines and tools, each of which can exist individually, according to a process that organizes design activities, and the use of the methods and tools. The application of methods and guidelines, and the organization and performance of the process, can be aided or enabled by the use of tools.

A method has multiple elements, which comprise the core idea of the method, representations in which design information is described, and a procedure. Core idea, representation and procedure build on each other (see Figure 1) and form the method, thus the method description should provide the necessary information about each element of the method as well as information about any tool implementation of the method if available or required. A method might have dedicated tools, shared tools with other methods or use generic tools (see Gericke et al. 2017).

Figure 1. Elements of a method from Gericke et al. (2017).

The method description should provide, besides explanations of each element of the method, information about possible adaptations of representations and procedures that allow the method’s use in different contexts, as well as information about the required rigour in the application of the method. Some elements of a method might allow adaptation while other elements, for example, those required for or related to compliance, should not be modified. For example, the same method could use alternative representations, such as using graphs instead of matrices. Method users should be informed about such options and limits of adaptation.

Dorst (2008) has criticized engineering research for concentrating on the activities required to carry out a task and therefore focusing on efficiency and effectiveness, while neglecting the object, the actor and the context in which these activities are carried out, however we see these an integral part of design methods where the actor is considered in the procedure and the context and the product in the intended use.

2.2. Current impact and relevance of methods in industry

The development of new means that support designers in their work (e.g., methods, tools, guidelines, processes and methodologies) is central to engineering design research (Blessing & Chakrabarti 2009). Analyses of method uptake by industry create a contradictory picture. While it is repeatedly stated that industry does not seem to use design methods (Araujo et al. 1996; Birkhofer et al. 2002; Geis et al. 2008; Tomiyama et al. 2009; Jagtap et al. 2014), many companies claim that design methods are central to their activities and enable them to be innovators in their field (Maylor 2001; Design Council 2007; Booker 2012) as evidenced by the many publications by industry experts in conferences and by the activities of professional bodies.

The validation of design methods in relation to the industrial context of deployment is problematic at best. Engineering design researchers refer to the Validation Square (Pedersen et al. 2000) as a reference framework for the validation process of design methods. The strength of this framework is that the evaluation establishes both efficiency and effectiveness of the design methods, by considering both theoretical and empirical validity criteria. Within the context of this framework, the fundamental limitation faced by academic engineering design researchers is with the ‘external validation’, that is, the reproducibility of the validation experiments results within the users’ environment, proving the utility when industrialists deploy the design methods. This invariably limits or delays the take-up of the design methods in industry, where in the face of commercial pressures proven methods are commonly preferred.

Assessing the dissemination and uptake of design methods is difficult, as companies may use methods in a modified form and may use different names for the methods they use (López-Mesa & Bylund 2010). Gericke et al. (2016) report that many of the practitioners interviewed for their study did not know the academic names of methods they use and many were not aware that they were working in a structured manner and were in fact applying a version of an existing method that they had come across in the past. Engineering design processes include methods, but there can be a complicated relationship between the methods actually used and the published versions of the methods and to designers’ perceptions of the methods.

However, measuring the impact of an academic research goes beyond an assessment of the direct uptake of its research results. Design researchers are domain experts that influence design practice by proposing, by knowing, and by teaching. Design research results are often based on analyses of practices and design problems of industry, thus provide analyses and propose good practices that have been shown to be successful. Research results influence the education of the next generation of experts as well as it influenced current and past generations. In this way, many of the underlying concepts of design methods and methodologies indirectly influence design practice (Eckert & Clarkson 2005; Cross 2018), even though this influence is slow and hardly traceable and measurable.

This research community impacts design practice by the direct transfer and uptake of research results, as well as through the highly qualified engineers (BSc, MSc and PhD) they train. Over the last decades design practice has undergone tremendous changes and many of these changes were enabled by research results, driven by research results or built on research results. For example, Quality Function Deployment (QFD), SixSigma, Failure Mode and Effects Analysis (FMEA) and many of the CAD tools started in early engineering design research (Hein 1994; Cross 2018).

While the assessment of the impact should not be reduced to a single measure – the uptake of methods, tools etc. by industry – it certainly is an important channel that deserves continuous reflection, improvement and adoption to new trends and needs.

2.3. Key challenges for improving the uptake of methods

The development of methods that industry can use is fraught with difficulties on multiple levels arising from both the way academia and industry work and from the way the two groups interact with each other.

A significant part of the research community lacks awareness of the plurality of methods and the implications for both the industrial application of our collective research and our academic credibility. The consequences of this phenomenon have been discussed for research around function modelling, where a multitude of internally consistent notions and resulting methods have been proposed but hampered their uptake because of their coexistence. Vermaas & Eckert (2013) state the problem clearly: ‘The coexistence of these different traditions is now hampering further developments and usages of functional description in academia and industry. At conferences, new results and applications of functional descriptions are presented, creating progress within the separate traditions but limiting opportunities for cross-fertilizations. In the dissemination of results to industry, academia effectively exports its separation in traditions, thus arriving at the less attractive proposition that industry should adopt the different ways of giving functional description and implement methods and tools that are not straightforwardly combinable’.

Methods for industry need to be built on a strong understanding of industrial practice; but this is not enough, we also need evidence that support our claims of improvement, that is, examples and evaluations that are based on problems that truly match challenges in industry, which go beyond the typically used toy problems. Managers in industry want to see real industrial examples and see measure of increased efficiency.

2.3.1. Causality

Claims behind the success and failure of methods in industry are assuming that there is a causal connection between the method and the results that arise from it. However, arguing causality is problematic for the following reasons:

Multicausality: No two design problems are the same and all design processes are subject to multiple constraints and characteristics that affect them. The success or failure of both products and processes can be due to many factors. This makes it difficult to attribute any improvements directly to the methods that are used. If the new generation of a product was designed faster using a new method, than the last generation, this might be due to the method, but it could also arise from a different amount of required change or different people. Conversely, a project might struggle in spite of good methods, because it is running in unexpected problems such as cliff edge effects in the product. An inappropriately chosen or used method can also have a negative impact on quality or lead time.

Hawthorne effect: The Hawthorne effect was originally discovered in the 1930s when a team of researchers attempted to change the working conditions in a factory. When they change the set up back to its original state after multiple modifications while achieving continued efficiency gains, they realized that the improvements they achieved were not due to the change they made, but the attention they have paid to the factory, the process and its workers.

Notions of causality: In evaluating methods, we need to create a causal connection between the methods and the effect it has for industry. This raises the question: what an appropriate notion of causality would be for methods? A typical notion would be counterfactuals (ref), that is, if A leads to B, then not A would also lead to not B. While this is appropriate in the context of risk and failure, it is unlikely to be possible to prove, that without a method a product would not have been successful. Therefore, it might be more appropriate to think of a causal push, that is, A makes B more likely. Translated to methods, this means that applying a method makes it more likely that something is achieved. However, if the method is associated with an improvement it is it difficult to prove that this can be attributed to the method. If no improvement occurs, it does not mean the method has failed, as other factors could affect the measure.

2.3.2. Academia

The way design research is operating under the pressure of academia also contributes to the challenges of introducing methods into industry. To a certain extend this might be a matter of perception as many publications claim that methods have been developed, before they have been successfully applied or tested in an industrial context (see, e.g., Pedersen et al. 2000 for the discussion on how to validate methods).

Lack of theory: In spite of isolated efforts, design research still does not have comprehensive theoretical underpinnings that enable us to predict how design processes behave, given the nature of the product being designed and the way the organization and the design process are configured. This deprives us of a theoretical means to assess the scope of methods and to support the generalization of processes or best practice into generally applicable methods.

Premature publishing: The publish-or-perish culture of academia pushes our community publishing of methods before they have been properly developed. A publication it is often the promise of a usable method, when in fact it only expresses the core idea of the method, maybe with an application example rather than a fully developed and evaluated method. Many ideas for methods are developed as part of doctoral theses. As an individual student can rarely develop and evaluate a method in its entirety methods are often published in a prematurely stage. If this student moves on to industry or loses the opportunity or interest to develop the method further, it never matures. However, publishing methods that are under development is an essential part of academic discourse.

Knowledge islands: Methods and entire approaches to engineering are often developed in response to the specific challenges the collaborating companies are facing, or in response to specific and narrow new technology development. This has contributed to different research groups or cluster of research groups having developed their own approach, view and terminology around design as the plethora of definitions and methods for functional modelling illustrates (Eisenbart, Gericke & Blessing 2013; Vermaas 2013). This would not be a problem, if work would build on each other or clearly articulate the differences. Attempts to compare and benchmark different methods and approaches against each other is still in its infancy. A special issue on benchmarking functional modelling illustrated that most authors thought of benchmarking more in terms of increased citation than a deeper comparison, which the editor still welcomed (Bohm et al. 2017) rather than an analysis of the strengths and weaknesses of their method compared to others.

Disciplinary silos: Academic research on methods for engineering design and product development still happens in narrow disciplinary boundaries and cross disciplinary collaboration occurs rarely even though for example the computer science, operations research and the technology management community work on related issues and use similar methods with different names. To create ecosystems of methods researchers from different fields need to come together around a shared understanding of the industrial context and its needs.

Lack of focus on validation: Much of design research in academia tends to focus on development or refinement of new methods with an emphasis on theoretical structural aspects, rather than the empirical validation of the methods in a real engineering and industrial context. This is also a reflection of the historic self-centric attitude in academic publishing of engineering design research, in that higher value tends to be attributed to rigorous papers that present an innovation or structural enhancement of a method, rather than empirical deployment enhancement studies, positioned in an external industrial context and focused on the social aspects of method enhancement and deployment.

2.3.3. Understanding of industrial practice

Academia and industry collaborate in many different ways. Individual researchers or research groups have often found their own ways of working with partner companies from case studies, student projects and funded projects to consultancy or exchange of stuff. However, the number of companies each individual or group can work with is limited and direct comparisons between competing companies is often infeasible or unethical. Nevertheless, there are some common issues around generating and sharing knowledge about industrial practise.

Publishing descriptions of practice: Engineering design is usually carried out in large teams. The development of highly complex products, such as aircraft involve 10,000s of people across the supply chain as does the development of the systems of systems. This makes it impossible for any individual or group to fully understand the challenges the product development processes face and the product development contexts in their entirety. However, even applied to narrow areas, few researchers understand industrial practice in particular beyond individual companies. Sharing understanding from practice can be difficult since publishing purely descriptive papers can be a challenge as reviewers demand methods or multiple case studies. Conversely industry papers that report on developments of methods or implementation of the methods are often considered less rigorous and difficult to publish.

Understanding differences between practitioners: Localized work culture plays a huge role in success and failure of methods. Companies are organized in different ways and often reorganize their structure and management. This can render insights outdated or irrelevant. Individual teams are motivated in different ways and respond to the introduction of a method based on past experiences. The time of method introduction can be critical since teams are very receptive, if the method addresses a problem they are currently faced with.

Academics from industry: Some academics have been working in industry before joining academia. This can be an enormous advantage provided the academics also have the methodological training and experience to conduct rigorous research; however, it can also be a source of bias. Even with an industrial background it is important that academics get exposure to multiple companies and sectors in particular, if they have spent their entire professional live in one company. The balance between academics with industry or academic backgrounds vary enormously between countries, which has generated research subcultures with slightly divergent objectives.

Funding for industry research: Government funding councils tend to be highly directive and tend to look for innovative research and often new technology. It can be a challenge to get method research funding in particular up to a level when methods would be fully described and validated. Many research groups obtain funding directly from industry. However, in this case the funders’ interests lie in addressing their own problems, rather than assuring a general applicability of methods, even if the results are presented to the academic community in those terms.

Trust between industry and academia: Method development requires sustained funding and long-term commitment by both industry and academia. This requires long-term personal relationships and an understanding by both parties that on route to robust methods and tools the collaboration will bring benefits for both parties, such as feedback on existing processes or activities in the companies by academics as part of their research. The necessity of personal relationships creates uncertainty for said long-term collaboration, in that the consequences of key stakeholders leaving might endanger such cooperation. As such, formalization efforts of collaboration might be one way to go, for example, establishing industry-academia research councils which manages and sustains such collaborations.

3. From isolated methods to a method ecosystem

This paper takes the position that instead of proposing isolated invalidated methods that are insufficiently evaluated for a wide-range of applications we need to reach a point where methods are robust, have a clearly defined scope and are embedded in an ecosystem of methods, that is, a set of methods that can coexist in a self-sustaining way.

Originally the term ecosystem was applied to ‘a biological system composed of all the organisms found in a particular physical environment, interacting with each other; in extended use: a complex system resembling this’ (Oxforddictionaries 2017). Over the past three decades, the term has been used for the increasingly complex integration of organizations, humans, materials and information flows across the product lifecycles. Based on a comprehensive review of literature, Tsujimoto et al. (2018) have defined the objective of the ecosystem in the field of management of technology and innovation as ‘To provide a product/service system, a historically self-organized or managerially designed multilayer social network consists of actors that have different attributes, decision principles, and beliefs’. In this sense, every product development process can be thought of as an ecosystem of its own. However, for the purpose of this paper, we think of methods and the tools that support them as an ecosystem.

3.1. ‘Paradise’ scenario

Before describing what best practice on method development could be, let us look what an ideal state of methods would be. To provide another analogy let us think of methods as tools in a builder’s toolbox. Novices have to learn to use the tools. When they are asked to cut a stone to a particular shape, they have to select the right tool and then focus their attention on how to use the tool to do tricky tasks, such as cutting out neat corners. Master builders understand and master their tools. They know where and in what situation to deploy a particular tool. They understand the sequence of the activities that need to be carried out and therefore knows when which tool needs to be available. Master builders are not focussed on the use of tool but concentrate on what they are building. The master can concentrate for example on the shape they are generating and what angle the corner should have. A master can generate the shape they want, not the shape they can create given their understanding of the tools. The reflection in action is a well-recognized aspect of design (Schön 1983). However, the reflection should be mainly on the object that is being designed, rather than on the designer’s ability to use the tools by which the work is being generated.

The master builder has some specialist tools, which enable him to carry out a specific recurring task very efficiently, such as moulds for particular shapes and general tools, where they need to think how the tools are applied to a given task. Master builders have a differentiated understanding which tool is appropriate.

The ideal scenario of methods would be an equally smooth interaction with multiple targeted methods so that the designers can really focus on what they are designing. The designers would also select their methods with ease and rely on the methods to deliver what they need. As a multitude of methods is used during the development process, these would fit together so that no time and effort is lost is divergent vocabularies and logics of modelling. The designers would also adopt new or enhanced methods to avoid crisis from developing. Design methods should not become invisible and should not require zero mental effort. Design methods should not make the work more complicated. A method can require attention and can require even substantial mental effort; however, they should not increase the effort and not distract from the work that needs to be done.

In short, in paradise, methods are unobtrusive and dynamic in the same way as to that of a pen lying on a notepad during a meeting. Invaluable and invisible, with only its absence marking its importance.

Like the master builders the engineering designers would have invested time and effort into learning their methods and deploy them on multiple projects so that they can be masters. At the same time, they would be open to invest time to adopting and integrating improved and new methods provided these offer a clearly perceived benefit. Academics would understand the needs of industry and direct their efforts to newly arising challenges and desires of industry. In dialogue and sympathy, they would develop new methods before their lack becomes a real issue, for example, the tools and methods to make best use of new technology would be in place.

3.2. We must be able to measure impact of methods

If we want industry to take up the methods developed by our community, we need to give them the confidence that the methods are ready to be deployed and add value to their operations. There we must be able to measure the effects that are claimed, such as effectiveness and efficiency or the time it takes to master the methods. This implies that it is necessary to clearly state the expected effects.

While there are theoretical challenges to measuring the effectiveness of methods, there are pragmatic actions we can take in order to assess methods, such as interviewing the participants, running evaluation questionnaires or doing a detailed comparison with other projects.

Like with products, changes and adaptation might be necessary once the method is actually used. Some companies already employ teams to select and monitor methods – ‘governance’ of processes and methods. However, if these teams are outside of the departments, they might be too distant from the actual process to assess it as they can neither observe the process or access the process data; and they might be ignored or resented by the practitioners. As a community, we need to work with these people and bring them into our community so that we can all build on their experiences. This has for example been accomplished successfully by the DSM community, that runs annual conferences with high industry participation.

The expected effects of using design methods can vary from being more efficient to being more creative or simply being able to achieve something that is usually too complicated or too complex to attempt, thus being more effective. An important part of effectiveness is also to have teams enjoy the work that they are doing more and freeing them up to be more creative and innovative. This indirect link between systematic design methods and innovation is often overlooked. Many methods are of course directly targeting creativity.

A system of measuring the effectiveness of methods carries risks for the designers who are using the methods and the research who generate them. As in other walks of live any measure carries with it the risk of being gamed. Instead of working towards the ultimate goal, the entities work towards maximizing their scores. Measuring also carries with it the risk of the Hawthorn effect. Measuring the effects of a method is fundamentally measuring the performance of the people which needs to be handled with care.

A change of culture in organizations is required to overcome some of these challenges. The introduction of methods needs to embrace the organization and communicate the rationale for methods. Rather than giving employees the feeling that they are measured, the results created through the method could be assessed and designers could become incentivized to improve and adapt methods.

At the same time, we need to be able as academics to evaluate the effect that methods are having in practice. While related to the performance measures industry would use, the academic criteria of improvement could be wider or more qualitative.

3.3. Research of practice and practicing informs the development of methods

The applicability of the design methods depends on the intended context. As part of the development of methods, we therefore need to aim to understand this context. It is of course not practically possible for researchers to try out a method in a large number of different context (Gericke et al. 2013).

The first step has to be to avoid overclaiming the area of application of methods. In research publications, we need to honestly report in which context a method has been deployed: what was the product? Was the method used in a real environment? What simplifying assumptions have been used? Many methods developed by researchers have been used only on a toy problem. What works for a mouse trap might not scale up to a helicopter. However, what we can do is to characterize the properties of the application case as accurately as possible and reflect over how these characteristics of the problem or organizational context have affected the success of the method. For example, a method that depends on a product Design Structure Matrix (DSM) in early stages of the product development process, like Change Prediction Method (CPM) (Clarkson, Simons & Eckert 2004), can work in the context of mature and incremental products. It also only provides benefit for products of a certain complexity, too simple and the method has little benefit, too complex and the product breakdown is either so abstract that vital characteristics are lost or so large that it is difficult to read the matrix.

Only an understanding of practice allows researchers to anticipate at which practical problems users of the method are likely to trip up. For example, an unsuitable visualization can make it very difficult for people to read dependencies. To return to the example of change prediction, a matrix is an excellent and complex way of seeing dependencies, but graphs are much better at seeing paths (Keller, Eckert & Clarkson 2006). The onus of making methods useable in different applications and contexts should lie with the developers of the method.

Practitioners also often abandon methods when they run into small problems that are time consuming to resolve. Therefore, they need guidance on how to deal with these issues. For example, one of the challenging issues when generating change propagation matrices are small components in the product breakdown, that have been overlooked in the past. If methods are presented with carefully described examples on which it has been validated, the implication of the scope of the method can to some extent be left to the imagination of the reader whether they can apply the method in their own context.

The need for a detailed understanding of the context raises the question how this can be instilled in the researchers who develop methods, who are often graduate students. Few books exist that describe engineering design practice (which notable exceptions of sociologically oriented books, e.g., Bucciarelli 1994; Henderson 1999). Therefore, the burden to give students exposure to industrial practice through joint projects or placement lies on the universities and industry collectively. In particular, researchers need to learn and reflect how companies use methods and how the methods are introduced into organizations in order to be able to deploy methods that can be applied themselves. Introducing new methods is a cascading process drawing on complementary skills. Of course, we cannot assume that all researchers have equal access to companies or have equal skills to work with organizations. Therefore, we need to generate an environment where people with different skills and inclinations work together on methods within groups or across universities. For example, in the development of CPM the empirical studies were done by engineers and the algorithms were largely coded by a mathematician who also joint into the empirical studies. The choice of the tools and methods is not always up to an individual organization but might already be prescribed by guidelines or standards which companies might choose or have to adopt to.

3.4. Ecosystem of methods

In this paper an ecosystem of methods is understood as a system of methods embedded in an organization, where methods operate in conjunction and where users implicitly understand how methods can be adapted and how they are connected to each other. This requires a degree of communality in the terminologies used as well as a clear articulation of the input and outputs of methods. Like in an ecosystem each method has its distinct characteristics and purpose, but the methods also overlap in scope and to a certain extend compete with each other. An ecosystem is not a fixed set of methods, but a system in which methods can be added if a need for them arises and multiple methods can fulfil similar roles. An ecosystem does not lead to a stringent recipe of how to proceed but provides and suitable methods and a structure in which the methods can be used. The same goal can be achieved in multiple ways, that is, through different combinations of methods, adaptations of methods and by a flexible/opportunistic choice of when to use a method or not (Bender & Blessing 2004). This choice is important to enable users to tailor their processes to the products and means of production they are addressing and also give them a sense of control over their processes. An ecosystem is therefore far richer and more flexible then a design methodology, which ties a set of methods together in a suggested structure. It provides numerous of methods for different purposes and ways to combine and supplement them.

Methods need to reach a certain degree of maturity over time, which might necessitate research on maturing and improving existing methods. This needs to be recognized as research in its own right and funding must be provided. Method development is not a game of method innovation, but of method maturation. In the context of academic research method improvements therefore need to be articulated and acknowledged as contributions.

As in natural ecosystems there is not only a single ecosystem of methods, but multiple clusters of researchers or communities of practice have their own ecosystems, for example as they belong to the same industry sector, nation or lead academic discipline. There might be some methods that are common in all or at least many different ecosystems, while other methods are highly specialized. The ability to combine is important as companies have their own in-house methods, that they want to combine with methods developed by the research community. Which method ecosystem is the right one and into which a new method is to some extend a matter of choice, but also a matter of suitability as some methods are developed specifically for a particular application.

Ecosystems are open, as researchers and industry experts move between companies. They evolve. At the same time an ecosystem also implies a degree of stability as people become familiar with methods and learn how to deploy methods. Just as animals learn to adapt to their ecosystems, methods also need to be given the chance to evolve to find their own niche in an ecosystem.

3.5. Ecosystem of the research

To achieve an ecosystem of methods we also need to generate a community of practice of researchers and practitioners, this will become even more important in the future as product development processes become more interdisciplinary and therefore the need for cross-disciplinary methods increases. We need to foster the collaboration across research groups, across disciplines and across noncompeting companies so that they can learn from each other (Gericke, Qureshi & Blessing 2013). As academics, we often have the privilege to work with a variety of different companies. As we train them in the use of the methods that we develop we might also be able to bring them together and enable them to exchange ideas and practices of the methods. This exchange is also an opportunity for us to learn how our methods can become more robust and applicable. Ultimately the onus is on us as researchers to create and foster communities of practice around the methods we are generating. Different communities of practice can also learn from each other, so that we build up the knowledge how to create and apply methods more efficiently.

The development of tools and methods requires a range of different skills and involves many different activities from understanding the industry contexts and developing the steps of the methods to the underlying theory development. Elements of method development might involve a different mix of academic disciplines to those that the companies use in deploying the methods. In understanding practice and developing methods we might need to work with natural and social scientists from different fields such as psychologists or sociologists. However, it might be difficult up front to anticipate which disciplines this might be before we have engaged in detail in the process.

It is important to communicate this to the funders who want in depth explanation as well as detailed ethics plans before we even have engaged into the research. Industry equally needs to understand this point, as they might need to provide funding to bridge these gaps.

Industry facing research in methods cannot be conducted in isolation from the development of the underlying design theory as well as the evolving technology. As such, the research must span all the way between pure theory-building and minimal-scope technology application. Therefore, the development of methods and the associated funding must plan this in to grow the field at large.

4. Implications

We expect that this paper and the expressed opinions will cause reactions – agreement as well as disagreement. We hope that this will stimulate a constructive debate on the subject and will help to improve the relevance and impact of our research and ultimately to improve the outcome of engineering work, thus contributing to the better of our society. We need a joint debate about what academia can provide and what industry needs.

The opinions expressed before would require a change of course in the design research community in several ways:

1. (i) We have to open up the design research community further. Given that design practice is expected to become more multidisciplinary and given that design processes (as prescribed as well as executed) have characteristics that are mutually dependent with the product/system that is being developed, the research community needs to actively attract experts from other engineering disciplines as well as other disciplines outside classical engineering. Moreover, we need to attract researchers from different fields and encourage them to use design as an application field for their own challenging questions. At the same time, we need to develop respect for the domain expertise of other fields, such as psychology or computer science, and not assume that design researchers can pick this up easily. Academic societies should more actively reach out to other scientific communities and establish ways to foster the exchange.

2. (ii) We need a better dialogue between the research community and industry, that goes beyond individual researchers working with individual industry experts. The exchange between academia and practitioners – in whatever form – would benefit from a more intensive participation of practitioners in academic events. The research community should develop alternative formats that provide value to its different stakeholders and should evaluate to what extent existing formats need adaptation (without reducing value for its current core-membership). In particular, we need a platform in which academics can exchange case studies and other descriptions of practise, which currently exceed the word limits of same journals and are not seen as sufficient contributions by others.

3. (iii) We need more work on underlying notion and concepts of design to enable us to engage in a dialogue rather than talking past each other using the same words, in a similar way to the scholarly work. Rather than highlighting the common elements, we need to analyze the differences to help us with the assessment of the scope and applicability of methods and to assess the implications of the methods that we are proposing.

4. (iv) We need to develop a code of practice around publications of immature methods, which enables industry to clearly identify well developed methods while allowing an exchange of ideas of our researchers. In particular, we need to encourage journals to value publications on the consolidation of the existing methods and the application of methods, which are currently often rejected because the reviewers see them as not sufficiently novel.

An ecosystem of methods needs communities of practice that have a shared understanding and can work together. We need to move beyond individual initiatives and personal contacts to create networks amongst researchers and industries. There are multiple examples of what has worked to achieve this partially in the past. They discuss tentative ideas, give each other feedback, attend each other’s events and give each other a chance to try out research. It is a role of the academic communities and professional societies to foster these kinds of networks, through events organized by special interest groups, industry participation at conferences or training offered for industry.

This points to another debate which the design research community needs to have around the rigor of design research. As a community we aspire to rigorous research and demand a high degree of validation of our research. However, in practice, many of our publications do not include a validation of the presented work. While this is not necessarily a problem in general, it is one for methods, since it undermines our credibility with industry.

To reach a greater synergy between different methods, it is necessary that academic researchers engage deeply with other proposals and articulate clearly where the similarities and differences and respective advantages lie. Finding a common ground in a first step to developing an ecosystem, as well increasing the academic rigor of the work. In Section 2, we have broken the concept of a method down into its constituent parts: the core idea, the representation, the procedures and the method description.

The core idea of a method (i.e., ‘the basic principle, technique or theory that the method employs’; Gericke et al. 2017) expresses the fundamental take on the problems it addresses, however the development of a method that can really be used requires multiple rounds of refinement of representation, the procedures and description. To get this right so that the methods can be used requires serious and collective effort, which needs to be recognized as research in its own right. Fundamentally, different core ideas are rare. A new one should only be proposed, if it could be thought of as a new paradigm or school of thought. Otherwise, we should acknowledge the common idea and build up a joint body of knowledge. It would be far clearer for industry and other researchers, if new research was presented as a significant advancement of a school of thought rather than yet another way of working.

As a community we therefore have to step up both academic rigour of our work and the depth of engagement with our user community: industry.

5. Conclusion

Design research is impacting design practice which goes beyond the pure uptake of individual methods. Using the uptake of all the methods proposed by academia as a success criterion is too narrow-minded and is unrealistic. As design researchers we advocate an innovation funnel concept for successful product development, where only a small percentage of the initially developed ideas will ever make it to market. The critique regarding the lack of uptake of design methods by industry seems to imply that this metaphor does not apply to the products of design research. When discussing the impact of our research, we need to deliberately manage expectations to avoid fostering a perception of design research, which is detrimental to our ambition to support industry. As a community we succeed if some of our methods make a difference to industry.

The impact of design research is created by different means – first of all through the students that are trained based on the research results including design methods and tools and by the systematic way of thinking they acquired that comes along with being involved with research in this area. The students impact design practice in a slow but sustainable way. The other form of impact comes from the direct transfer of methods to practice.

While training the next generation of engineers is a powerful pathway to impact, it is important to also improve the way research results are transferred directly to industry. Therefore, it is important to accept the realities of industrial practice, such as methods and tools are embedded in an ecosystem of methods. Not all methods are applicable in all circumstances and often methods have to be adapted to the contextual needs of practitioners. Moreover, we need to train students and practitioners to perform such adaptations of the ‘textbook’ versions of the methods we propose.

An improvement of the direct transfer will ultimately allow us to feedback experiences and changing needs into the continuous improvement of methods and training of the next generation of engineers. Such an improvement will make this feedback much faster, thus helps to avoid lagging behind what industry needs. This is a task that requires collaboration of academia and practice as both will benefit from it.

Using feedback from industry as an enabler for a dialogue that informs the continuous improvement of design methods implies that we should question if the ambition to support designers always requires the development of new methods. Maybe, we have enough methods and should instead focus on improving and adapting them. Refining a method is a long journey of many improvements, for which only few researchers have the time, passion and resources. Being able to adapt them according to the context-specific needs of practitioners requires a deep understanding of design practice which we have to develop. What is required is a healthy mix of refinement of existing methods and development of new methods that complement the existing ecosystem of methods.

Improving and developing methods that fit into an ecosystem requires thinking about the whole design process not just the individual design activity that is primarily supported. It requires us to think beyond the individual method to understand its dependencies and interactions with other members of the eco system. Understanding the relationships, a method has within the ecosystem, requires to clearly assess and articulate the scope and impact of each individual method. This, besides other means, will allow us to move from owning a method menagerie to effectively contributing to the evolution of method ecosystems in practice.

As an academic community we need to learn to acknowledge incremental development of methods as a contribution to the body of knowledge of design research. This includes welcoming publications on industrial practice which sets the context for methods and publications on increments of methods.

As an academic community we need to work on the channels for communicating with industry. We have to rethink established channels but also to develop new channels or utilize channels that exist in other fields.