Artificial intelligence and cognitive science in design
Design is a fundamental capability, which is elusive to define and describe, although contemporary considerations commonly include addressing transformation and embodiment. Questionnaires, interviews, and observations are often used to study designers’ cognitions, behaviors, and performance. However, these approaches are subjective and qualitative, rely on how researchers record and interpret the data, and may not provide direct and quantitative evidence for understanding design. The development of neurophysiological and biometric technologies, such as eye-tracking (ET), electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), and heart rate variability, has provided the opportunity to better understand human cognition and behaviors (Borgianni and Maccioni, Reference Borgianni and Maccioni2020).
In design research, progress has been made in applying neurophysiological and biometric measures for studying design cognition and behaviors. For example, Goucher-Lambert et al. (Reference Goucher-Lambert, Moss and Cagan2019) employed fMRI in a design concept generation task to explore the impact of inspirational stimuli on idea generation, and uncovered two distinct brain activation networks showing the differences between idea generation with and without stimuli. Shealy et al. (Reference Shealy, Gero, Ignacio and Song2023) used fNIRS to measure the cognitive and neurocognitive changes of designers thinking aloud and not thinking aloud, and showed that designers who were required to think aloud consume more neurocognitive resources. Howell et al. (Reference Howell, Jackson, Edwards, Kilbourn-Barber, Bliss and Morgan2023) utilized ET glasses to assess the sketching behaviors of designers, and showed different eye gaze patterns between advanced and intermediate sketchers. These studies employing objective measures have provided useful insights to understand the inherent phenomena of design, which is challenging to achieve using conventional methods.
The use of artificial intelligence (AI) has become common and often featured in AI for Engineering Design, Analysis and Manufacturing (AI EDAM) publications since the 1980s when the journal was founded. Many studies have been published in the journal focusing on using AI in design, particularly for varying design activities. This includes supporting idea and concept generation (Luo et al., Reference Luo, Song, Blessing and Wood2018; Sarica et al., Reference Sarica, Song, Luo and Wood2021; Hanifi et al., Reference Hanifi, Chibane, Houssin, Cavallucci and Ghannad2022), concept and product evaluation (Chen et al., Reference Chen, Yu, Chu, Chen and Yu2021), design simulation (Uddin et al., Reference Uddin, Arafat, Kazim, Farhan, Niazi, Hayat, Zeid and Kamarthi2019), data analysis (López et al., Reference López, Moreno, Popa, Hernández-Peñaloza and Álvarez2019), and design automation (Kang et al., Reference Kang, Dy and Kane2023). Recently, large language AI models, such as OpenAI’s GPT-3 and GPT-4, Google’s Gemini, and Meta’s LLaMA, which possess extensive common knowledge and powerful semantic reasoning abilities, have been used to support design. For example, Zhu et al. (Reference Zhu, Zhang and Luo2023) employed and fine-tuned GPT-3 to generate bioinspired designs in natural language forms for supporting concept generation. Zhu and Luo (Reference Zhu and Luo2023) provided several data-driven workflows to utilize GPT models and control their knowledge and reasoning to be used in generating novel and useful design concepts. Chen et al. (Reference Chen, Sun and Han2023) explored the use of DALL·E, a text-to-image model based on GPT-3, for producing creative ideas in pictorial formats, which achieved a similar level to novice (human) designers operating without the aid of AI tools. These studies have shown the remarkable use of AI in enhancing various aspects of the design process, supporting the Industry 4.0 transformation (Luo, Reference Luo2023).
Cognitive science and AI are two core pillars of cutting-edge design research. Cognitive science offers the advantage of uncovering underlying mental processes and mechanisms of humans in design, which provides a better understanding of design and enables the development of better design strategies. AI presents the ability to process and analyze massive amounts of data, which in turn empowers trained AI models to make predictions, uncover patterns, and generate insights in a range of design activities. Although cognitive science and AI in design research focus on different aspects, both research disciplines are aimed at more innovative and efficient design outcomes and processes.
Therefore, it would be beneficial to gather the insights of state-of-the-art research in both cognitive science and AI in design to illuminate promising future directions for research in design. In this thematic collection, we aim to bring cognitive science and AI together, which shed light on an emerging trend of applying AI techniques and neurophysiological and biometric measures to better study, understand, and support human designers in the design process.
Applications of AI and cognitive science in design
This thematic collection includes a variety of conceptual, theoretical, and empirical studies on the advances in applying AI techniques and cognitive science for the analysis and exploration of design.
Hu et al. (Reference Hu, McComb and Goucher-Lambert2023) explored the use of hidden Markov modeling (HMM) to study the patterns in neurocognitive activation data related to design concept generation. HMM is a generative model that uses a probabilistic approach to predict a sequence of unknown variables and has been used in discovering temporal patterns in varying types of design behavior data. fMRI data of participants generating solutions to design problems were collected and then analyzed by using HMM. Twelve distinct states are inferred, with dynamic transitions and activation patterns, linking to varying brain regions and cognitive functions. The states with a higher likelihood of occupancy are more activated in the brain regions, which involve several cognitive functions, such as memory retrieval, visual processing, semantic processing, and executive control. This study shows the potential of using machine learning (ML) techniques in studying design neurocognitive data, which can better describe the brain dynamics in design cognition.
Chiu et al. (Reference Chiu, Lim and Silva2023) used natural language processing (NLP) for visualizing design project team and individual progress in the educational context. Design statements contain useful information about the students’ cognitive process, including both divergence and convergence, throughout the design project, while NLP models could be used to capture certain paths of these mental thoughts by picking up keywords and producing quantitative graphs from textual data. The authors explored several ways of using NLP to measure the students’ mental progress made through in a design course and indicated the use of Word2vec for word embedding and Euclidean distance for measuring similarity seem to be the most appropriate methods for assessing design progress in the educational environment. The study shows the potential of using NLP techniques to capture highly complicated patterns of thoughts that are iterative twisting and turning.
Sun et al. (Reference Sun, Li, Zhou, Lou, Li and Zhang2023a) proposed a framework and a design process for helping designers create user experience (UX) values for ML-enhanced products. ML-enhanced products refer to products that have employed ML to solve complex tasks like humans, such as smart speakers, while UX values refer to the perceptions, emotions, behaviors, physical, and psychological responses offered by a product. However, it is cognitively challenging to add UX values to ML-enhanced products, due to cognitive loads caused by the use of multidisciplinary design knowledge and considerations of unique characteristics of ML. The authors proposed a UX value framework and a co-creating ML-enhanced products’ UX design process to provide guidance for designing ML-enhanced products with ML, stakeholders, and context as co-creators, which helps designers avoid cognitive overload. This study shows the inclusion of designers in participating in the ML lifecycle could better support the generation of growable and transparent design solutions.
Yin et al. (Reference Yin, Zuo and Childs2023) explored the use of EEG to decode cognitive factors in creative processes in the design context for studying creativity-related cognitive factors. Creativity is considered a cognitive process related to cognitive factors such as retrieval, recall, association, and combination. Previous studies have examined such relations qualitatively, but the quantitative relation remains to be determined. The authors developed an EEG-based decoding method, containing seven steps from EEG induction data collection to coefficient calculation, to identify which creativity-related cognitive factors occurred in a creativity process. A case study involving 30 participants was conducted, of which the results indicated that association is the dominant factor for higher creative output quality levels and recall is the dominant factor for lower levels. This study has initiated the use of neurocognition methods to quantify human cognitive processes. It also encourages design researchers and practitioners to better understand the cognitive factors underpinning creative processes to improve creativity levels.
Wang et al. (Reference Wang, Zhu, Chen, Liu, Sun and Childs2023) investigated the most appropriate metric for measuring AI model generated design images containing combinational creativity. Although several metrics exist, such as Inception Score, Fréchet inception distance, Generated Image Quality Assessment (GIQA), and Contrastive Language-Image Pre-training, for measuring the quality of computer-generated images, it is unclear whether these metrics could be used to assess the creativity aspects. A set of images containing combinational creativity produced by DALL⋅E and human designers was collected and then assessed by both human design experts and computational metrics. The results revealed that GIQA is the closest to human evaluations, which could be potentially used for assessing combinational creative designs in image forms. This study shows the possibility of using computer techniques to evaluate the creativity of AI-generated designs.
Zhang et al. (Reference Zhang, Huang, Yang, Wang, Tang, Han and Liang2023) investigated how the realism level of virtual hand designs in virtual reality affects a user’s sense of embodiment. The sense of embodiment represents the user’s cognitive awareness of manifestation, including the sense of body ownership, agency, and self-location. ET, self-report questionnaire, intentional binding, and proprioceptive measurement were employed in this study to understand the influence of hand designs on embodiment. k-Means, which is often used for data clustering in ML, was employed to cluster eye behavior data. The study results indicated that low realism hand designs showed the lowest scores of embodiments, while using human hand designs led to higher user attention. This study has provided practical guidance in virtual hand design for VR applications.
Sun et al. (Reference Sun, Huang, Wu, Yang, Han and Yue2023b) evaluated the feeling of control in virtual objective on two-dimensional (2D) interfaces by measuring the users’ sense of agency. The different degrees of freedom (DoF) of the virtual object translation modes and 2D interface types would impact users’ performance. To better understand such impact, the authors compared the participants’ sense of agency, which is a psychological aspect of the feeling of control, across different virtual object translation modes (1DoF and 3DoF) and interface types (mouse-based, touch-based, and hand-held augmented reality [AR]) by employing self-report, task performance, and EEG. k-Means and principal component analysis were applied to EEG microstate maps to better analyze the data. The study results revealed that 3DoF mode and AR interface provide users with less sense of agency, affecting design quality and creativity, compared to the others. This study has suggested several design recommendations for improving users’ feeling of control in virtual object translation on 2D screens to better support creative idea generation.
Tehranchi et al. (Reference Tehranchi, Bagherzadeh and Ritter2023) proposed a cognitive user model for testing interface design by predicting human behavior and performing relevant tasks. Computational models for stimulating human intelligence are often incomplete and lack interactions with the environment. Cognitive user models, which combine task knowledge and cognitive architecture, could support decision-making processes involving human–computer interactions through interface design. The authors compared three cognitive models, involving human participants, and proposed a cognitive user model of human error and correction based on user keypress errors. This model has enhanced prediction capabilities for interface designs. The study has shown the importance of predicting human behavior for improving interactive design to support computational models.
This thematic collection encompasses a wide range of studies exploring various aspects of applying AI techniques and cognitive science in design. Some studies have focused on how AI can be used to augment and support humans in design (Sun et al., Reference Sun, Li, Zhou, Lou, Li and Zhang2023a; Wang et al., Reference Wang, Zhu, Chen, Liu, Sun and Childs2023), some explored new user cognitive models for design (Tehranchi et al., Reference Tehranchi, Bagherzadeh and Ritter2023), while others probed human design cognition using neurophysiological and biometric measures (Hu et al., Reference Hu, McComb and Goucher-Lambert2023; Sun et al., Reference Sun, Huang, Wu, Yang, Han and Yue2023b; Yin et al., Reference Yin, Zuo and Childs2023; Zhang et al., Reference Zhang, Huang, Yang, Wang, Tang, Han and Liang2023). Additional studies have exploited AI for analyzing design cognitive data (Chiu et al., Reference Chiu, Lim and Silva2023; Hu et al., Reference Hu, McComb and Goucher-Lambert2023; Sun et al., Reference Sun, Huang, Wu, Yang, Han and Yue2023b; Zhang et al., Reference Zhang, Huang, Yang, Wang, Tang, Han and Liang2023). These studies have collectively contributed to a deeper understanding of design and indicated the potential of bridging AI and cognitive science to understand and support humans in varying design activities.
Future research directions
Amidst the advancement of AI in design research, humans currently remain at the center of design. However, many previous AI design studies seem to have neglected the role and importance of humans in design. For most of these studies, design tasks are performed independently by AI without the involvement of humans. Although the outcomes produced by these AI are often used to support human designers and benefit end users, human intelligence and intervention were not involved in the AI, such as training and testing the algorithms for creating a continuous feedback loop to better support design to meet the needs of humans.
In this thematic collection, Wang et al. (Reference Wang, Zhu, Chen, Liu, Sun and Childs2023) assessed the creativity of AI-generated designs by using both human experts and computational techniques, Tehranchi et al. (Reference Tehranchi, Bagherzadeh and Ritter2023) showed the importance of human behavior predictions in computational models for design, and Sun et al. (Reference Sun, Li, Zhou, Lou, Li and Zhang2023a) indicated the inclusion of human designers in ML lifecycles could better support the generation of design solutions. These three studies have shown the significance and potential necessity of integrating human expertise and intelligence into AI design process. AI can provide valuable support for design, while the human touch ensures the designs produced truly meet the needs of humans. This leads to our first proposition for future research.
Research direction 1: Human-in-the-loop AI for design. The inclusion of humans as the central role in AI design process, where AI is used to augment human capabilities and offer insights while humans amend the process through feedback, to create human-centric design solutions.
As indicated in the preceding, several different neurophysiological and biometric technologies are employed in design research to study design cognition, emotions, and behaviors. Each of the measures has its own advantages. For instance, EEG has high temporal resolution which is suitable for exploring brain activities related to neural oscillations in mental tasks or specific events in time, and fMRI has high spatial resolution which is appropriate for investigating where activation occurs in the brain during tasks, and ET provides eye and pupillary responses associated with emotional or cognitive processing (Hay et al., Reference Hay, Duffy, Gilbert and Grealy2022; Skaramagkas et al., Reference Skaramagkas, Giannakakis, Ktistakis, Manousos, Karatzanis, Tachos, Tripoliti, Marias, Fotiadis and Tsiknakis2023). The use of multimodal measures, combining the data collected by using several neurophysiological and biometric technologies, could provide more robust and comprehensive results, and has been increasingly adopted in recent cognitive science research (Ergan et al., Reference Ergan, Radwan, Zou, Tseng and Han2019; Debie et al., Reference Debie, Rojas, Fidock, Barlow, Kasmarik, Anavatti, Garratt and Abbass2021; Skaramagkas et al., Reference Skaramagkas, Giannakakis, Ktistakis, Manousos, Karatzanis, Tachos, Tripoliti, Marias, Fotiadis and Tsiknakis2023).
However, in design research, most studies have focused on using one neurophysiological/biometric measure instrument only, which might have limited the findings. In this thematic collection, Yin et al. (Reference Yin, Zuo and Childs2023) employed EEG, Hu et al. (Reference Hu, McComb and Goucher-Lambert2023) used fMRI, while Sun et al. (Reference Sun, Huang, Wu, Yang, Han and Yue2023b) used EEG and self-report and task performance, and Zhang et al. (Reference Zhang, Huang, Yang, Wang, Tang, Han and Liang2023) utilized ET and self-report questionnaire. To adopt the latest cognitive science research multimodal approaches and incorporate conventional design research approaches, it leads to our second proposition for future research.
Research direction 2: Multimodal measures for design. The use of multimodal measures, involving use of approaches such as neurophysiological/biometric technologies (e.g., EEG and ET) and conventional measures (e.g., questionnaires and observations), to better understand human design cognition and behavior.
AI has enabled the discovery of patterns and classification of data in large and complex datasets for achieving a wide range of purposes. This provides the opportunity for using AI techniques to analyze cognitive data, such as brain signals and ET data, to yield further insights (Saeidi et al., Reference Saeidi, Karwowski, Farahani, Fiok, Taiar, Hancock and Al-Juaid2021). In addition, the advancements in AI have continuously promoted the results of brain encoding and decoding, interpreting brain signals into text, vocal language, and images. For instance, the acoustic interpretation of EEG brain signals, converting EEG to sound using an AI-driven attention mechanism (Gomez-Quintana et al., Reference Gomez-Quintana, O’Shea, Factor, Popovici and Temko2022); the interpretation of human thoughts, captured via fMRI, into words by using GPT (Tang et al., Reference Tang, LeBel, Jain and Huth2023); and the reconstruction from EEG signals to corresponding images based on diffusion model (Zeng et al., Reference Zeng, Xia, Qian, Hattori, Wang and Kong2023). Bringing together AI and cognitive science has thereby offered methods and approaches to study design from new perspectives. The convergence of the two disciplines could lead to a deeper understanding of the intricate cognitive processes that underpin design and enable brain–computer interface technologies.
Utilizing the power of AI algorithms could better process data and extract features from design cognitive data, such as neurophysiological/biometric data acquired in the design context, and transform them into natural language, acoustic, and visual forms to study designers’ and end-users’ thoughts and intentions. This leads to a new and effective way of studying design cognition and enables human–machine interactions for design. In this thematic collection, Hu et al. (Reference Hu, McComb and Goucher-Lambert2023) employed HMM to study patterns in fMRI data, Sun et al. (Reference Sun, Huang, Wu, Yang, Han and Yue2023b) applied k-means and principal component analysis in EEG microstate maps, Zhang et al. (Reference Zhang, Huang, Yang, Wang, Tang, Han and Liang2023) used k-means to cluster ET data, and Chiu et al. (Reference Chiu, Lim and Silva2023) used NLP to analyze written documentation containing design cognitive processes in a design project for visualizing the design progress. These studies have shown the potential of utilizing AI to analyze and interpret design cognitive data to unlock the inner workings of the human brain in design, and lead to the third proposition for future research.
Research direction 3: AI for design cognitive data analysis and interpretation. The use of AI techniques, for example, ML and deep learning, to analyze neurophysiological/biometric data and other data containing cognitive thoughts, offering unprecedented insights into the complex design cognitive processes.
Framework for integration of AI and cognitive science in design
AI and cognitive science are two significant disciplines in design research. It is believed that bridging the two disciplines together with human-centered design will provide the opportunity to uncover more profound insights into design, and ultimately drive design research forward. Based on prior studies and the papers included in this thematic collection, and inspired by recent frameworks on design creativity (Childs et al., Reference Childs, Han, Chen, Jiang, Wang, Park, Yin, Dieckmann and Vilanova2022), data-driven innovation (Luo, Reference Luo2022), and artificial empathy for human-centered design (Zhu and Luo, Reference Zhu and Luo2024), a framework for integration of AI and cognitive science in design has been proposed. The aim of this framework is to provide a structured approach to help design researchers utilize and synthesize both cognitive science and AI in design synthetically for supporting the development of human-centered design methods, strategies, and solutions. The framework has three modules “Humans,” “Cognitive Science,” and “AI,” as shown in Figure 1, and emphasizes their synthesis in design research.
Figure 1. Framework for integration of artificial intelligence and cognitive science in design.
“Humans” is the central module of the framework, and refers to designers, end-users, stakeholders, and others who engage in the design process. This module emphasizes the inclusion of human intelligence in design, such as designers’ thoughts and intuitions and end-users’ insights and feedback, which drive and guide the design process. “Cognitive Science” is a module focusing on the understanding of human cognitive processes and behavior in design through employing neurophysiological/biometric technologies, such as EEG, fMRI, and ET, and conventional measures, such as questionnaires and interviews. The “AI” module uses advanced computational techniques, such as ML and deep learning, to analyze and process varying types of data, including data collected in the design process and external design-related information, to understand design and augment human design capabilities.
Each module plays a distinct role, while the three modules are also interconnected. The interactions between “Humans” and “AI” indicate the inclusion of humans in AI design process to ensure human-centric solutions, which embodies Research Direction 1. The interactions between “Humans” and “Cognitive Science” suggest the use of multimodal measures to collect design cognitive data for more comprehensive and robust results, which aligns with Research Direction 2. The interactions between “Cognitive Science” and “AI” promote the use of AI techniques to analyze and interpret design cognitive data for a deeper understanding of design, which refers to Research Directions 3.
With all three modules together, the framework forms a continuous and iterative loop for exploring and enhancing design. For instance, this could provide real-time feedback to designers during design. Designers’ brain signals are collected and analyzed using AI, which could indicate the designers’ cognitive states, such as attention, reasoning, and perceptions. The AI would therefore provide real-time feedback to the designers, prompting correspondence suggestions, such as recommending new design resources or approaches to better engage the designers, to tackle the challenges faced. The designers’ cognitive data would then be collected and analyzed again to validate the feedback provided and provide new feedback if needed. The synergy of the three modules could also guide designers in coming up with ideas or products that align with end-users’ subconscious preferences to meet their needs and lead to more customized solutions; enable designers to produce sketches and ideas directly from mental thoughts to rapidly visualize and iterate design concepts; and support designer–user communication and collaboration through utilizing AI-interpreted mental thoughts.
The proposed framework for application and integration of AI and cognitive science in design provides a structured approach to study design, where the interactions among the three modules complement and enhance each other, and delivers deep insights into design, creates human-centric design solutions, leads to more robust and comprehensive results, and enhances the design process. The continuous and iterative loop offers the opportunity to explore design in a comprehensive manner, enabling design researchers to investigate the underlying cognitions of design for fostering a deeper understanding of and supporting the needs of humans, such as designers and end-users, involved in design.
Concluding remarks
We hope this thematic collection stimulates more interest in this topic, teeming our research community with excitement about the potential applications of AI and cognitive science in design. Analysis of research papers included in this thematic collection suggests three promising future directions of research of AI and cognitive science in design:
-
1) Human-in-the-loop AI for design.
-
2) Multimodal measures for design.
-
3) AI for design cognitive data analysis and interpretation.
A framework, incorporating the three future research directions, has also been proposed to inspire and guide design researchers interested in the topic to better explore human-centered design methods, strategies, and solutions, as well as potentially lead to the next generation human-centered design support tools and systems.
Ji Han is a senior lecturer in Design and Innovation at the University of Exeter. His research interests include design creativity, data-driven design, and AI in design. His work places a strong emphasis on exploring new design theories and computational models to support design innovation. Prior to his current post at Exeter, he was a lecturer in Industrial Design at the University of Liverpool, where he served as the Program Director for Product Design Engineering and Industrial Design programs. He has published over 60 academic papers to date and has received several best paper awards. He holds a PhD degree in Design Engineering from Imperial College London and an MSc degree in Engineering Design from the University of Bath.
Peter R.N. Childs FREng is the Professorial Lead in Engineering Design, Innovation Design Engineering Professor at Large, Co-Director of the Energy Futures Lab, and was Founding Head of the Dyson School of Design Engineering at Imperial College London. His general interests include creativity, innovation, design, fluid flow and heat transfer, sustainability, and robotics. Prior to his current post at Imperial, he was director of the Rolls-Royce supported University Technology Centre for Aero-Thermal Systems, director of InQbate and professor at the University of Sussex. He has contributed to over 200 refereed journal and conference papers, and several books, including the Handbook on Mechanical Design Engineering (Elsevier), now in its third edition, as well as books on rural urban migration, inclusive sports technologies, vehicle design, and temperature. He is Editor-in-Chief of the Journal of Power and Energy, Founder, Director, and Chairperson at Q-Bot Ltd., and Director and Chairperson at BladeBUG Ltd.
Jianxi Luo is a professor of Systems Engineering at the City University of Hong Kong. Luo holds PhD and S.M. degrees from MIT and M.S. and B.E. degrees from Tsinghua University. He serves on the editorial boards of many engineering design journals, including AI EDAM, JED, Design Science, RIED, ASME JCISE, and IEEE TEM, and on the executive committee of the Council of Engineering Systems Universities. Luo was a former chair of the INFORMS Technology Innovation Management and Entrepreneurship Section. Previously, he was a faculty member at New York University and Singapore University of Technology and Design and a visiting scholar at Cambridge University and Columbia University. He focuses on artificial intelligence for engineering design in the latest research and is a global top 2% scientist by Stanford and Elsevier.
