Fundamentals and issues of user experience in the process of designing consumer products

Abstract

User experience (UX) application in the practice of engineering and product design is still limited. The present paper provides insights into research on UX design and recommendations for design practitioners by pointing out common criticalities. These outcomes are achieved through a literature review on how UX relates to design. First, issues in benefitting from UX understanding in design are identified with a specific focus on theoretical contributions. Second, experimental papers investigating UX and design are analysed in relation to previously identified issues. Although issues are present to some extent in all the contributions, the empirical studies dealing with UX in design are overall valid. The results highlight UX’s support in revealing design requirements, but its capability of steering design processes is arguable, as concrete guidelines for practitioners are not well described. Based on identified issues, the authors propose a checklist to make UX studies in design more reliable and their outcomes more comparable.

1. Introduction

Understanding, addressing and, most importantly, predicting user experience (UX) for new products are relevant to matching consumers’ requirements and needs. UX is recognised as a fundamental asset in avoiding inadequate products (Tiwari, Jain, & Tandon 2016; Christoforakos & Diefenbach 2018) and supporting an extended and more satisfying usage of those products (Feng & Wei 2019). Although the importance of UX in design is widely recognised in the literature (Pucillo & Cascini 2014; Bongard-Blanchy et al. 2015; Li & Hölttä-Otto 2020), it is not clear how to exploit the full potential of UX outputs more broadly in the design process.

The UX concept is particularly complex and multifaceted (Law & van Schaik 2010; Sun & Teng 2017). Indeed, UX captures the whole process of interacting with a product, including affective and cognitive aspects, assessing aesthetic quality, usability and utility aspects. Consequently, the study of UX cannot be limited to the usability or utilitarian side of the human–product interaction (HPI) (McNamara & Kirakowski 2006; Merholz 2007; Norman & Nielsen 2017). (User) experience goes much beyond the typical assessment schemes, which cover only partial elements of the whole experience. This complex interaction between different modalities, levels of processing and several factors that shape a UX makes it particularly difficult to investigate the design process and to decide how to implement the most adequate UX model.

Another challenge in investigating UX in design is posed by the many fields of application of this concept (Law & van Schaik 2010). Psychology, human factors, anthropology, philosophy, computer science, as well as technical subjects like engineering and design are just a few examples of the variety of heterogeneous fields where UX is studied and potentially applied. Each of these fields has its own approaches and characteristics, which broaden the definition of UX generating misleading interpretations that are difficult to synthesise (Law et al. 2007; Sauer, Sonderegger, & Schmutz 2020).

This unclarity affects studies and contributions aiming at analysing UX from a theoretical perspective. A considerable shortcoming is the frequent failure to analyse the holistic quality of UX (Satti et al. 2019; Hussain, Mkpojiogu, & Husin 2021). In other words, scholars recognise the complexity and multifaceted nature of UX, but then they tend to focus on a few particular elements only (Berni & Borgianni 2021a). Such elements are identified but seldom studied comprehensively (Hussain, Mkpojiogu, & Husin 2021).

This leads to an ineffective application of UX within the design practice. Indeed, most of the contributions analyse UX just theoretically developing frameworks and models that are usually not validated through practical case studies (Carbon 2019a). Hassenzahl & Tractinsky (2006) stressed the lack of empirical studies in the specific field of Human Computer Interaction (HCI). Only recently, thanks to technological development, most studies focused their attention on HCI (Azofeifa et al. 2022). UX has been investigated and applied to develop new information and communications technology (ICT) devices and other emerging interaction technologies such as virtual reality (VR) or augmented reality (AR). This interest overshadowed other design fields, such as industrial and engineering design, which could have benefitted from the application UX outputs as well. Based on a literature analysis and to the best of authors’ knowledge, no scholar has claimed that this gap has been filled.

In this context, the objectives of the present paper are:

• • to identify the most recurrent issues in studies dealing with the application of UX in design, which is pursued through a scoping review (Section 2);

• • to understand if and how empirical and experimental contributions have considered or solved the issues identified in the literature, which is pursued by means of a systematic search (Section 3) and subsequent analysis (Section 4) of UX applications in design.

2. Background

As mentioned in Section 1, UX captures the whole process of interacting with the product; beside assessing the usability, functionality and aesthetic qualities of a product, UX considers subjective qualities of the HPI such as perception, emotions and feelings (Mäkelä & Fulton Suri 2001; Hassenzahl & Tractinsky 2006; Sward & Macarthur 2007; Kuniavsky 2010; Law & van Schaik 2010). Experience goes much beyond the typical assessment schemes, which cover only partial elements of this whole experience. UX is indeed based on expectations (Kaasinen et al. 2013), associations (Ortlieb 1866; Breitschaft & Carbon 2021) and knowledge (da Silva, Crilly, & Hekkert 2015), as all kinds of experiences are very subjectively driven (Kuniavsky 2010; Fokkinga, Desmet, & Hekkert 2020). This complex interaction between different modalities, levels of processing and the several factors that shape a UX makes it particularly difficult to investigate. Besides that, it is even harder to capture the holistic quality of UX as it is a phenomenological quality embedded in situational, social and cultural contexts (Carbon 2019a).

The subjectivity and complexity of UX create a challenge for studying and analysing this concept. Although the importance of UX in design is well established, it is still not clear how to exploit the full potential of UX outputs in the different design phases. Such unclarity is due to theoretical and practical issues affecting UX research and practice. Sections 2.1 and 2.2 illustrate the complexity of the issues by analysing the literature through a scoping review of papers critically discussing UX in design; this involves both a theoretical perspective (Section 2.1) and a practical/operational one (Section 2.2). The issues identified are attributed a name, which will be used in the residual of the paper. Section 2.3 summarises the issues encountered from both described perspectives into a list.

2.1. Theoretical issues

In this subsection, the identified issues related to the theoretical development of the UX concept, its definitions, its elements and interpretations are synthesised and described.

The difficulty in defining UX boundaries concerning the concept of ‘usability’ is reported by several scholars (Følstad & Rolfsen 2006; Sauer, Sonderegger, & Schmutz 2020) as soon as the term ‘UX’ was introduced by Norman, Miller, & Henderson (1995). Donald Norman claimed that UX goes far beyond the limited concept of ‘Usability’, justifying the introduction of such a new term. More than two decades later, however, the versatility of the meaning of ‘UX’ led to an overuse of this concept, which lost its initial strength and significance. Actually, Norman himself complained about the abuse of his term (Merholz 2007), which is sometimes misused or is not the real focus of the research if ‘UX’ is considered formally. The lack of clarity around the term ‘UX’ was one of the first issues that emerged from the literature, which caused fragmentation in the UX research (hereinafter, issue terminology).

To better understand UX, Berni and Borgianni Berni & Borgianni (2021b) summarised the various key elements of UX, making order among the fragmentary definitions in the literature. The scholars addressed this fragmentation in terms of conceptual contributions proposing interpretations, definitions and theoretical frameworks of UX. Key components (user, interaction, system and context) and dimensions (ergonomic, affective and cognitive experiences: definitions thereof are given in Section 3) were identified. Components and dimensions were categorised into fundamental elements and influence factors, with a particular focus on their impact on the interaction output. The authors strongly suggest analysing both key components and UX dimensions in a holistic way, since most of the studies consider those elements separately instead of as different variables affecting the same experience. Otherwise, the risk is to focus only on certain key components while neglecting others. This is the case of the context where the interaction takes place. While elements such as ‘user’ and ‘system’ are often considered, ‘context’ is largely neglected despite its acknowledged importance in UX. The reason could lay behind the different ways in which context can be interpreted, namely at least the following four dimensions according to Berni & Borgianni (2021b).

1. 1. Cultural. Cultural studies interpret and analyse the cultural background of users to understand how differences in language, habits and norms affect the interaction with the product (Ortiz & Aurisicchio 2011).

2. 2. Social. Forlizzi (2008) stresses the importance of the social aspect of the context as interaction of multiple actors with a product. Focusing on this interpretation of the context could help designers understand and think about the dynamics of social behaviour while designing product (Ortiz & Aurisicchio 2011). The HPI includes various elements, which are overall ascribable to the psychological and social reality, the so-called Umwelt (Carbon 2020), which therefore includes cultural aspects too.

3. 3. Physical/spatial. The space factor can be intended as both physical surrounding (featured by the location, light conditions, temperature and so on) and the technology used to display the interaction itself, for example, virtual or physical interaction (Rebelo et al. 2012).

4. 4. Temporal. Interactions are strongly affected by the timing of the HPI, including the experience accumulated over time, leading to different expectations (Becattini et al. 2017). The consideration of the temporal context in HPI is one of the main challenges of UX in design, as explained below.

One way to study the context according to a temporal interpretation is to consider UX as a dynamic concept that changes over time (Kujala et al. 2011) – an approach which turned out fruitful also in the related area of art experience (Muth & Carbon 2016; Muth, Raab, & Carbon 2017; Muth, Hesslinger, & Carbon 2018). This can be done in several ways:

• • Studying different kinds of interactions at different phases of use. Yoon, Kim, & Kang (2020) divided the interaction into seven phases: before purchase, during purchase, unboxing, first use, familiarisation, use, disposal or repurchase. The scholars have highlighted a substantial diversity in the positive experiences of users according to the phase of product use. To study UX, scholars employed retrospective techniques. Participants were asked to recall and evaluate an experience rather than interact with the product directly. This led participants to filter the evaluation through their memory of a past event that may be affected by emotional biases and inaccuracies. In other studies, the experience is not recalled from the past but is projected in the future through scenario building. Participants must imagine a situation and project their potential interaction using future-scenario techniques (Sleeswijk Visser & Visser 2006) (hereinafter, issue retrospective/projected).

• • Considering the context as the user’s background, with their preferences and expectations. UX is based on expectations, associations and knowledge, as all kinds of experiences are very subjectively driven. Expectations, which are largely overlooked in UX research, play a role in the mental representation of the product experience; these are plainly intertwined with people’s knowledge and experience, stimulated by what is sensed out of the product, and consequently transformative (Pettersson 2017).

• • Studying the context as Zeitgeist factor (Carbon 2011). UX of products is modulated by Zeitgeist-dependent effects which play a major role when UX is investigated over a more extended period (see Carbon 2011) – and such longer perspectives are important to understand the usage and experience of products.

While efforts to address UX in a holistic way are scarce, experiences are more diffusedly classified as positive or negative. Some scholars focus on the positive–negative distinction to understand which output should be used to improve a product within the design process. Therefore, positive and/or negative experiences have been extensively studied in the literature (Hassenzahl 2010; Kim 2012; Kim & Christiaans 2012; Fokkinga & Desmet 2013; Yoon, Pohlmeyer, & Desmet 2017; Fokkinga, Desmet, & Hekkert 2020; Yoon, Kim, & Kang 2020), even though scholars do not always agree on which of the two aspects of UX is the most useful in design. For example, Kim & Christiaans (2012) and Kim (2014) developed a framework focusing on an empirical cross-cultural study aimed at understanding which users’ characteristics related to products can lead to users’ negative experiences. This work is helpful in foreseeing unwanted negative experiences to avoid them. According to Yoon, Kim, & Kang (2020), it is better to seek the elements that arouse a positive experience, rather than just mitigating or avoiding unwanted ones. This view is strengthened by Hassenzahl (2010), who claims that eliminating suffering and frustrations caused by undesired features can improve a product but it does not ensure a positive experience.

Three dimensions of experience can be identified beside the general division between ‘positive’ and ‘negative’. Those dimensions have been identified as ergonomic, cognitive and emotional experience (details are provided in Section 3) depending on whether the focus is on the system, its features and functionality, on the user and how it perceives the system or which feelings arise from the interaction between the user and the system.

Each of these types of experiences is characterised by the focus on specific aspects that come from the HPI resulting in different approaches used to investigate UX in the literature. The main approaches are three and they somehow correspond to the main experiences described above.

• • The ergonomic approach is strongly related to the functional side of the HPI and lined to the concept of affordances (Crilly 2011; Carbon 2019b).

• • The user-centred design (UCD) approach is closely connected to the users, their cognitive perception of a product and their involvement in the design process through evaluations (Wong, Khong, & Thwaites 2012; Lei et al. 2018; Chen, Mata, & Fadel 2020).

• • The emotional design approach is linked to the emotional evaluation of a product or system (Norman 2004; Bagnara & Smith 2006; Agarwal & Meyer 2009).

As for key elements and for the different kinds of experiences, the boundaries between these approaches are not well defined. However, each approach presents specific characteristics. By studying each approach separately, the risk is to limit research on some aspects only, while other elements may be poorly investigated. Otherwise said, if some studies analyse UX only through an emotional approach, the research could be incomplete lacking evidence on the functional and ergonomic side of the HPI (hereinafter, issue comprehensiveness).

2.2. Practical or operational issues

The unclarity characterizing the theoretical research on UX also affects those contributions that investigate UX from a practical perspective. Indeed, when it comes to applying UX within the design process, other issues emerge. Few contributions investigate the involvement of UX in the design process for products belonging to engineering design and everyday items. Such products are characterised mostly by tangible components users can directly interact with and manipulate (Berni & Borgianni 2021c). Although UX has a wide range of applications, it seems that experimental design research on UX is limited to the fields of

• • automotive (Williams, Attridge, & Pitts 2011; Lawson, Salanitri, & Waterfield 2016; Shi, Bordegoni, & Caruso 2020),

• • healthcare (Harte et al. 2017; McCarthy, Ramírez, & Robinson 2017; Taylor, McDonagh, & Hansen 2017)

• • and design for spaces (Brown et al. 2010; Jung et al. 2018; Ergan et al. 2019).

Conversely, most of the experimental activity available in the literature is limited to examining UX in the field of HCI (Bargas-Avila & Hornbæk 2012; Córdoba-Cely & Alatriste-Martínez 2013). Products belonging to this field are indeed electronic devices and ICT products, which are featured by two specific components: software (or digital and intangible) and hardware, related to the physical and tangible parts of the devices (Berni & Borgianni 2021c). The reason of such an application field is justified by the origin of the term ‘UX’ itself popularised by Norman, Miller, & Henderson (1995), who gave rise to UX research in HCI. However, the investigation has been mainly restricted to HPI and, markedly, utilisation, especially in electronic and ICT devices. This reduces the analysis of the experience to a very limited and overemphasised set of dimensions such as usability and affordances, neglecting other important dimensions such as emotional processing (Pucillo & Cascini 2014).

Such a strict focus raised other two issues: the intangible components (software) such as services, web app and digital interfaces are usually more investigated than the tangible ones (hardware) (Bargas-Avila & Hornbæk 2012). This leads to a lack of generalisability of methods to study UX, limiting the validity of findings and the applicability of methods to many artefacts. Such predominance could be due to the chance to collect objective data straightforwardly in web-based and digital systems and through the opportunities enabled by the Internet of Things (Voet et al. 2019), which is not clearly feasible in the predominant number of consumer goods.

A second issue descends from the popularity of UX in HCI. UX studies often involve advanced technologies and systems such as VR or AR. However, many of these studies aim at investigating the performance of such technologies (e.g., sense of presence and interactivity) rather than focusing on the represented product itself (Gaffary et al. 2017; Eroglu et al. 2018; Kwon et al. 2019; Shor et al. 2019; Kim et al. 2020) (hereinafter, issue instruments).

The focus on the interaction with those technologies and their evaluation may lead designers and researchers to lose sight of the applicability of the outputs obtained in these studies. Sometimes, it is difficult to understand exactly what stage of the design process these outputs can be more effectively applied to. Some scholars do not specify which design phase would benefit the most from the application of such objectives. Otherwise, if the utility of the research output is mentioned, it is often referred to the design process in general. In other words, it is difficult to clearly extrapolate design objectives of the contributions (hereinafter, issue clarity of design objectives).

The literature offers a contrasting view on UX data collection. It is unclear whether a more objective or subjective data collection approach to measure UX should be preferred. According to Norman (2004), UX is characterised by emotional components specific to human beings and their feelings. This would make it difficult to measure UX through data collected more objectively (Chou 2018). Sutcliffe (2009) prefers pragmatic approaches focusing just on aspects such as usability and efficiency, which are considered easier to address objectively. The use of more sophisticated procedures and tools, markedly biometric and neurophysiologic devices, has emerged, but established routines and paradigms to investigate design evaluation and product experience are far from being fine-tuned according to a recent review (Borgianni & Maccioni 2020). It can be observed that design studies use biometric tools to extract information about people’s non-conscious behaviours, but this information is frequently related to specific domains of experience (emotions, use, cognition, affordances) without any example targeting UX as a comprehensive phenomenon. In contrast, other scholars believe that a mixed data collection approach is preferable where subjective data collection methods such as questionnaires, interviews and Likert ratings are placed alongside objective methods (Bitkina, Kim, & Park 2020) (hereinafter, issue subjectivity).

The lack of clarity in data collection methods could lead to the development of UX methodologies, approaches and protocols for their own sake. In other words, scholars risk focusing more on the construction of the methodology to evaluate UX, neglecting the potential application of the UX outputs within the design process (hereinafter, issue method demonstration).

Experiments and case studies are a means to validate new methodologies to study UX and demonstrate its operational usefulness in the different stages of the design process. In order for the experimental results to be generalisable, experiments need to have a good ecological validity. In this regard, the selection of the participants for the experiments deserves much attention. While considering participants as a necessary aspect of empirical studies, insufficient attention has been paid to samples. Generally, studies may recruit either experts or novices, depending on whether the research entails, for example, usability testing or product appreciation. Here, experts are generally selected based on their knowledge/experience and they diffusedly include ‘insiders’, for example, people participating in the development of the products to be evaluated. Conversely, even if the estimation of the sample size has a long tradition (Lewis 1994; Schmettow 2012), the recruitment of novices and laypersons (often people who cannot afford the respective products or would not use them in real life) is far from being representative of any population. In studies evaluating UX, especially in design, these participants are primarily recruited among students or a restricted set of users that often share many features. Moreover, the characterisation of participants is generally made through a set of questions to determine their attitude, while the sample composition is seldom compared to the population of potential users. This lack constrains generalisation and hinders any possibility to explain data further. This might be a reason behind the poor operational benefit of UX studies in terms of steering subsequent design phases (Pucillo & Cascini 2014) (hereinafter, issue participants).

Yet, as long as design phases are concerned, UX studies in design focus mainly on early and final stages of the process. Poor attention is paid to the intermediate stages of prototyping and concept testing. Indeed, UX is used a priori to identify user’s needs, preferences, emotions for idea generation, brainstorming and concept ideation or a posteriori at the end of the design process when the concept is fully developed, and core design decisions cannot be reversed anymore. Hayat et al. (2020) claim the importance of UX in early stages. Users’ evaluations are often used for idea generation and brainstorming. Users are usually involved in brainstorming sessions to understand their needs and preferences in terms of design decisions and product features. In other contributions, UX is used at the end of the process when the product is almost fully developed. Users’ needs, preferences and emotions (UX outputs) are identified for concept creation, but it is seldom tested whether the final concept (design output) really reflects the tastes, preferences and needs of users as potential buyers (Ning, Goodman-Deane, & Clarkson 2019). This suggests that the intermediate design stages concerning prototyping need to be further investigated.

The exploitation of UX outputs could help reduce the gap between design intentions and user perception leading to the development of satisfactory products requiring less time and investments. The mismatch between designers’ intentions and what users perceive is emphasised by several authors (Crilly, Maier, & Clarkson 2008; Crilly 2011). In particular, Ning, Goodman-Deane, & Clarkson (2019) underline how the HPI translates into a cognitive challenge to understand the design decisions of a product. In fact, designers rely on their experience, intuition and common sense to make design decisions. According to da Silva, Crilly, & Hekkert (2015), users’ knowledge of the design intentions could also influence their appreciation of a product. The scholars also point out that research has not empirically addressed the problem to whether and how such knowledge can influence the evaluation of a product.

2.3. Main issues in UX in design

Based on the above analysis of the literature, the identified issues have been grouped into eight categories. Table 1 summarises them as follows.

• • The first column indicates the name of the identified issue category, as above.

• • In the second column, an illustration of the issue is provided concisely.

• • In the third column, the criteria used to detect the identified issue category in selected research papers (see Section 3) are reported. In some cases, there can be more than one identification criterion distinguished by the capital letters ‘A’, ‘B’ or ‘C’

Table 1. List of the issues identified in the literature on UX in design

3. Gathering and analysis of pertinent literature

The issues summarised in Table 1 were then used to categorise UX applications in design gathered from the literature through a systematic search and review, which has largely benefitted from the terms found in the bullet list below (especially those in quotation marks) used as search keys. Particular attention has been paid to contributions published in design journals. The aim is to verify to what extent the experimental research on UX in the industrial, engineering and product design suffers from the reported issues, and if and how it addresses and overcomes these issues. For this reason, the authors collected a comprehensive sample of articles evaluating or measuring UX in design to be analysed insightfully, as common in systematic search and reviews. Papers were identified by searching the Scopus and Google Scholar databases. The latter was used due to the possibility it enables to search in the full text of contributions; however, as some results might be of limited scientific authoritativeness, their sources were checked before including possibly pertinent papers in the selected sample of articles. The inclusion criteria to determine the articles’ pertinence are listed below.

• • UX experiments and applications are illustrated along with results that are potentially useful or applicable in the design field. Papers developing models, methods and frameworks have been included only if they validated their theoretical output through a case study or an experiment.

• • The term ‘UX’ is clearly mentioned. Papers who just referred to ‘HPI’ or other terms with no mention to UX have been excluded.

• • UX is evaluated with tangible products, for example, consumer goods, industrial or engineering products, furniture, appliances and their components. Papers investigating ICT products have been included only if the evaluation of the product included hardware components. Those evaluating just the digital interface have been excluded.

• • Interaction is sufficiently extended, such as it can potentially give rise to emotional and functional experiences, whether these are analysed or not. Interaction is not intended as touching, visual observation and assembly/disassembly only.

At the end of the selection process, 63 papers constituted the final sample, which are listed in Table 2 in chronological order. The criteria to define whether the issues apply to each contribution are included in Table 1. The classification based on these criteria took place in a consensual way between all the four authors. Therefore, for each issue category, it has been established (see Table 2):

• • If the issue criteria are met (Y in Table 2), and therefore the contribution suffers from the specific issue;

• • If the issue criteria are not met (N in Table 2);

• • If the information presented in the paper did not allow the authors to establish the presence of the issue or if some circumstances made the issue irrelevant (n/a).

Table 2. Sample analysed; ‘Y’ indicates the presence of the issue, ‘N’ the absence of the issue while ‘n/a’ means that the issue is not applicable due to lack of information

Here, it is to be pointed out that the analysis concerned the presence and absence of issues, and it could not identify proposals indented to explicitly solve these issues because they have been altogether defined in the present paper as an original contribution.

It is also worth noting that ‘comprehensiveness’ does not undergo the same classification criteria of the other classes. Berni & Borgianni (2021a) listed all the UX key components (user, system, context) and UX dimensions (affective, cognitive, ergonomic) targeted in each contribution to assess the comprehensiveness of UX in the studies. The following aspects have been considered for the identification of the key components.

• • ‘User’: the paper focuses on users, their needs and expectations; it follows the UCD approach; it considers user knowledge, background and behaviour; it involves users in co-design sessions and evaluations; it is based on customisation or personalisation.

• • ‘System’: the paper focuses on new product development processes; it describes product features, appearance and aesthetics; it is based on product assessment or product re-design.

• • ‘Context’: the paper focuses on the cultural meaning and the circumstances taking places during an interaction; it considers the context as an environmental phenomenon and/or temporal aspect investigating UX over time. Here, the definition of context builds upon (Zimmermann, Lorenz, & Oppermann 2007), where this is featured by five fundamental categories, that is individuality, time, location/environment, activity and relations.

• • A slash ‘/’ is used when it was not possible to identify any key component.

The following aspects have been considered for the identification of the UX dimensions, which are further explained by means of definitions leveraged here for the scopes of the subsequent classification.

• • ‘Affective’: the paper focuses on any emotional responses when dealing with the product, based on the common understanding of the affective experience (Alben 1996; Norman 2004; Chitturi 2009; Sutcliffe 2009; Norman & Nielsen 2017). Feelings, preferences, emotions, empathy, satisfaction, value and meaning are considered. The reference definition used for affect to the scope of the classification is borrowed by Desmet & Hekkert (2007), where the scholars interpret the meaning assigned in psychology and adapt it to the field of experiences and design. Affect is involved in ‘all types of subjective experiences that are valenced, that is, experiences that involve a perceived goodness or badness, pleasantness or unpleasantness’.

• • ‘Cognitive’: the paper focuses on how the user cognitively perceives and appreciates the product and its aesthetic through the human senses (Crilly, Moultrie, & Clarkson 2004; Hassenzahl 2004; Desmet & Hekkert 2007; Siu, Ng, & Chan 2011). This includes sensorial perception, cognitive and aesthetic perception as well as the further cognitive processing including the association with already made experiences with this or related products. The reference definition for the purpose of the classification follows (Cohen 2020), where cognition is based on a psychological construct, namely unobservable elements of human mind, and how people perceive and process information.

• • ‘Ergonomic’: the paper focuses on the dimension of experience related to usability, affordances, comfort, safety, efficiency, effectiveness and satisfaction of using a product beyond those characteristics not ascribable to perception and emotions (Pucillo et al., 2016; Pucillo & Cascini 2014; Zunjic, 2017). It therefore refers to usability, effectiveness, functionality, use, usage, affordances and understanding of the design intention. The reference definition of the ergonomic dimension of UX is borrowed by Wilson (2000), where ergonomics is intended as the ‘theoretical and fundamental understanding of human behavior and performance in purposeful interacting socio-technical systems’.

• • A slash ‘/’ is used when it was not possible to identify any UX dimension.

It is important to note that the mere analysis on which UX dimensions (affective, cognitive and ergonomic) studies focused does not imply that these three dimensions of UX are disjunctive. On the contrary, they all share some common grounds, but importantly, we will try to identify on which dimension(s) the corresponding core work was accomplished. In the classification process, the authors have considered what actually has been assessed and evaluated in the UX study regardless of the terms used in the specific contribution, that is the above definitions have been employed for the classification purposes.

The Supplementary Materials include a spreadsheet with an extended version of Table 2, where all the assignments (Y, N, n/a, /) are commented and justified.

4. Analysis of the classification results

In this section, the results from Table 2 are presented and commented. In Section 4.1, the authors provide a quantitative analysis of the number of ‘Y’, ‘N’ and ‘n/a’ per each category of issue. In Section 4.2, papers with a limited number of issues are then considered as possible best practices and described more extensively. Since the ‘comprehensiveness’ category follows a different classification criterion (see Section 3), the results will be commented on separately in Section 4.3.

4.1. Overview of the results in quantitative terms

Table 3 presents the count of each ‘Y’, ‘N’ and ‘n/a’ per each category and subcategory of issues. The last column reports the total count of presences or absences of issues.

Table 3. Analysis and count of the number of ‘Y’, ‘N’ and ‘n/a’ per each category of issues

No paper has just ‘Y’ or ‘N’. These data indicate that the categories of issues identified from the literature are validated and present to some extent in all the contributions. However, the prevalence of ‘N’ (Y = 292; N = 580) shows that empirical studies investigating UX in design are overall valid despite the inevitable presence of some limitations. More in details:

• • Despite the prevalence of ‘absence of issue’, there are some categories where the number of ‘Y’ is close to the ‘N’. This is the case of ‘terminology’ (Y = 28; N = 35), where the large number of ‘Y’ suggests that there is still a fair amount of indefiniteness about framing the UX concept at a theoretical level. In particular, the boundaries between usability, UX, its key components and dimensions are often unclear. This frequently leads to consider the UX not as a single concept but as a part belonging to a broader concept.

• • It is worth noting that the prevalence of ‘N’ in ‘method demonstration’ suggests that many studies mention the benefits of their research outputs in design. However, the high number of ‘Y’ in ‘clarity of design objectives A’ indicates that the utility of those outputs within the design process phases is seldom clarified(Yeh, Gregory, & Ritter 2010; Williams, Attridge, & Pitts 2011; Gkouskos, Normark, & Lundgren 2014; da Silva, Crilly, & Hekkert 2015; Zuo et al. 2016; Yoon, Pohlmeyer, & Desmet 2017; Zhou et al. 2017; Zhang et al. 2018; Alonso-García et al. 2020).

• • The large number of ‘n/a’ in ‘participants’ means that, often, data are not acquired directly from potential end users. Other data acquisition techniques have been implemented:

  • ○ online data mining (Chien et al. 2016; Ghajargar et al. 2017; Yang et al. 2019)

  • ○ user information processed through artificial intelligence (Koonsanit & Nishiuchi 2021).

In other cases, (see the ‘Y’ in ‘participants B’), UX is studied not only through the interaction and evaluation of end users but also through expert evaluation or analysis of concept case studies where authors comment on the potential interaction between the presented product and the user (Bordegoni et al. 2014; van der Bijl-Brouwer & van der Voort 2014; Bongard-Blanchy et al. 2015; Karana, Barati, & Rognoli 2015; Yoon, Pohlmeyer, & Desmet 2017).

• • The conspicuous presence of issues of ‘subjectivity’ indicates that mixed methods of data acquisition are still underutilised.

4.2. Strengths and weaknesses of UX studies exhibiting few issues

The majority of ‘N’ allowed the identification of potential best practices that could be used as a target to conduct empirical UX studies. Following the classification in Table 2, the authors arbitrarily included as ‘best practice’ all the papers (see Table 4) presenting maximum three issues (Y) and two ‘n/a’ designations. These quantities are chosen so to provide for a sufficiently rich description of possible references for future studies.

Table 4. Identification of contributions presenting a possible best practice, that is characterised by a maximum of three ‘Y’ and two ‘n.a.’ and the vast majority of ‘N’

By means of Table 4, readers can notice that the designated best practice examples still suffer from two major issues: ‘participants’ and ‘subjectivity’. These are diffused also if the whole sample of analysed papers is considered (see Table 3). Conversely, the issues ‘methods demonstration’ and ‘clarity of design objectives’ have (almost) no entry despite their relative frequency in the whole sample. Details follow.

The contributions listed in Table 4 can be considered as best practices because they oftentimes present advantages over other studies included in the sample:

• ○ clarity in defining UX since 8 studies out of 10 do not present the ‘terminology’ issue; as a consequence, the presented findings can be actually considered representative of the role of UX in design; hence, results are comparable because they make reference to a solid theoretical construct instead of vague ‘approximations’ of UX;

• ○ in these contributions, neither is the focus on the test of new methodologies, nor are objectives vague (see the issues ‘methods demonstration’ and ‘clarity of design objectives’). These two aspects are possibly logically connected. The ‘best practice’ studies clearly put in practice past knowledge and allow the visualisation of possible applications of UX in design. This is strengthened by their diffused use of a well-defined conceptualisation of UX (see above). Altogether, they are characterised by a superior methodological approach.

Residual issues are supposedly related to contingent factors. ‘Participants’ is the second most frequent category of issues; despite the sample size and characteristics could depend on the objectives of the research, this problem can no longer be neglected as it is increasingly considered as a strong limitation of the validity of design studies (Cash et al. 2022). This clearly applies to the use of UX in design too. Those studies in which the sampling of participants has been overlooked or has been made based on convenience should clearly address this limitation. In some circumstances, it could be advisable to repeat existing studies to increase the validity of or disconfirm previously achieved results (Cross 2018) rather than running research of poor future impact. Moreover, almost all the selected best practices suffer from the issue related to subjectivity. This takes place despite the literature suggests using mixed methods for data acquisition to analyse both quantitative and qualitative data. Here, it is worth noting that the acquisition of objective data takes place through tools and instruments that have made inroads in design only recently (Borgianni & Maccioni 2020). On the one hand, the papers designated as best practices are, surprisingly, relatively old – just 2 out of 10 have been published in the last 5 years. On the other hand, the presence of new technologies lends itself to focus on their usability and usefulness rather than the outcomes in terms of UX in design. This might explain the contextual absence of ‘method demonstration’ issues and the simultaneous presence of the ‘subjectivity’ issue.

4.3. Comprehensiveness as a proxy of the holistic nature of UX in design

As regards ‘comprehensiveness’, the key components (user, system and context) and dimensions (affective, cognitive and ergonomic) of UX have been counted following the criteria listed in Section 3. In Table 5, the number is summarised of papers that made UX key components (columns) and UX dimensions (rows) explicit. More elements were attributed when multiple key components and dimensions were made explicit in the articles. Table 5 also provides an overview of the different combinations of key components and dimensions. The number at the intersection of columns and rows represents the number of articles analysing those combinations. Under each column and row, there is the total number of each key component and dimension.

Table 5. Analysis and count of the ‘comprehensiveness’ issue. The number refers to the studies considering key factors (user, system, context) and UX dimensions (affective, cognitive, ergonomic)

The total number of dimensions is rather balanced, where ‘cognitive’ results the least considered, while ‘ergonomic’ is the most frequent. Many papers consider two dimensions at a time, but only one studies all the dimensions at the same time (Zuo et al. 2016).

On the other hand, the score for each key factor is not as balanced as for the dimensions. Indeed, ‘system’ emerges as the most considered key factor, while the least considered is ‘context’. Many papers study more than one key factor at once, but only two consider all the key factors at the same time (Hoyos-Ruiz et al. 2017; Yang et al. 2019).

The same imbalance appears when it comes to different combinations of key factor dimension. The least frequent is ‘context cognitive’ followed by ‘context affective’. Another unusual combination is ‘user cognitive’, while unsurprisingly the system is generally studied in combination with all the UX dimensions homogenously.

Among all the papers, Hoyos-Ruiz et al. (2017) are the closest to provide a holistic view of UX in design. The scholars consider all the key factors and two of the three dimensions. No paper considers all the three key components combined with all the three dimensions contextually.

To summarise, all the key factors and dimensions of UX are overall considered in contributions investigating UX in design. However, some key factors correlated with specific experiences are considered more frequently, as in the case of ‘system cognitive’ and ‘system ergonomic’, while other combinations are still mainly overlooked (‘context cognitive’ and ‘context affective’). Since ‘context’ is the most neglected key factor, future experimental UX studies should investigate the context insightfully in a cognitive perspective. Despite context has been partially overlooked, it has proven to be a crucial factor in studying products’ perceived value (Boztepe 2007), and judgements (Becattini et al. 2020), and more knowledge might give rise to effective means to design experiential contexts along with products (Petermans, Wim, & Van Koenraad 2013).

5. Further discussions and implications

5.1. Are UX outputs considered as an end goal in design only?

The authors have not assigned ranks, priorities and importance to the listed issues. It can be nevertheless remarked that this paper stems from the observed troublesome application of the UX concept in the design process. Otherwise said, it has been questioned whether the leveraging of UX in design is an objective per se or it can be considered as a tool to enhance what is designed. In this respect, some studies are worth highlighting that show a good experimental activity and present the possible benefits of UX within the design process clearly.

Some of those studies are more oriented to improve the design process through early user evaluations or co-working sessions where end users are not only directly involved for user requirements detection but play a role also during concept development. This is the case of Sleeswijk Visser & Visser (2006), where potential users served first to detect needs and were later employed in future-scenario techniques to perform and design the interaction between future users and the designed system. The scholars suggest also to ‘re-use’ the same sample of participants in different phases of an iterative design process. Pettersson (2017) used a similar ‘future-scenario-acting’ technique to analyse user expectations, understanding their characteristics and highlighting their changes over time. During their study, participants reflected and evaluated the concepts of future cars through enactment and drawing. Another good example of co-design between designers and potential users is provided by Chamorro-Koc, Popovic, & Emmison (2008), whose research outputs are summarised in design principles to guide the design of product usability. The scholars focused particularly on the forms of representation of a product concept. Yet, as for representation forms and technologic support media, the study of Ferrise et al. (2013b) is an example of employment of VR and mixed prototyping technology for multimodal interaction. In this study, user evaluation provided feedback for adjusting the flows of the product instantaneously. De Crescenzio et al. (2019) used VR too for concept evaluation and preferences detection to evaluate large-sized products such as aircraft cabin interiors at early stages of the process. Through VR, the scholars simplified the evaluation and the re-design process avoiding time and costs for creating physical prototypes.

Other studies involve users mainly to detect previously undisclosed custom requirements. Belz (2006) used UX outputs from the interaction between customers and a digital camera to find potentialities, flows and new users’ needs, preferences and behaviours to improve their product. More recently, McCarthy, Ramírez, & Robinson (2017) had the similar aim to identify users’ requirements, behaviour and preferences, but they studied UX from a more emotion-oriented perspective in the field of healthcare design. The identification of young users’ requirements while using a medical device for diabetes monitoring served designers to create several new device concepts. Each concept adapted to the different contexts and use situations highlighted by young users during preference detection. Lin et al. (2016) considered the context as a factor to be implemented in the design of products as well. The scholars focused on the analysis of the ‘cultural’ meanings of the context. The scholars used a specific cultural object to raise awareness of cultural pattern and identify hidden cultural meanings to be used as input to develop new concepts. Their study showed how it is possible to include cultural features into new products that can appeal young consumers.

All these examples show how UX can improve the product during its development through the detection of users’ needs and preferences. Some studies focus more on the iterative nature of the design process and therefore, their research output has been used to improve the design phases through iterations. This is highlighted in Torres et al. (2009), who present a case study to prove best practices for UCD in the military field. They developed a taxonomy whose information can be leveraged throughout the entire system to guide an end user in the maze of complex tasks during evaluation and prototype interaction. Liikkanen & Reavey (2015) used ‘resonance testing’ to collect evaluation feedback and revise the design concept iteratively during its development. Another example is the work made by Yoon, Kim, & Kang (2020). The scholars used retrospective techniques to analyse the dynamic nature of UX over time. This paper reveals how the patterns of positive UX in relation to a product vary over the usage life cycle, from before purchase to disposal/repurchase. Keeping in mind such information can be of paramount importance for designers to optimise the iterative process of the development of new products.

Overall, studying UX in design is not effortless, especially when the aim is to achieve scientific soundness. It is not easy to avail of a sufficient number of participants to form a representative population, collect both objective and subjective data though mixed methods and use the UX outputs to establish the next steps in the design process form. However, the contributions above can be used as valuable best practices despite some limitations.

5.2. Issues as a checklist for the planning of future UX-based design studies

Although the presented list of issues cannot be considered comprehensive, the highlighted issues have been justified both in a theoretical (through the scoping review of critical aspects in UX) and a practical perspective (through the systematic search of UX applications in design). These verified issues lead to the implication that upcoming studies can be designed in a way to minimise the number of known issues and their impact on the validity of research.

Otherwise said, future research involving the application of UX in design can benefit from the paper’s findings and use the issues as a checklist for designing experiments accurately. The process the authors propose to plan UX experiments in design is illustrated through the flow diagram made available in the Supplementary Materials a pdf document. The flow diagram guides a researcher through the evaluation of their current design of experiments and helps fix possible shortcomings by addressing all issues and identification criteria included in Table 1, as well as report insurmountable limitations due to peculiar contingency factors. The documentation of what of UX is actually studied, how this is done and the residual limitations allows future studies of UX in design to be effectively compared. In detail, the structure of the flow diagram sequentially foresees:

• • The definition of what is intended for UX, the designation of the willingness or need to address UX in a holistic way and the determination of the studied domains (issues ‘terminology’ and ‘comprehensiveness’);

• • The declaration of intended objectives and expected results of UX-based design studies (issue ‘method demonstration’);

• • The fine-tuning of the experiment through the sampling of participants, the use of required technologies, the setting and timing of interactions, the employment of the needed tools to extract the intended data (issues ‘instruments’, ‘participants’, ‘retrospective/projected’ and ‘subjectivity’);

• • The reporting of the experiment and how the design of the involved products is affected by the UX-based study and the general implications for design research (issue ‘clarity of design objectives’).

Obviously, the flow diagram is intended as a dynamic representation of recommendations to follow for a proper experimental UX-based study in design and the identification of additional issues can lead to its reformulation. The sequence of presented issues is proposed by the authors, but scholars might be willing to follow a difference logical order.

6. Conclusions

The paper has addressed how the umbrella concept of UX interacts with design. For this aim, the literature has been explored and reviewed in a twofold perspective. First, regarding issues in benefitting from UX understanding in design have been gathered and inferred through a scoping review. Second, regarding experimental papers dealing with UX and design have been analysed in relation to previously identified issues through a systematic search. Particular attention has been paid to operational aspects, that is, the clarification of how UX analysis affects or reorients the design process. Obviously, the presence of issues is not to be intended as a lack of scientific reliability. It just denotes research fragmentation, especially regarding theoretical issues and aspects to be considered to enhance the significance and the robustness of procedures studying UX to derive design implications. It is also evident that some issues might be intrinsic to the objectives of some contributions, which were limited and were not intended to develop repeatable procedures to streamline the use of UX in product design. In other terms, the issues are not to be interpreted as an universal system of basic requirements for all the possible studies of UX in design, though the authors urge scholars to consider the issues and if they could apply to their specific research. Moreover, scenarios can materialise in the future where scholars lack resources (monetary, equipment, know-how, time) to overcome pertinent issues; to this scope, the checklist suggests documenting the presence of limitations.

Markedly, in the authors’ intentions, the list of issues can be used as a guideline to improve the design of UX experiments for product development scopes, as fully documented in Section 5.2. This is particularly relevant because, based on the publication dates of contributions included in Table 4, there is no tendency to a decreasing number of issues in recent studies; this might be due to the increasing complexity of products and means of representation used in recent studies, which possibly leads to overlook other aspects. Therefore, the issues and guidelines could help make future works involving UX in design more comparable, which is a general research need besides an expected trajectory following the present publication.

As future work, the authors will build an experimental study based on the flow diagram described in Section 5.2 and reported in Supplementary Materials. In parallel, in order to increase the scientific value of the paper’s findings, the classification of issues in terms of comprehensives, clarity and non-redundancy might benefit from a validation made by experts in the field as a common practice; the same applies to the flow diagram.

The results of the present study highlight that UX is capable of revealing evaluations, preferences, unspoken needs and design requirements. While this kind of outcomes could be expected, one might have questioned whether the process leading from UX analysis to indications useful to designers is so straightforward. Actually, while many studies mention design, the actual contribution is not fully elicited, more concrete examples are available and commented on in Section 5.1. Here, some sources describing UX in design and showing the unique contribution UX outputs can provide are reported. The results also show that UX is relevant in different design phases. This seemingly supports the stance that sees UX as an umbrella concept and culture affecting design throughout its process and management thereof, for example, Minichiello, Hood, & Harkness (2018).

This work is expected to rise UX researchers and practitioners’ awareness towards the issues in UX research. The categories of issues are the original contribution of the paper and should guide UX researchers and practitioners in the identification of those aspects that are more relevant for their study and/or product development. In addition to that, the authors suggested some of the best practices to be followed as a guide.

While the paper is expected to encourage reflections into research on UX design and practical recommendations for design practitioners, it is not exempt from limitations. In this respect, the categorisation of issues is inherently subjective; it is also difficult to verify whether the categories of issues are comprehensive. However, the identified issues can be considered relevant and capable of distinguishing past contributions’ strengths given the large diffusion of both Y and N in Table 2. The classification process was guided by the criteria of Table 1, which allowed the authors to identify the presence and absence of issues in a straightforward way. The indications in the spreadsheet included in the Supplementary Materials are supposed to demonstrate that the classification process is sufficiently repeatable. An additional limitation is the fact that UX experiments in the field of software development were excluded from the analysis, although some sources from HCI have been used to infer UX issues. On the one hand, the number of UX studies for designing computer interfaces is huge and not manageable in a single paper. On the other hand, the field of HCI has its own peculiarities, especially regarding the followed design processes, and the gathered sources would have likely been hard to compare with examples from engineering and product design.