A conceptual framework for designing human-centered building-occupant interactions to enhance user experience in specific-purpose buildings

Abstract

This research proposes a novel conceptual framework that combines the concepts of Human-Computer Interaction (HCI) and Ambient Intelligence (AmI). The proposed framework aims to shed light on the importance of considering the needs and the social interactions of various building occupants in different types of buildings and designing HBI strategies accordingly. Specifically, we take educational buildings as a case that is less explored in the HBI research and apply the proposed framework, investigating how HBI strategies and interactions should be designed to address the needs of students, as primary occupants. Focus groups and semi-structured interviews were conducted among students in a flagship smart engineering building at Virginia Tech. Qualitative coding and concept mapping were used to analyze the qualitative data and determine the impact of occupant-specific needs on the learning experience of students. “Finding study space” was found to have the highest direct impact on the learning experience of students, and “Indoor Environment Quality (IEQ)” was found to have the highest indirect impact. The results show a clear need to integrate occupant needs in designing HBI strategies in different types of buildings. Finally, we discuss new ideas for designing potential Intelligent User Interfaces (IUI) to address the identified needs.

1. Introduction

With the advancement in technology, buildings are transforming into smart environments. Similar to smart cities, the smart building concept is beyond a single application, and should be regarded as an infrastructure of information that can serve different applications (Zubizarreta, Seravalli & Arrizabalaga 2016). In this sense, smart buildings can be seen as one of the essential building blocks of a smart city infrastructure (Oliveira & Campolargo 2015; Al Sharif & Pokharel 2022). By leveraging advanced technologies and data-driven solutions, smart buildings contribute to the realization of smart city objectives, including sustainability, efficiency, resilience, and quality of life improvements (Ahvenniemi et al. 2017; Giffinger et al. 2007; Nam & Pardo 2011; Hollands 2020). For example, smart buildings generate vast amounts of data related to energy consumption, occupancy patterns, indoor air quality, and building performance. This data can be shared and integrated with citywide data platforms, enabling holistic analysis, decision-making, and resource allocation at the city level (Caragliu, Del Bo & Nijkamp 2011). To this end, smart buildings are equipped with sensors to collect different types of data from the building itself, e.g., building temperature, humidity, occupancy, and its occupants, e.g., their comfort levels. These sensor nodes are connected through sensor networks, and transfer the collected data to building control systems. Control systems then make decisions for actions to improve a process/condition in the building depending on the goal of the system, e.g., improving building energy efficiency. The decisions made by control systems are executed through output devices (actuators). Such systems usually operate autonomously, and human involvement is restricted to limited building user-interfaces (UIs). Smart homes can be considered as an increasingly rising application of smart buildings, offering a wide range of applications aimed at enhancing convenience, comfort, energy efficiency, and security of occupants (Al-Fuqaha et al. 2015; Catarinucci et al. 2015; Costanzo et al. 2017; Gardiner et al. 2016; Lee & Hong 2019; Liu, Lee & Chen 2017; Zhang et al. 2019).

With the objective of increasing human (occupants and building managers) interaction with smart buildings and enabling their engagement in the design, operation and maintenance of buildings, a novel research area “Human-Building Interaction (HBI)” has emerged (Jia, Srinivasan & Raheem 2017). HBI is a convergent field that represents the growing complexities of the dynamic interplay between human experience and intelligence within the built environment (Becerik-Gerber et al. 2022). Previous works attempted to define HBI through the lenses of two main prevailing areas of research, namely, Human-Computer Interaction (HCI) and Ambient Intelligence (AmI). Human-Computer Interaction (HCI) encompasses the study and refinement of computer systems and technologies to enhance the way people interact with them. It delves into comprehending and enhancing the synergy between humans and computers, aiming to render technology more user-friendly, efficient, and engaging (Boy 2017; Carroll 2003). The HCI-based approach defines HBI as the application of HCI in the domains of architecture and urban design, to enable efficient occupant interactions with the built environment, focusing on human values, needs, and priorities (Alavi et al. 2019). On the other hand, Ambient Intelligence (AmI) refers to a paradigm in computing and artificial intelligence (AI) aiming to develop smart environments that are sensitive, adaptive, and responsive to the presence of people. The goal of the AmI approach to HBI is to enhance users’ experience by embedding intelligence into their surroundings, enabling context-aware and proactive interactions. It makes use of ubiquitous computing devices in buildings to add various capabilities such as awareness of occupant needs, intelligent interaction with occupants, forecasting occupant behavior, and taking necessary actions (Nembrini & Lalanne 2017). While HCI and AmI share common goals of improving human-technology interaction, they differ in their application when applied to HBI. HCI tends to focus on digital interfaces and systems, while AmI extends this to include interactions within physical spaces by embedding intelligence into the built environment.

HBI and the bilateral impacts that a building and its occupants’ behavior have on each other have been a point of interest in recent research. Alavi et al. (2019) divided the scope of HBI research into three categories namely: “People/User,” “Built Environment/Building,” and “Computing” presented as three concentric circles. Their proposed diagram has three interrelated dimensions of “Physical,” “Spatial,” and “Social” which describe various HBI research directions (Figure 1). The “Physical” dimension represents the environmental conditions such as temperature and humidity; the “Spatial” dimension represents attributes such as building space utilization, and the “Social” dimension represents collaboration between building occupants such as indoor traffic management. These aspects are correlated to each other and can be combined to propose new research and design directions. For instance, a combined example of “Spatial” and “Social” can be demonstrated by smart collaborative workspaces (Alavi et al. 2019).

Figure 1. HBI scope-dimension diagram [5].

Current HBI applications mainly fall under the physical and spatial dimensions of HBI implementation. The physical applications mainly focus on equipping buildings with building automation systems (BAS) to automate various building operations such as heating, ventilation, and air conditioning (HVAC), lighting, etc., to either improve energy efficiency or increase occupant comfort. The spatial stream of HBI research focuses on building space management and enhancing occupants’ safety by facilitating interactions between occupants and buildings. The current BASs are mostly energy-focused (Shen et al. 2016) and aim to enable facility managers to access building systems through controllers and interaction points. AmI-based HBI systems add to conventional BASs by including an intelligent layer in addition to the operational layer considered in BAS (Figure 2). The operational layer in AmI systems works similar to BASs and consists of operating systems, databases, ubiquitous computing, and so forth The added benefit of building AmI systems over BASs is the presence of an intelligent layer that uses computational methods such as Machine Learning (ML), Natural Language Processing (NLP), and so forth to perform predictive analytics (Green, Heer & Manning 2015; Lieberman 2009). Similar to BASs, the data required for conducting computations in AmI systems is collected from occupants and building environments using sensors. AmI systems make use of the collected data and take necessary actions automatically by using output devices such as actuators and robots (Dongre & Roofigari-Esfahan 2019; Almeida et al. 2018).

Figure 2. Ambient Intelligence (AmI) framework.

Current HBI practices focus on enabling building-centric improvements and can be grouped into three main aspects: energy efficiency, occupants’ comfort and safety, and space management:

Energy efficiency: The application of Human-Building Interaction (HBI) has proven to significantly enhance energy efficiency in buildings and has been extensively studied (e.g., Yan et al. 2015; Yang, Jia & Guan 2016; Grabe 2016; Chen et al. 2017; Abuimara et al. 2019). Understanding how occupants interact with building systems and spaces can provide valuable insights into energy usage patterns (Heydarian et al. 2020). In this approach, HBI technologies, such as occupancy sensors and smart meters, are used to track the occupants’ behavior, preferences, and schedules. Analyzing this data allows building managers to identify opportunities for energy savings by optimizing heating, cooling, lighting, and ventilation systems based on the occupancy patterns. For example, Wang & Heydarian (2019) proposed an approach to collect and analyze psychological and environmental data to build occupant behavior models and pair them with targeted interventions to increase energy efficiency. Jia et al. (2017) used agent-based modeling (ABM) to develop occupant behavior models from the data collected using various sensors and surveys to improve the accuracy of energy estimation. Abraham, Anumba & Asadi (2017) used ML methods to train the occupant behavior-related energy utilization data and predict the energy consumption patterns of a building. A socio-technical energy management system by the name “BizWatts” was developed by Gulbinas, Jain & Taylor (2014) to save energy by providing real-time, appliance-level power management and socially contextualized energy consumption feedback to the occupants. The authors further expanded BizWatts to understand the impact of organizational occupant behavior on energy savings (Gulbinas & Taylor 2014). Ahmadi-Karvigh et al. (2018) developed algorithms to recognize user activities at peak and non-peak hours to estimate the energy wasted inside a building. He et al. (2022) investigated how conversational interactions through proactive virtual assistants in a simulated smart home ecosystem can influence occupants to take energy-saving behavior.

Occupant Comfort and Safety: Over the past few decades, attention in buildings’ design and operation has gradually shifted from promoting only energy efficiency objectives to also addressing human comfort and well-being (Soleimanijavid, Konstantzos & Liu 2024). Researchers developed a wide range of control algorithms ranging from rule-based controls to complex learning approaches that can fully capture occupants’ personalized preferences in smart buildings. This stream of work leverages technology to create more personalized and responsive building environments to optimize comfort while minimizing energy usage, e.g., dynamically adjusting indoor environmental conditions, such as temperature, lighting, and ventilation, to meet occupants’ preferences and comfort requirements (Andargie, Touchie & O’Brien 2019; Day et al. 2020). Occupants can use interfaces such as smartphone apps or voice-controlled assistants to adjust temperature, lighting, and other building parameters according to their preferences. By giving occupants greater control over their environment, this approach promotes energy-efficient behavior without compromising comfort. For example, Nguyen et al. (2010) developed a system that deals with advanced load management strategies and real-time wireless communication techniques to reduce peak consumption while maintaining thermal comfort. Alavi et al. (2017) proposed a schematic model of comfort and demonstrated an interactive tool called “Comfort Box” that collects subjective feedback from occupants about their perception of comfort in buildings. Jazizadeh, Marin & Becerik-Gerber (2013); Jazizadeh et al. (2014) used the building data obtained from sensors and the occupant data obtained from wearable devices to develop personalized thermal comfort prediction models. Similar models were also used by Li, Menassa & Kamat (2017) to determine optimum HVAC control strategies for buildings.

Space management: In another stream of HBI research, building and occupant data were used to make the best use of building spaces. Verma, Alavi & Lalanne (2017) used sensing and participatory data to understand how occupants use spaces in a building, aiming at optimizing building space utilization (Li et al. 2017). “Twitter Bots,” autonomous tweeting robotic agents, were used to engage occupants in a process of providing everyday feedback about space usage (Verma et al. 2017). Understanding occupants’ behavior in building emergencies can also help in designing safer buildings (Mitchell Finnigan, Clear & Olivier 2018). To ensure occupant safety inside buildings, Cheng et al. (2016) used building sensor data and a smartphone application to guide the occupants in case of fire emergencies (Liu & Becerik-Gerber 2022). Chen, Liu & Wu (2018) utilized sensor data to guide firefighters to quickly locate the fire inside a building, using LED light indicators placed at various locations in the building (Lin et al. 2020). Other researchers also use innovative approaches in designing interactive building elements, including walls, roofs, windows, doors and facades which are essential building blocks for homes and offices. Instead of the conventional static nature of these elements that aim to only constitute the “frame” fora building, this stream of work proposes the use of interactive technology to make some of these building elements more dynamic and adjustable (e.g., Streitz, Geißler & Holmer 1998, Abo Elenien et al. 2015, Bader et al. 2019, Okur & Karakoç 2019).

2. Proposed extended scope-dimension diagram

Mapping the current HBI research directions on the scope dimension diagram presented in Figure 1 shows less attention to the social direction of HBI developments, denoting a need to consider social interactions between occupants in providing a pleasant occupancy. Additionally, the scope-dimension framework presented in Figure 1 does not fully consider the inter-relationships between the three HBI directions, e.g., how social-physical dimensions should be combined to address occupant interaction needs while improving the energy efficiency of the buildings. As a result, we developed an extended HBI scope dimensions diagram to better understand HBI research’s current and future directions (Figure 3). To this end, firstly, we expand the scope of the HBI research presented in Figure 1 by considering different factors such as level and types of occupant-building interaction. The scope “People” denotes involvement of the occupants in HBI and can be collected through the participatory involvement of occupants or the use of sensors. In participatory involvement, occupants are contributing via surveys, questionnaires, and so forth (active engagement), whereas sensory involvement includes occupant data collection via sensing technologies (passive engagement). The scope “Built Environment” denotes the level of interaction that occupants have with the building. These interactions can be basic, e.g., the opening and closing of a door, or smart, e.g., the use of interfaces to collect data about perceivable comfort levels of occupants or regulating the Indoor Environment Quality (IEQ) parameters inside the building. The scope “Computing” denotes the level of analysis performed on the collected participatory and sensory data and can be operational or intelligent. Operational computing is supported by operating systems, communications, databases, ubiquitous computing, etc., while intelligent computing is supported by ML, NLP, Computer Vision, etc.

Figure 3. Proposed HBI scope-dimension diagram.

Existing HBI literature, when mapped on the proposed diagram, reveals interesting insights. The mapped HBI research is shown as oval shapes on the proposed scope-dimension diagram shown in Figure 4. As previously mentioned, common HBI applications include improving energy efficiency, occupant comfort, and safety and space management. As can be observed in the Figure 4, energy-driven applications (to increase energy efficiency) lean towards physical aspect of physical-social interactions and are more widely studied to cover a range of interactions and applications. The comfort-driven studies lean more towards the social aspect of social-physical interactions representing those developing human behavior models to control energy consumption and thermal comfort. Safety-driven studies, e.g., indoor real-time location sensing (RTLS) for emergency navigation, cover the physical-spatial dimension. The studies in this category have a limited scope, using sensory data received from indoor RTLS, and not including participants’ subjective feedback through participatory data and intelligent computing. Finally, space-driven research, e.g., the use of RTLS to increase the spatial comfort of occupants, denotes the oval shape in the spatial-social dimension.

Figure 4. Existing HBI applications mapped on the proposed scope-dimension diagram.

Mapping existing applications on the proposed scope-dimension diagram more specifically denotes the extensive focus of the literature on improving building efficiency as opposed to addressing occupant needs, and presents a pressing demand to investigate the social-spatial and the physical-spatial aspects of HBI. While current HBI design focuses on either the building processes or the pursuit of outcomes from the managerial side, it tends to forget that humans are not just another resource (Ehkirch & Matsumae 2024). As mentioned before, such social interactions are highly dependent on the type of building, which dictates the functionality of its spaces as well as the demographics of occupants and their interaction goals. The main focus of the current HBI research on specific-purpose buildings is again improving energy management in different types of buildings such as residential (Yu et al. 2011; Chen et al. 2015; Tak et al. 2023), healthcare facilities (Mallak et al. 2003, Kalay & Schaumann 2021), offices and commercial buildings (e.g., Rijal, Humphreys & Nicol 2009; Duarte, Den Wymelenberg & Rieger 2013). Limited research also explored occupants’ specific needs in these building types; A study investigated the impact of ambient lighting on patients’ well-being and recovery rates in hospital environments. Other researchers proposed the use of interactive digital displays (Taher et al. 2009) and Smart Desks (Aryal et al. 2019) and chairs (Labeodan et al. 2016) in office environments to promote collaboration and productivity among employees.

3. Research objectives

As we delve into the complexities and possibilities of HBI, it becomes evident that understanding the dynamics of human-building interactions is not only about designing smarter buildings but about reimagining the relationship between occupants and their built surroundings. Understanding occupant needs is essential to designing smart buildings that meet the demands of their occupants and provide optimal comfort, safety, and functionality. Occupant needs in different types of buildings vary based on factors such as the building’s function, user demographics, and environmental conditions. While Human-Building Interaction (HBI) research has made significant strides in understanding and addressing the needs of building occupants, several limitations persist, including:

Limited understanding of user preferences: HBI research often relies on generalized assumptions about user preferences and behavior, which may not accurately reflect the diverse needs and preferences of building occupants;

Generalization of findings: HBI research often focuses on specific building contexts, leading to findings that may not generalize well across different types of buildings. For example, solutions that work well in office buildings may not be applicable or effective in educational or healthcare settings due to differences in user needs and building functions;

Complexity of occupant needs: different types of buildings serve diverse occupant populations with varying needs and preferences. HBI research does not adequately capture the complexity of these needs and design solutions that address them comprehensively.

As such, there is a pressing need for studying occupant demands in specific purpose buildings, including educational buildings, to tailor HBI strategies for each building type based on their occupants. Educational buildings are one of the important specific-purpose buildings that play a vital role in the community. Their architectural design and the environment that they create not only impact the students’ learning outcomes (Mulrooney & Kelly 2020), but also can be an ideal location for community engagement (Cureton & Gravestock 2019). Current research overlooks the potential of HBI implementation in educational buildings in creating learning environments that are inclusive, engaging, and supportive of student success, thereby addressing the social dimension of HBI implementation for these buildings.

We hypothesize that addressing occupant-specific needs in buildings can have a positive impact on the experience of the primary occupants, including improvements in the learning experience of students in educational buildings. To address the aforementioned gaps, the objective of this research is to promote an occupant-centric approach towards designing HBI strategies for different types of buildings. We explore occupant needs and bidirectional dynamics of occupant-building interactions in educational buildings. First, we develop a novel HBI framework that combines HCI and AmI to enable bidirectional and multi-modal interactions between occupants and buildings. We then investigate the application of the proposed framework in enhancing the learning experience of students (as primary occupants) in educational buildings. A proof-of-concept qualitative study was conducted to explore the potential of the proposed approach in understanding and addressing the needs of occupants in educational buildings. Finally, we propose viable interventions to mitigate the identified occupant-specific needs in educational buildings.

4. Proposed extended human building interaction (HBI) framework

For a long time, HCI and AI were considered divergent fields of study because HCI encourages human involvement to make UIs more user-friendly whereas AI aims to construct human-like intelligence by using mathematical models (Cheng et al. 2016). However, researchers from the HCI community and AI community are now coming together to improve the interaction between humans and AI systems by designing intelligent user interfaces (IUIs) (Chen et al. 2018). Voice and text-operated assistants such as Apple’s Siri, Google Assistant, Amazon’s Alexa and ChatGPT are some examples of IUIs. Similarly, the HCI and AmI-based definitions of HBI can be combined. In this research, we propose a novel vision for an HBI Framework that makes use of this integration, aimed at creating IUIs for buildings, as shown in Figure 5.

Figure 5. Proposed approach to Intelligent User Interfaces (IUIs) design for HBI.

The overlap between HCI and buildings represent augmented buildings with intelligent interfaces that are designed to facilitate interaction with occupants. The overlap between AI and buildings represents buildings that have AI-based systems to intelligently control building operations. Our vision of HBI lies at the intersection of HCI, AI and buildings, and focuses on improving the interactions between occupants and buildings through designing IUIs for buildings. This vision aims to facilitate real-time, bidirectional, and multi-modal interactions between the occupants and buildings. Figure 6 illustrates a model of the proposed HBI framework that is comprised of an intelligent layer to perform predictive analytics, and IUI to facilitate real-time, bidirectional, and multi-modal interactions between the occupants and the building. In this framework, data required for computational analysis is collected from occupants through both passive engagement (i.e., sensing) and active engagement (occupant participation). Participatory data ensure the active involvement of occupants in the HBI system. Similarly, data from the buildings is collected through real-time sensing as well as historical data available in the building databases, as opposed to conventional sensing as used in AmI systems.

Figure 6. Left: Proposed Human Building Interaction (HBI) framework; Right: occupant-building data pool.

Building databases required for this purpose contain all essential data about the building that cannot be measured using sensing techniques, including as-built documentations such as plans, operation manuals, and so forth Similar to the AmI systems, the controller in the proposed HBI framework has an intelligent layer that uses AI to forecast occupant behavior and how predictions should be translated to actions in the building. The decisions made by the controller are translated into useful information and automatic actions. These automatic actions benefit both the occupants and buildings, as opposed to benefiting the building only, by providing context-aware information or guidance to occupants or actuating specific sensors in the building. The information is then conveyed to the occupants through multi-modal interactions such as audio, touch, gestures, smell, and so forth (Chen et al. 2018). Such closed loop of interaction between buildings and their occupants increases the efficiency of the building, while ensuring enjoyable and productive occupancy by the users.

5. Application of the proposed HBI framework for educational buildings

The proposed HBI framework takes a human-centric approach in establishing bidirectional, and multi-modal interactions between occupants and buildings that are tailored to their needs. The proposed framework can be generally applied to different building types. The following steps map the approach to implement the proposed framework for each building type: (1) identify the building type and its primary occupants; (2) identify the primary occupants’ specific needs and preferences; (3) identify the data types required for addressing the identified needs and the respective data acquisition methods; (4) collect and analyze data and extract the required knowledge to make automated decisions based on the application; and lastly, (5) design HBI Intelligent User Interfaces (IUIs) to convey the decisions to the occupants.

In this section, the application of the proposed framework to enhance the learning experience of students, as primary occupants, in educational buildings, is presented through a case study that was carried out at Virginia Tech. To this end, first, the primary occupants and their needs were identified. The impact of the identified needs on the learning experience of students, identified as primary occupants, were then evaluated. The existing BAS in the selected building (Goodwin Hall) was also assessed. Finally, potential technologies to complement the current BAS to achieve the added benefits of the proposed HBI framework to improve the learning experience of students were evaluated and suggestions were provided.

5.1 Identify building type and primary occupants

The occupant-centric approach taken in this research implies that before any HBI strategy can be designed, the type of the building and its primary occupants should be identified. This is considered as the first step towards addressing occupant needs in interacting with different types of buildings. This is because an occupant-building relationship is dynamic and can change depending on the function of the building and the purpose of occupancy. We present three scenarios to explain this phenomenon: (1) Different occupants might have different demands from the same building type, e.g., a student might need information about the study resources in a university building whereas a facility manager might need to know about the energy consumption of the building. (2) Different occupants might have the same needs from the same building type, e.g., both student and facility managers might want to know available parking spaces near the building. (3) The same occupants might have different needs from the same building type; e.g., one student might need information about the study resources, but another might want to know about the events happening in the building. Ideally, all building occupants should benefit from the improved interaction with buildings, but priority should be given to addressing the needs of primary occupants to offer optimum occupancy.

5.2 Identify and analyze occupant needs

As explained through the proposed extended scope dimension diagram, occupant needs can be identified through collecting participatory data from occupants as well as monitoring their behaviors using sensing technologies. Participatory data collection involves conducting semi-structured interviews, focus group discussions, surveys, and so forth where occupants are directly questioned about their needs in a building. Sensing techniques such as thermal cameras can sense if an occupant is showing signs of thermal discomfort. When collecting participatory and sensory data from occupants, occupant concerns such as data privacy and protection should also be acknowledged. Addressing privacy issues requires implementing privacy-by-design principles, adopting transparent data practices, and ensuring compliance with privacy regulations and standards. Building occupants must be informed about the data collection and usage, be provided with options to consent and have control over their personal information, and be given recourse for addressing privacy concerns or data breaches. Additionally, other stakeholders, including building owners, designers, and technology providers, must collaborate to develop ethical guidelines, privacy policies, and technical safeguards that protect occupants’ privacy while enabling the benefits of HBI technologies in the building.

5.3 Identify data types and collect and analyze the data

The first step in leveraging data-driven insights is collecting information from different entities in the building, including the occupants and building elements. The concept of data pool in the proposed framework (Figure 6) is adapted from the concept of Data Hungry Homes (DHH) (Lee-Smith et al. 2019). DHHs do not simply collect, create, use, or transmit data but are hungry for it and need routine data feeding to function. Once the occupant needs are identified, the next step is to identify various data types that are currently being exchanged or should be potentially exchanged between occupants and buildings to address the identified needs. A data type may be specified for many reasons: similarity, convenience, or to focus attention. The data type in this research is categorized based on the information that can be extracted from the data, e.g., location of occupants, or vibration of building floors. “Occupant data types” can be described as the data collected from the occupants based on their needs, and “building data types” can be described as the data that has to be collected from the building to address the needs of its occupants. The types of sensory data that are currently being exchanged can be identified by examining the existing infrastructure and BAS of the building. A thorough understanding and collection of all the required data types will lead to the development of an “Occupant-Building Data Pool” that gives an estimate of the existing and potential occupant-building relationships (Figure 6). The length of the “Interaction Arrows” denotes the efficiency of interaction between the occupants and buildings. The area of “Data Pool Circle” denotes the amount of data shared between the occupants and the building. By increasing the amount of data sharing, the area of the circle will be bigger and consequently, the length of the arrows i.e., efficiency of interaction, will increase.

As mentioned earlier, the collection of data for the identified data types can be via sensing and participation from the occupants and via sensing and databases from buildings. Commonly used technologies to collect “occupant data type” include Infrared (IR) tags, Radio Frequency Identification Devices (RFID) tags, Bluetooth Low Energy (BLE) tags, Ultra-Wide Band (UWB) tags, cameras, mobile phones, smart bands, and interfaces that document real-time feedback. The sensors that collect “building data type” include energy meters, temperature sensors, relative humidity sensors (RH), light sensors, sound sensors, smoke sensors, CO₂ and CO sensors, and so forth Technologies such as Building Information Modeling (BIM) can be used to host the database of the building. The selection of technologies for collecting the required data depends on the existing sensing infrastructure of the building. If the existing infrastructure does not support the collection of a data type, then the building should be efficiently equipped with an appropriate sensing technology to collect the required data-type.

5.4 Increase building “affordances” through designing intelligent interfaces

Don Norman in his book “The Design of Everyday Things” (Norman 2013), describes how difficult it was for him to operate a building element as simple as a door. Since modern buildings have more complex elements than just doors, the struggles occupants might face when operating such elements can hinder their experience in the building. To minimize such struggles, intelligent interfaces should be designed to maximize building element affordances. The concept of affordances was defined by Gibson (1977) as the information about how people could interpret and interact with an object. Gibson states that affordances are independent of perception, meaning perceptual information is always stored in an object even if it is not perceived by the users. Norman emphasizes the importance of perceivable affordances and uses the term “signifiers” to indicate perceivable affordances. Traditionally, building designers have relied on semiotics to show the necessary information to the users inside a building. However, there are several underutilized and hidden affordances in a building that can be exploited by using advanced interfaces such as Augmented Reality (AR) and smart building-informed artifacts. Building artifacts, when equipped with emerging technology, can act as interaction channels between the occupants and buildings. There is scant research considering affordances in HBI approaches. The concept of “smart desks” was presented to address individual indoor environment preferences by using sensors to monitor the environment and ML to learn occupant preferences (Alavi et al. 2017). Rule-based chat-bots were also developed to manage plugged-in appliances through smart plugs in an office environment to involve occupants in the building’s energy management and encourage energy savings. Another example is the use of programmable robots to control physical objects and reconfigure spaces in buildings, also called programmable environments, used to create modular and reconfigurable rooms (Satu et al. 2018).

The intelligent interfaces proposed here go beyond knowing the needs of the occupants to improve the efficiency of the building, and instead aim to create a bi-directional interaction between occupants and buildings to address both their needs. For example, in our case study, students were found as the primary occupants of the educational buildings based on the time they spent in the building. On the other hand, finding study resources was found to be the main concern/need of students in a university building. Finding study resources can involve finding designated study spaces in the building, exploring the study spaces with unoccupied furniture, and in higher levels of affordance, filtering spaces based on preferred level of noise, type of lighting, and ultimately being navigated to the space found as a result of the search. Such a simple search will require the following data to be available for decision-making: 3D building models that specify types of spaces in the building (available in the building database), smart furniture that sense being occupied (augmenting building furniture with weight sensors), sensors that collect occupancy levels (e.g., vibration or motion sensors), lighting and noise levels. These sensors should communicate the collected data with the building pool. Finally, there needs to be interfaces (in the building or on mobile phones) to allow occupants (students in this case) to filter spaces based on their preference and navigate them to the desired space. For example, a student can search for study spaces on a touch screen at the entrance of a university building, filter the spaces based on their availability and type of furniture, find heat maps of each of the filtered spaces to realize the occupancy, noise and lighting level of the spaces, select one, and be navigated to the selected space.

6. Case study: enhancing the learning experience of students in Goodwin Hall

In this section, a step-by-step application of the proposed framework to enhance the learning experience of the students in Goodwin Hall, as a representative technology-enhanced building at Virginia Tech, is presented.

6.1 Goodwin Hall building type and its primary occupants

Goodwin Hall is an academic building at Virginia Tech with 160,000 sq ft of space. GH consists of 5 floors in an L-shaped design. It is the flagship smart building which houses 40 instructional and research labs, 8 classrooms, an auditorium, and 150 offices for several engineering departments making it the main home for the engineering students (mechanical and chemical). Some of the courses are open to other majors requiring large classrooms, which adds a transient population of students occupying the building daily. The first floor hosts the classrooms and the auditorium that are usually used for undergraduate engineering classes (Figure 7a). The upper floors have the same footprint and host laboratories and faculty/ staff offices (Figure 7b). Goodwin Hall has around 240 accelerometers attached to 136 sensor mounts throughout the building’s ceilings that can measure all vibrations made inside the building. It is also equipped with temperature and CO2 sensors to regulate the indoor climate.

Figure 7. Goodwin Hall’s (a) first floor plan, and (b) 2–5th floor plans.

The primary purpose of any academic building like Goodwin Hall is to provide an adequate and efficient learning experience for the students. Primary occupants of Goodwin Hall include undergraduate and graduate students whereas the secondary occupants are faculty members and administrative staff. Considering the number and capacity of classrooms, laboratories and graduate offices and the average working hours for graduate students (~15–20/per week) as well as an average of 15 credit hours for undergraduate students, the students are considered the primary occupants of the building. Other occupants include faculty/staff, and temporary occupants including visitors participating in events held in the auditorium.

6.2 Occupant needs in Goodwin Hall

To identify the students’ needs and preferences while occupying the building, we conducted focus groups and semi-structured interviews. Qualitative analysis was conducted to identify important categories of needs. These categories were grouped using concept mapping to identify various themes and relationships among categories, codes, and subcodes.

6.2.1 Qualitative data collection

Focus groups and semi-structured interviews were conducted among 12 undergraduate and 7 graduate students. Although qualitative data collection methods (in this case study, in-depth focus groups and semi-structured interviews) are perceived as less rigorous compared to quantitative methods, they are known to offer unique strengths and advantages for exploring the complexity, and diversity of human experience and perspective (Denzin & Lincoln 2018). During focus groups and semi-structured interviews, the participants were questioned about their positive and negative experiences in the building, potential applications and advantages of building IUIs for improving their learning experience, as well as their preferred interaction modalities and potential data privacy concerns. All focus groups and semi-structured interviews were audio-recorded with consent from the participating students.

Designing a qualitative research plan and defining its structures and attributes (e.g., sample size) is impacted by various aspects such as research topic, cultural content, the scope of the research, and access to diverse participants (Marshall et al. 2013). While previous studies show that there is no fixed equation for calculating the right number of participants in a qualitative study, there are some measuring factors that can help the researchers define and justify their sample size. Research shows that in qualitative research, small sample sizes allow for in-depth analysis of individual cases, enabling researchers to explore complex phenomena with rich detail (Mason 2018). As such, the sample size can be made based on joint evaluation of epistemological and practical concerns that are derived from preliminary research and during the study preparation (Robinson 2014). “Data saturation” (Glaser & Strauss 2017) is also a concept that identifies the point in a study where no additional data or new knowledge is being collected. In our case, the final three interviews demonstrated repeating data, where many shared experiences similar to the previous interview and focus group sessions were emphasized by the participants and no new knowledge was added.

6.2.2 Qualitative data analysis

The collected audio data was transcribed to text, and qualitative coding including In-vivo and Focused tools were used to identify the main occupant need categories, codes, and subcodes from the raw text data. Initially, audio-to-text transcription was done using web-based services that resulted in a total of 12 transcribed documents. These documents were manually checked to remove possible errors in transcription. In-vivo coding was used to assign labels to the data collected from the focus groups and semi-structured interviews. It uses words or short phrases from the participants’ sentences in the transcribed data as codes. Focused coding was further used to categorize the most frequent or significant codes and subcodes and develop categories. Qualitative coding was conducted using MAXQDA software. Two iterations were conducted to validate the reliability of the coding that resulted in a total of 580 In-vivo codes. In the first iteration, 228 In-vivo codes were identified and in the second iteration, the number of In-vivo codes was increased to 352. Both iterations were performed by the same researcher with a gap of 2 weeks to account for the potential bias in qualitative coding. Intra-coder reliability tests were performed on the In-vivo codes from the two iterations to check their validity. Percent agreement and Cohen’s Kappa statistics were conducted to check the intra-coder reliability of the In-vivo codes. The purpose of intra-coder reliability is to improve the quality of the identified In-vivo codes. The result of the intra-coder reliability test is shown in Figure 8.

Figure 8. Percent agreement and Cohen’s Kappa statistic for intra-coder reliability test.

Focused coding was performed on the second iteration to group significant In-vivo codes into categories, codes and subcodes. Focused coding has its roots in grounded theory (Giles, de Lacey & Muir-Cochrane 2016) and is used to identify the codes that are relevant to the research themes. Grounded theory comes into play where data collection and analysis happen simultaneously. This informs the collection process of what to collect and the analysis process of what is relevant and what needs further analysis (Flick 2013). In our case, focused coding helped recognize the codes that were not relevant to further processes and thus 51 codes were dropped. As a result, 301 of the 352 In-vivo codes were considered for focused coding and the rest were discarded due to their insignificance to the research theme. The remaining 301codes went through another round of coding and were assigned weights according to their frequency of occurrence. All the categories, codes and subcodes were assigned a weight from 1–10 based on their frequency of occurrence and the emphasis given by the participants in the focus groups and semi-structured interviews (Table 1).

Table 1. Frequency and weight of categories, codes and subcodes identified from focused coding

6.2.3 Results of qualitative data analysis

Concept maps were created to explore relationships among the identified codes and subcodes to assess their impact on the learning experience of students in the building (Figure 9). The key themes that emerged from the concept mapping exercise were “Navigation,” “Learning,” “Comfort,” “Stress Management” and “Safety.” The theme “Navigation” includes the codes “finding professor,” “finding parking,” “traffic management,” “indoor navigation,” and “finding study spaces.” “Learning” consists of codes “finding study spaces” and “study resources.” “Comfort” is concerned with “audio,” “visual” and “thermal” comfort levels of the occupants when conducting primary activities such as attending classes or studying. “Stress management” includes “receiving notification about the events happening in the building.” “Safety” includes the protection of occupants in case of an emergency such as fire or shooting inside the building. A summary of the themes with their cumulative weights is shown in Table 2. The theme with the highest cumulative weight is “Comfort,” followed by “Navigation” and “Learning.”

Figure 9. Concept map showing code relationships of category “Occupant Needs.”

Table 2. Themes and cumulative weights in category “Occupant Needs”

The codes “finding study spaces” and “study resources” were each found to have a direct impact on learning; Many undergraduate students highlighted that even though Goodwin Hall has a lot of open spaces, it is difficult for them to find an appropriate place to study in the building. Finding study spaces is a common problem that many undergraduate and graduate students usually face in educational buildings and libraries. Students also expressed the difficulties they faced when searching study resources such as proper furniture, power outlets, and whiteboards in the buildings. The most important code observed in Figure 10 is “comfort” and although it does not directly contribute to the learning experience, it has a significant indirect impact on enhancing the learning experience by providing students with a comfortable environment. “Comfort” was a primary concern for the graduate students who spent more time in Goodwin Hall. They complained about the lack of windows in their labs and how it makes them feel disconnected from the outside world and even claustrophobic. Lack of “audio comfort” was a problem for both undergraduate and graduate students, the undergrads preferred “study spaces” that were less noisy, and the graduate students complained about being disturbed by noises while working in their labs. “Thermal comfort” was also important for all students, they demanded more control over regulating the indoor temperature especially in offices. “Stress management” was a concern for graduate students because they spend a lot of time in the building and need some means to help them relax in between their studies. A possible solution for “stress management” was identified as “event notification,” to inform the students about the events happening in the building so they can attend in their free time. “Finding professors” was also a concern for the graduate students because most of them are working as graduate assistants with a professor and sometimes face difficulty to locate their respective professors. “Traffic management” is related to “finding parking” as well as receiving information regarding building occupancy at different times. Providing “safety” denotes protection in cases of possible emergencies such as fire and intruders. All participants were concerned about their data privacy and security but were willing to share their location and schedule data as long as the shared data remains anonymous and does not include any identifiers.

Figure 10. Relative importance percentages of identified codes on learning experience.

6.3 Identify and collect “Data Types” from Goodwin Hall

The “occupant data types” and “building data types” identified from the focus groups with primary building occupants were then categorized based on the main need codes. The required sensing and interaction strategies to address the occupants’ needs regarding the theme “navigation” are shown in Table 3 as an example. It can be observed that most of the “building data types” are concerned with location, which can make use of Goodwin Hall’s accelerometers. A downside of accelerometers is that they can only determine the location of moving objects and not stationary objects. For the location of stationary objects, sensors such as RFID, BLE, and Wi-Fi can be used.

Table 3. Data collection and interaction strategies for theme “navigation.”

Similarly, within the theme “Learning” “building data type” for the “Study Resources” and “Finding Study Space” which are the two identified codes with direct impact on “Learning Experience” include building layout, furniture locations and availability, power outlet and whiteboard locations, and so forth Currently there is no existing “building data type” or collection means available for any of the mentioned codes. “Building data types” for “Comfort” include luminance, noise level, and temperature and these are measured with light sensors, microphones, and temperature sensors, respectively. “Occupant data types” for “Comfort” include location and subjective feedback through surveys. These along with sensing techniques and interaction strategies for “Comfort” are shown in Table 4.

Table 4. Data collection and interaction strategies for theme “Comfort.”

6.4 Potential interaction strategies to address occupant needs in Goodwin Hall

In this section, we discuss a few conceptual design ideas for smart building interaction systems that can address the above-mentioned occupant needs in our targeted building. The existing interaction strategies in Goodwin Hall include TV screens located at the main entrance to show current occupancy, thermostats to show space temperatures, and fire alarms to warn in case of a fire emergency. The interaction modalities preferred by the students in descending order of preference included vision, audio, touch, and gestures. As discussed previously, after identifying data types, relative acquisition sources, and collecting the required data from the building and its occupants, the collected data should be processed, and the extracted information should be conveyed to the occupants. To this end, the building should be equipped with user interfaces to exchange the information with occupants who need it.

The concept of affordances can be used when using the building as a medium of information exchange. Traditionally, building designers have relied on semiotics to show necessary information to the users inside a building, but there are several underutilized affordances in a building which can be exploited by using Augmented Reality (AR). Mobile applications can be designed to act as an interface between the occupants and the extracted knowledge from the collected data. Mobile-based applications allow occupants to access real-time data about the building at their fingertip. These apps typically provide intuitive interfaces with features such as scheduling, notifications, and environment monitoring. In the context of the targeted building, the mobile app can be augmented with AR to project information such as navigation to searched study spaces. The capability of AR to superimpose valuable information processed from the data on top of physical objects and its ability to interact with the occupants using various modalities makes it a good fit for a smart building interaction system. With wide access to smart devices such as phones and tablets, as well as the emerging use of smart glasses, AR integration would be an invaluable tool for enhancing the HBI by creating a path for building occupants to learn and interact with available but invisible information in a space. Other information exchange means can include:

Interactive Kiosks/Touchscreen Panels: Interactive kiosks or displays can be installed in common areas or building lobbies, providing occupants with information about building amenities, events, sustainability initiatives, and safety protocols. These kiosks may also offer interactive maps, directories, and wayfinding functionalities to assist users in navigating the building. Touchscreen panels can be installed in common areas or individual rooms within buildings, providing occupants with heat maps of occupancy, building space allocations, centralized control over lighting, and so forth These panels feature user-friendly interfaces with graphical displays, icons, and customizable options.

Voice Assistants: Voice-controlled interfaces, such as Amazon Alexa or Google Assistant, are increasingly integrated into smart buildings, allowing occupants to control various building systems and devices using voice commands. Voice assistants offer hands-free interaction and can perform tasks such as adjusting lighting, setting temperatures, or playing music to promote stress management.

Environmental Monitoring Furniture: Smart furniture pieces equipped with sensors for monitoring indoor air quality, temperature, humidity, and other environmental parameters can be placed specifically in study spaces and offices. These pieces may include built-in air purifiers, aromatherapy diffusers, or climate control systems to create healthier and more comfortable working environments. They can also indicate their availability, helping students find unoccupied study spaces.

Modular Furniture: Modular furniture systems allow users to customize and reconfigure their living or workspace according to their changing needs. These adaptable furniture pieces can be easily rearranged, expanded, or collapsed to optimize space utilization and accommodate diverse activities and functions.

7. Discussion

In our increasingly connected and technologically-driven world, the interaction between humans and buildings has become a pivotal anchor in shaping the way we live, work, and learn in our surrounding built environment. Human-Building Interaction (HBI) represents the convergence of architectural design, user experience, and technological innovation, offering transformative opportunities to enhance the user experience and functionality, efficiency of the built environment. In reviewing the current literature, the following gaps were observed:

First, as we delve into the complexities and possibilities of HBI, it becomes evident that understanding the dynamics of human-building interactions is not only about designing smarter buildings but about reimagining the relationship between occupants and their built surroundings. As such, considering both the occupant and building types and needs is crucial in designing the best interaction strategies for several reasons: different building types serve distinct functions and accommodate specific activities, occupants, and operational requirements. In the same vein, different users have unique preferences, needs, and behaviors, which influence their interactions with the built environment. By considering the type of building along with occupant characteristics such as age and purpose of occupation, HBI designs can be tailored to improve the operation of buildings while accommodating diverse occupant requirements, thus enhancing their satisfaction and well-being. By suggesting an integrated HCI-AmI approach to HBI, the proposed framework aims to shift the focus from only designing digital interfaces or adding intelligence to buildings to enable prediction with the ultimate goal of improving building operations, to a consideration of how a comprehensive view can benefit both the building and its occupants. To this end, the proposed framework offers an approach that aims to evaluate the needs of the building and its primary occupants, and accordingly provides a structure for establishing bi-directional knowledge transition between the built environment and its users to benefit both entities.

Second, the social aspect of HBI highlights the transformative potential of buildings and technologies to foster social connections, promote inclusivity, and strengthen communities, ultimately enriching human experience in the built environment. However, the social aspect of HBI is largely overlooked in the current one-sided solutions that are provided for general-purpose buildings. The need for considering the social aspect is of foremost importance in some buildings like educational buildings, as social interactions in these environments are a part of the building function and should foster collaboration, communication, emotional well-being, and a sense of community among occupants, i.e., students and faculty. This study takes a closer look at the specific needs of occupants in specific-purpose buildings, i.e., educational buildings, and aims to shed light on how a bi-directional HBI approach could take into account both the functionality of the building as well as the particular needs of its occupants and benefit both.

The feedback collected in our proof-of-concept user study shows that proper implementation of such an approach in educational buildings has the potential to enhance learning experiences, support teaching practices, and create conducive environments for academic success. The feedback received from the study participants highlights the following as the potential outcomes of implementing an occupant-centric approach to improve human-building interactions:

Creating active learning spaces with sufficient lighting and proper air quality: such environments could transform traditional building spaces into dynamic, interactive learning environments that facilitate active learning, collaboration, and student participation; Personalizing learning spaces and experiences: learning spaces in the building could be optimized based on occupant preferences, interests, and abilities. Fostering collaboration and communication: designing spaces and user-interfaces that facilitate collaboration, communication, and knowledge sharing among students, teachers, and peers within educational buildings.

Third, user interfaces are crucial elements in enabling a bi-directional interaction between buildings and their occupants as they serve as the primary means for occupants to interact with and control building systems and provide their preferences. A well-designed UI could enhance the usability, accessibility, and effectiveness of HBI, ultimately contributing to occupant comfort, satisfaction, and productivity in the built environment. The results of our user study demonstrated the importance of designing relevant UIs in educational buildings and the following were noted as important in designing proper UIs:

Ease of use: UIs designed for HBI in any building including educational buildings should be intuitive and easy to use, allowing occupants to navigate and control the desired functionality effortlessly. A well-designed UI reduces the learning curve for occupants, making it easier for them to interact with it.

Customization and personalization: UIs should enable customization and personalization of building spaces/settings based on user preferences, needs, and habits. Occupants should be able to adjust indoor environmental factors such as lighting, and temperature, as well as more advanced interactions such as setup of furniture and spaces, to create personalized environments that promote comfort, productivity, and well-being.

Data visualization and analytics: UIs can present data and analytics in a clear and understandable format, enabling occupants to interpret and act upon insights provided by the building. While graphs and dashboards provide visual representations of the environmental conditions and occupant behavior, other technologies such as virtual reality (VR), and augmented reality (AR) can provide interactive learning opportunities, simulations, and experiential learning activities that capture users’ interest and motivation.

Adaptability and scalability: the designed UIs should be adaptable and scalable to accommodate evolving occupant needs, technological advancements, and changes in building requirements over time. Flexible UI designs allow for future expansion, customization, and integration of new functionalities, ensuring that the applications remain relevant and effective in the long term.

As such, by leveraging technology and user-centered design principles, a comprehensive HBI approach such as the one presented here can transform different types of buildings, including educational buildings, into dynamic hubs that encourage improved interaction between humans and their surrounding built environments.

8. Conclusion and future works

This paper compiles existing research on “Human-Building Interaction (HBI)” to identify gaps in HBI theories and their applications. The results show existing gaps in taking occupant-centric approach towards designing HBI systems that consider the specific needs of the occupants in different types of buildings and enable bi-directional knowledge transition between the two. To address this need, we propose a novel vision to develop a comprehensive HBI framework that integrates the concepts of HCI and AmI with HBI to increase the intelligence of building and their interaction with occupants by identifying the type of the building and its primary occupants, investigating the needs of the identified occupants, and finally designing user interfaces that enables bi-directional interaction between buildings and occupants based on the identified needs of both. Such an approach facilitates bidirectional and multi-modal interactions between occupants and the built environment. A proof-of-concept study was conducted at Virginia Tech to evaluate if and how implementing the proposed occupant-centric approach can enhance occupants’ i.e., students, experience in an educational building. Focus groups and semi-structured interviews were conducted among undergraduate and graduate students to identify their needs in the studied technology-enhanced educational building, Goodwin Hall, and whether the currently available technology in the building is helping address their needs. The collected qualitative data was analyzed using qualitative coding and concept mapping. The results revealed that the HBI aspects emphasized by the occupants, i.e., students, in educational buildings were widely different from the standard factors considered in designing conventional HBI strategies. Even when promising to improve occupant comfort, the underlying goal of the conventional approaches is usually to improve the efficiency of the building operations. Based on the results of our study, the key occupant need themes were related to “Navigation,” “Learning,” “Comfort,” “Stress Management” and “Safety,” which are not usually considered when designing HBI systems for buildings. This emphasized the importance of considering the building type and the particular users in designing HBI strategies that benefit both the buildings and their occupants.

Qualitative data analysis also revealed that the code “study resources” has the highest direct impact and the code “comfort” has the highest indirect impact on the learning experience of students in the studied building. This result further highlights the requirement to consider both the building and occupant types in designing appropriate HBI systems and technologies. The existing sensing technologies, collected data types and interfaces were studied to create a data pool for Goodwin Hall. The results show that the existing data and data collection methods, i.e., accelerometers, temperature, humidity and CO2 sensors, are inadequate to address the identified needs of the occupants and additional sensors to measure comfort, stress, and safety are also needed. Based on the identified themes from qualitative data analysis, the features of the HBI system for Goodwin Hall should be selected to make indoor navigation easier, provide essential study resources, help in maintaining indoor environmental quality, and ensure a stress-free and safe social atmosphere. We then discuss how different UIs can be integrated into the building to address the identified needs. For example, various interaction principles including building affordances and advanced technologies such as voice assistants and AR can be used to provide the occupants with useful information and provide ubiquitous personalized services encompassing teaching/learning and psychological needs based on occupants’ contexts.

While the conducted user study was limited in terms of sample size, the results still yield valuable insights, generate hypotheses for further investigation, and inform the design and development of future research studies. Due to the unavailability of the required infrastructure, the proposed IUIs were not implemented in the targeted building and their potential impact was not investigated. In our future work, we will implement and assess the impact of the proposed IUIs to address the needs identified in this research. A more comprehensive study of implementing the proposed framework in buildings with different functionalities will also help further understand the impact of the building and occupant types on HBI design requirements.