Harnessing robot experimentation to optimize the regulatory framing of emerging robot technologies

Abstract

From exoskeletons to lightweight robotic suits, wearable robots are changing dynamically and rapidly, challenging the timeliness of laws and regulatory standards that were not prepared for robots that would help wheelchair users walk again. In this context, equipping regulators with technical knowledge on technologies could solve information asymmetries among developers and policymakers and avoid the problem of regulatory disconnection. This article introduces pushing robot development for lawmaking (PROPELLING), an financial support to third parties from the Horizon 2020 EUROBENCH project that explores how robot testing facilities could generate policy-relevant knowledge and support optimized regulations for robot technologies. With ISO 13482:2014 as a case study, PROPELLING investigates how robot testbeds could be used as data generators to improve the regulation for lower-limb exoskeletons. Specifically, the article discusses how robot testbeds could help regulators tackle hazards like fear of falling, instability in collisions, or define the safe scenarios for avoiding any adverse consequences generated by abrupt protective stops. The article’s central point is that testbeds offer a promising setting to bring policymakers closer to research and development to make policies more attuned to societal needs. In this way, these approximations can be harnessed to unravel an optimal regulatory framework for emerging technologies, such as robots and artificial intelligence, based on science and evidence.

Policy Significance Statement

Our contribution advances the knowledge on how to use science for robot policy. The article zooms in on a new knowledge-policy model for framing the issues relating to emerging robotic technologies, particularly wearable robots. Its significance is threefold. First, it highlights the lack of evidence-based mechanisms to support policies for robot technologies. Second, it introduces the PROPELLING project (which stands for “pushing robot development for lawmaking”), a project funded by the H2020 Eurobench project that looks into how robot testing facilities could generate policy-relevant data to improve technical standards. Third, it presents the advantages and some of the challenges of using evidence-based mechanisms in policymaking for emerging robotics and suggests some strategies to tackle them.

1. Introduction

Robotics and artificial intelligence (AI) are growingly featured in healthcare contexts due to their increased roles and capacities in performing surgery helping in rehabilitation or therapy, currently upshot of the need to reduce human contact (Alemzadeh et al., 2016; Aymerich-Franch and Ferrer, 2020; Fosch-Villaronga and Drukarch, 2021). Advances in healthcare robotics and AI may entail incredible progress for medicine and healthcare soon and could eventually help repair misdiagnoses and their very high consequences for society (Singh et al., 2014). Still, inserting robots in remarkably sensitive domains raises puzzling legal, social, and ethical considerations (Vallor, 2011; van Wynsberghe, 2013; Palmerini et al., 2016; Fosch-Villaronga, 2019). As with other emerging technologies, this field is dynamically and rapidly evolving, primarily due to its revolutionizing capabilities in increasing productivity and resource efficiency. However, robots’ rising autonomy, fastness, increased roles and capabilities, and novelty are questioning existing regulations’ fitness (Leenes et al., 2017; Fosch-Villaronga and Mahler, 2021). Indeed, technology is capable of leaving the law behind at any phase of the regulatory cycle, pointing to the problem of regulatory disconnection, where either “the covering descriptions employed by the regulation no longer correspond to the technology” or “the technology and its applications raise doubts as to the value compact that underlies the regulatory scheme” (Brownsword, 2008; Brownsword and Goodwin, 2012).

Information disparity seems to be one of the main drivers behind such a disconnection. As science moves faster than moral understanding, people even struggle to articulate their unease with the perils novel technologies introduce (Sandel, 2007). It is also common to see inventors and users sidelining ethical considerations while focusing on the practical considerations of efficiency and usability (Carr, 2010). Regulation is not immune to those problems. On the contrary, information asymmetries between corporations, developers, and regulatory agencies are increasing, impeding the enactment of frameworks closer to reality and more attuned to the real problems that technology poses.

Regulators often operate in a regulatory environment where it is difficult to enter into the conversation, let alone intervene adequately. Should these asymmetries continue, technology companies “[will] have a lock on how their products work while underfunded and understaffed regulators will continue to struggle not only to understand the technology but to articulate their concerns” (Guihot and Moses, 2020). Moreover, (robot) developers will continue to struggle to implement legal provisions into their designs, resulting in constant disconnects between policy goals and safe technology (Kapeller et al., 2021). Equipping regulators with technical knowledge on design and practices could help them understand the regulatory needs of a specific and novel technology (Athey, 2017). However, solving such information asymmetries among developers and policymakers raises questions with respect to what kind of information is needed to bridge this gap. In this article, we explore how regulatory interventions to specific robot types can be grounded on research and development (R&D) information, including their risks and mitigation strategies.

PROPELLING, which stands for “Pushing robot development for lawmaking,” is a financial support to third parties (FSTP) from the Horizon 2020 (H2020) EUROBENCH project that explores how robot testing facilities could generate policy-relevant knowledge for robot technologies. Using ISO’s safety standards for lower-limb exoskeletons as a case study (ISO 13482:2014 on personal care robots), PROPELLING sets different experiments to advance the knowledge and anticipate a regulatory response for fear of falling (FoF), protective stops, and instability in collision in exoskeletons. Its premise is that the generated knowledge can be harnessed to help policymakers unravel an optimal regulatory framework for emerging technologies. That is, one based on science and evidence.

This article introduces some difficulties in regulating emerging robots in Section 2 and focuses on how the lack of data makes it challenging to define adequate interventions. Section 3 introduces PROPELLING as a model for addressing those difficulties and serving as an evidence-based model for robot policymaking. The idea behind it is to generate policy-relevant data from testbeds to help unravel an optimal regulatory framework for emerging robot technologies. Section 4 focuses on ISO 13482:2014 (Robots and robotic devices—Safety requirements for personal care robots) to highlight the significance of experimentation for improving the standard’s content. Section 5 presents the challenges of experimentation and early lessons learned. The article concludes with a summary and a reflection on how science could support policies framing robot development.

2. The Difficulties of Regulating Emerging Robots: Lack of Information, General Principles Codes, and the Quest for Better Norms

A literature review reveals a paucity in harnessing R&D outcomes to improve existing regulatory instruments (AbouZahr et al., 2007; Höchtl et al., 2016; Athey, 2017; Fosch-Villaronga and Heldeweg, 2018). Consider the role of evidence in influencing food and nutrition-related public health policy. The United Nations’ Food and Agriculture Organization (FAO) has been developing methods, data repositories, and training programs to strengthen countries’ capabilities to support interventions on credible data and statistics within this field. The case of Mexico’s tax on sugar-sweetened beverages is an example where the correlation between increases in sugar consumption and the rise of obesity during the previous decades informed the adoption of an excise tax on sugar-sweetened beverages (Rocha and Harris, 2019). However, while the chemical, food, and pharmaceutical industries established years ago use evidence-based models that ensure the safety of these products EU-wide, these frameworks have yet to be seen within robotics regulation.

The regulatory landscape for robot devices are, on the contrary, currently populated by a myriad of technology-neutral regulations (Leenes et al., 2017), abstract codes of conduct, and trustworthy-based ethical guidelines (HLEG AI, 2019b) that lack the necessary empirical grounding to inform researchers, designers, and developers’ practices adequately. Accordingly, they fail to tackle the complex problem of giving appropriate guidance to robot developers, who often face the task of developing design measures to make robots fit those abstract values (Lipton, 2018; Valori et al., 2021). Not rarely do developers find themselves in a position with unclear regulatory guidance and in which they usually end up not integrating policy goals entirely. In this scenario, they are often left to decide if and if so, how the development process can be continued: (a) if positive fit, valorize; (b) if unclear legal fit, ask clarification, permission, and assume negative; and (c) if negative/no fit, stop, adapt, lobby, and ignore (Fosch-Villaronga et al., 2018).

Lack of adequate information on both sides is one reason for the scarcity of adequate interventions, as impacts are hard to predict during the early stages of technology development (Collingridge, 1980; Genus and Stirling, 2018). However, how much regulators know about technology, its effects, or the regulatory environment is critical for ensuring emerging developments that comply with existing norms. This results in what Koops distinguishes as known unknowns—that is, we know the technology and that it has some impacts, but do not know what those impacts are—and unknown unknowns—where even the existence of the impact is ignored (Koops, 2010).

Emerging robotics, particularly lower-limb exoskeletons, are prone to these two types of phenomena. These body-borne devices are “inextricably intertwined with the human body” (Mann, 2012), and, as such, they raise particular questions along different dimensions that range from safety to data protection, dignity, agency, control, and trust (Kapeller et al., 2020). However, the content of those impacts remains largely undertheorized. For instance, the impact of data security on users’ safety remains largely unknown. Exoskeletons are cyber-physical systems directly fastened to the user’s body (Greenbaum, 2015b; Fosch-Villaronga et al., 2018) that can have vulnerabilities in security that can compromise the correct functioning of the device but the extent of that impact is still wholly undetermined (Morante et al., 2015; Fosch-Villaronga and Mahler, 2021).

Beyond specific impacts, the methods for assessing risks also remain a contested matter (Tucker et al., 2015; Ranchordás, 2021). He et al. (2017) documented the lack of uniformity on inclusion and exclusion criteria for testing subjects, along with the reports of adverse events and identified risks. This lack of evaluation methods and reporting systems opens up a gulf between safe devices and those whose hazards remain unknown due to poor framing and testing (Tucker et al., 2015). Furthermore, due to their close interaction with users, these devices that have traditionally been assessed against physical safety requirements introduce a cognitive dimension that involves psychological aspects from the user (Martinetti et al., 2021). Some can relate to shared control, visual appearance, but others to the very FoF from a device that will make them stand up and walk again. However, the nature of these impacts and how they can be appraised remains unexplored (Cruciger et al., 2016; Fosch-Villaronga, 2019). Hence, there is room for revealing some of the unknown unknowns that affect safety and that regulators are yet to be even aware of as potential hazards (except for a few remarkable exceptions, see Pons, 2010; Fosch-Villaronga, 2019; Kapeller et al., 2020).

Estimating risks and knowing what safeguards apply in each case to make the human–robot interaction safe is thus a central challenge for regulating exoskeletons and other emerging technologies. However, crafting policies on the verge of further development demands, on the one hand, dealing with scant to nonexistent data on hazards, and, on the other, predicting how technology will evolve, including its impacts. Indeed, if policymakers wished to regulate robots based on empirical data, they would undoubtedly be puzzled by the sort of information required, whether the sources are generalizable, or whether data from isolated experiments could provide the basis for governing real-life risks. When it comes to robots, the problem of insufficient information goes beyond mere information provision availability, as even the test methods have not been put into action, nor have they been broadly evaluated (ISO/TR 23482-1:2020 Application of ISO 13482:2014 safety-related test methods). This is particularly relevant because, despite the advances and many clear benefits that medical robots may bring for society, systems that exercise direct control over the physical world have the potential to cause harm in a way that humans cannot necessarily anticipate, control, or rectify.

As Wischmeyer and Rademacher (2020) put it, “while the belief that something needs to be done is widely shared, there is far less clarity about what exactly can or should be done, or what effective regulation might look like,” an uncertainty that unfortunately is at the expense of user rights (Fosch-Villaronga and Heldeweg, 2018). Bringing regulators and developers closer in an environment where robot devices and testing methods are developed and validated offers a promising path to solve the information conundrum of emerging technologies, which has already been tested in Japan for robot technologies (Weng et al., 2015), but that has yet to happen in other parts of the world. We attempt to advance in this direction within the project PROPELLING, which aims to pilot robotic testbeds as information sources for policy purposes.

3. PROPELLING: Pushing Forward Robot Development for Lawmaking

Assessing risks through experimentation is essential to ensure robot safety and compliance with existing norms. The anticipation of hazards and reflections on appropriate safeguards often happen in testing beds, where prototypes’ characteristics are improved to meet safety standards. In this scenario, benchmarking proves to be a vital instrument to assess technology readiness. Combined with testbeds, they indicate whether a specific technological solution is suitable for the society and safe to use.

With that in mind, the European Union’s H2020 funded EUROBENCH, a program that aims at setting the first framework for the development of benchmarks for robotic systems (Torricelli and Pons, 2018). Its purpose is to develop methods and tools to assess devices in a rigorous and replicable way. These reflections are mainly restricted to developing prototypes in light of their further move into real-life applications. PROPELLING departs from the premise that they could also provide knowledge to improve regulations by, among others, establishing new safety requirements for uncovered challenges or reformulating existing criteria that are inconsistent with how technology works.

In this sense, PROPELLING gathers a multidisciplinary team to reflect on the possibility to harness experimentation as a source of evidence-based knowledge for policy interventions for robots. In particular, PROPELLING will execute EUROBENCH’s protocols to improve not only a specific device but the overall standard for wearable robotics. In this sense, PROPELLING will use the EUROBENCH test facilities, including software and databases, to assess a combination of indicators and control algorithms.

The project’s primary focus is lower-limb exoskeletons—specifically, Technaid’s Exo-H3—and ISO 13482:2014, the first safety standard devoted to personal care robots. It thus tests particular safety requirements for systematically appraising safety assessments and regulations for marketed lower-limb robotic exoskeletons. Instead of focusing on how the device fits the standard, it seeks to understand if ISO 13482:2014 addresses safety sufficiently and comprehensively. In concrete, the project focuses on some of the safety requirements revolve around topics identified in the context of the H2020 Cost Action 16,116 on Wearable Robots, ¹ such as those concerning psychological aspects (e.g., FoF), push recovery algorithms, and different user categories (see Kapeller et al., 2020, 2021).

There are indeed some indications that ISO 13482:2014 fails to address safety comprehensively and accurately. For instance, while balance loss is the second cause of falls among older adults, travel instability measures do not apply to lower-limb exoskeletons (ISO 13482:2014 Annex A.1 Hazard item 59). Another example is that, while obstacles can pose a risk to normal gait (e.g., stairs and objects), ISO 13482:2014, however, states that collisions with safety-related objects, other robots, fragile safety-related objects, walls, permanent/unmovable barriers “are not applicable to restraint-type physical assistant robots” (ISO 13482:2014 Table 7, p. 45). Among the myriad of hazards and safety measures, PROPELLING focuses on the FoF, control algorithms for protective stops and graceful collapsing, and instability in a collision (see next section). These elements have in common that they remain unaddressed within the ISO 13482:2014 as specific hazards or safety measures. These aspects serve as a pilot study to appraise the possibilities of harnessing information from experimentation and R&D to improve existing standards.

To that end, PROPELLING experiments with a set of volunteers (see Table 1) to understand how safety requirements apply to various categories of users. The EUROBENCH foresees 2 weeks of testing at different times: the first week at the beginning of the project and another in the middle of the project. Since the experiments take some time, the project usually provides two volunteers to do the experiments. FSTP projects can bring two other subjects depending on resource and time availability. Following approval by EUROBENCH of our experiment design for the first week of testing, a diverse group of four volunteers participated in our experiments. Since our inclusion criteria included not using a lower-limb exoskeleton in the past and representing different ages (young, middle age, and older adult) and sex groups (male and female), we decided to bring the four volunteers that matched each of the user groups that we envisaged. We did not include participants who self-reported dizziness or a neurological disorder, and all of them provided their consent to conduct the experiments and process their data.

Table 1. PROPELLING volunteers

Given the constraints of our FSTP, we chose different users for our experiments to help advance the knowledge on how different hazards impact different user groups. We did so because, since its publication, ISO 13482:2014 has included a statement recognizing that more specific standards demand more comprehensive numeric data on different categories of people (Introduction, vi). Currently, the standard only recommends taking into account typical body sizes of the intended user population to avoid demanding postures or ensure easy operation (ISO 13482:2013, Section 5.9.2.1). It does not distinguish between women and men, nor does it have specific requirements for people who might have different bodies or have certain conditions demanding specific adaptations.

Failing to consider differences between uses and integrate intersectional aspects (i.e., the interconnected nature of social categorizations such as race, class, and gender as they apply to a given individual or group) is particularly significant for lower-limb exoskeletons and other wearable robots because of the intimate human–robot interaction. Unlike other technologies, these robots work assembled to the users’ bodies, which often depend on them to walk, lift things, or even remain to stand. They thus challenge how users experience themselves and their bodies (Kapeller et al., 2021). That is why testing methods and design choices that are oblivious to intersectional aspects will likely prevent many users from accessing this technology’s benefits (Fosch-Villaronga et al., 2020). As long as ISO 13482:2014 does not consider any special safety requirements for different types of users, it could be that a personal care robot is certified under the standard with disregard as to its safety or accessibility. This gap brings about uncertainty regarding the protected scope of the framework for different types of users and questions to what extent diversity and inclusion as concepts are sufficiently considered and how they are (Søraa and Fosch-Villaronga, 2020).

PROPELLING’s expected outcome is thus to use experimentational settings to test several aspects that ISO 13482:2014 does not address comprehensively, and we do so by inviting different users from different age and sex groups. The overall aim is to use evidence-based knowledge to help put forward specific, informed recommendations to revise ISO 13482:2014 for lower-limb exoskeletons.

4. Generating Evidence-Based Knowledge to Improve Exoskeleton Framing in ISO 13482:2014

Wearable robots, particularly exoskeletons, are emerging technologies pledging to make physical work more accessible and safer in the industry and return functionality to patients in rehabilitation. Wearable robots include hardware—actuators and sensors—and control algorithms geared toward augmenting, training, or supplementing human motor functions (Greenbaum, 2015a). These robots are intimately intertwined with the human body, profoundly impacting how users experience their environment and how others experience them (Kapeller et al., 2021). Although some private standards have been developed in the past that could frame the issues arising from these technologies, our, and the research of others, shows these do not frame robot technology development accurately (Zwitter, 2014; Calo et al., 2016; Leenes et al., 2017; Fosch-Villaronga, 2019).

In the following subsections, we explain how ISO 13482:2014 lacks safeguards for aspects such as FoF and instability in a collision. We also indicate how the standard could benefit from having more specific guidance before, during, and after protective stops and graceful collapsing mechanisms are activated. We also explain the tests that we ideated to generate data for these aspects. While these are not the only aspects requiring attention in the ISO standard (Fosch-Villaronga, 2019; Boada et al., 2021; Salvini et al., 2021), these are the aspects we chose to focus on given the scope of the PROPELLING project.

4.1. FoF as a hazard and developing predictors for risk of falling

Exoskeleton users do not only suffer from physical impairments. Due to their particular condition or a result of an accident, users’ cognitive capabilities may not be as they used to be, and they may suffer from episodes of confusion (Jacquin et al., 2014). Wheelchair users may be afraid to get on a robotic device that will help them walk once again after such a long time without walking. FoF is among the leading causes behind reduced mobility, independence, and quality of life among elderly populations (Grimmer et al., 2019). For exoskeleton technology, the FoF can compromise and constrain the device’s performance that aims to restore a regular gait pattern and not a feared one (cf. Heinemann et al., 2018; Fosch-Villaronga, 2019). Although such a psychological aspect is so tightly connected with safety, ISO 13482:2014 does not include FoF as a potential hazard associated with lower-limb exoskeletons and, therefore, the standard does not foresee any safeguards to prevent that from happening.

PROPELLING proposes a method to test whether FoF is a hazard associated with lower-limb exoskeletons in Table 2. It seeks to monitor different users’ heart and respiration rates, along with their heart rate variability and galvanic skin response, while using the device in different conditions (see Table 2). These include (a) seating, (b) standing, and (c) walking conditions. It also includes fear-related scenarios. The second set of testing protocols entails perturbing users while wearing the device and measuring its recovery from those perturbations. Both settings are complemented with questionnaires aimed at measuring their mental stress and heart acceleration (Pisotta et al., 2021b).

Table 2. PROPELLING scenarios, protocols, and performance indicators for fear of falling policy change

These tests generate data that we can measure against specific performance indicators that work as predictors of the presence of FoF. These include stress, understood as the state of mental or emotional strain caused by adverse circumstances. Stress is measured following a method based on fuzzy logic, which renders the performance indicator. These, in turn, have input the subject’s galvanic skin response, heart, and respiration rate, and heart rate variability (Tagliamonte et al., 2021). The remaining performance indicators—namely, Perceptibility, Acceptability, Functionality, and Usability—are measured based on a multifactor questionnaire piloted within the EUROBENCH consortium (Pisotta et al., 2021b; Tagliamonte et al., 2021). The questionnaire includes four factors—Perceptibility, Acceptability, Functionality, and Usability—and different subfactors within each of them. These factors measure, among other things, whether users felt safe and in control of their bodies while using the exoskeleton. Each item is assessed based on a 7-point Likert-type scale, where one means “I strongly disagree” and seven means “I strongly agree” (Pisotta et al., 2021a; Tagliamonte et al., 2021). It also includes a consistency scale that assesses the user’s reliability when responding to similar questions (Pisotta et al., 2021a; Tagliamonte et al., 2021).

In addition, the time and facility to recover from body sway allows correlating that fear to potential safety hazards. This testing set can also make FoF act as a predictor of actual risks of falling. Although many exoskeleton-related studies mention the risk of falls, most of them conclude that there is no risk of falls just because no falls were observed in clinical trials (He et al., 2017). However, as He and colleagues point out, “it is likely that the safeguards and task conditions followed in those clinical trials will be distinct from those imposed by settings outside the clinic” (He et al., 2017). In short, it is possible that “the risk of actual falls in these studies was completely mitigated because of the overhead harness” and other conditions of the experimentation (He et al., 2017).

The problem here, thus, is one of choosing the adequate experimentation setting. Erroneous scenarios or designs not reproducible in further research bring the risk of being incapable of informing regulators. Moreover, poor designs overlook significant risks, such that potential hazards might remain out of the radar of policymakers. Indeed, a missing step between experimentation and policymaking is the lack of indicators for determining the risk of falling as a hazard and the subsequent need to address the issue. That circumstance points not to the nonrecognition of the hazard itself but the lack of appropriate benchmarks in the emerging field of robotic exoskeletons. There is, however, research pointing to the convenience of using gait changes and FoF as a reliable predictor for those risks of falling (Rivasi et al., 2020). PROPELLING taps on those studies to pilot the use of indicators on FoF, not only as a source of novel hazards but as an appropriate set of benchmarks for predicting the risks of falling.

4.2. Initiating protective stops

Protective stops and graceful collapsing are essential components in robotics to ensure safety. ISO 13482:2014 distinguishes between emergency stops and protective stops. Developers are obliged to include the former, which is manual and whose function is to stop the device in an emergency. In contrast, protective stops should only be included in the risk assessment demands and can be automated. Their function is to safely control the safeguarded hazard (ISO 13482:2014, Section 6.2.2.1, General Table 2—Comparison of emergency and protective stops). Still, the standard mentions that the latter kinds of stops should also cause the stop of all hazardous motion by removing or controlling power (ISO 13482:2014, Section 6.2.2.3, Protective stop). In general, the standard invites manufacturers to define personal care robot halting. However, it does not include specific guidelines on scenarios where stops should be initiated, whether there could be automated protective stops, nor measures to address faulty or hampered stops.

The standards also leave the question of reachability open. ISO 13482:2014 does not clarify under which circumstances the red button should be activated or what happens once activated. It also does not set limits to whether other persons could activate it or under which circumstances this could be automatically done. These questions arise in contexts in which exoskeletons have the protective stops in the rear, giving the impression that another person should be able to “quickly access” the protective stop. Also, when the exoskeleton has climbing-stair functions incorporated, the question is raised whether protective stops be activated during stair climbing. In those scenarios, users’ safety will be compromised if the stop is in an unreachable part of the device or if other actors can easily reach it. It would also be compromised if the device automatically stops. These design choices could not only hamper reachability by the user, but they could also compromise their safety if commenced at the wrong moment, that is, when the user is close to a stair and could fall, or when the user is about to cross the street during a red light (Fosch-Villaronga, 2017).

Furthermore, the insertion of protective stops raises many unanswered ethical questions. It may translate into dependable devices that cannot be used without supervision, which might lead the user to feel a loss of control as she is no longer capable of stopping the robotic device. It may also raise questions about the typology of protective stops, the differences between stopping mechanisms—the big red button to entirely stop the robot or the protective stop used from a tablet—or whether and under which circumstances they can be activated automatically. Relatedly, no mechanism helps the user or the caregiver “stop” the processing of personal data, nor is there a test for validating protective stop mechanisms and graceful collapsing (Fosch-Villaronga, 2019).

To tackle that lack of guidance and the confusion this may cause, PROPELLING suggests executing two scenarios, one replicating unstructured environments and the other characterizing impacts on muscle coordination, to specify instances in which protective stops should be automatically activated or when their activation poses a greater danger to the user (see Table 3).

Table 3. PROPELLING scenarios, protocols, and regulatory concerns for protective stops

Note. For more information about the H2020 Eurobench protocols, see http://eurobench2020.eu/wp-content/uploads/2020/09/EUROBENCH-benchmarking-scenarios-description_v2.pdf (accessed 2 September 2021).

Abbreviations: LDS, local divergence stability; MoS, margin of stability.

Here, the used benchmarks aim at providing information on the interaction between users of different ages and genders and the exoskeleton. Those indicators inform the implementation of design measures and provide data for standardizing testing methods. They also provide information that could correlate with other indicators to discover the impact of cyberattacks and faulty features on users’ safety. This may suggest the need to standardize novel safeguarding alternatives and testing criteria for cyber-physical systems and protective stops, potentially impacting standards for personal care robots.

4.3. Instability in collision

Another critical yet unexplored hazard is instability in a collision. While the standard specifies different safeguards to protect against instability in collision for person carriers, it does not do so for physical assistant robots or exoskeletons. However, since users may also travel some distance with wearable robots, they may also encounter difficulties if they collide with different objects or persons. Indicators about different subjects’ ability to maintain balance in collision might suggest ISO 13482:2014 has room for improving how to address collision instability specifically for lower-limb exoskeletons, which remains a hazard that is not associated with them. Simultaneously, information about stability in unstructured environments could indicate validating measures to address the risk of falls. PROPELLING aims to use users’ stability as an indicator of the capability of different subjects to maintain balance while wearing an exoskeleton. Moreover, information on different ages and genders could suggest specific standards for distinct categories of subjects.

In light of the lack of standardized methods, there is a risk that the evidence gathered will fail to frame the technology appropriately. Without the tools for unearthing safety hazards in a manner that could be replicable, we run the risk of ending up with off-shot experiments that lack the necessary reproducibility for informing general regulations (see Table 4).

Table 4. PROPELLING scenarios, protocols, and regulatory concerns for instability in collision

Hence, PROPELLING aims to unearth whether gaps in test facilities, particularly concerning the determination of risks of collision and instability, fail to provide adequate data to inform regulators. This information is essential to understand what gaps the testing facility has in providing a good variety of scenarios to test different safety requirements (for instance, on slippery terrains).

5. Toward Evidence-Based Policymaking for Robots: Early Lessons Learned and Discussion

5.1. Challenges posed by evidence-based policymaking for wearable robots

Experimentation and evidence may tell us “what works” and “what are the risks” of developing new technologies. Still, what kind of evidence will deliver the necessary results, how reliable the information should be, or which sources deserve attention are recurrent questions policymakers may ask. These questions point to unresolved issues and tensions in the regulation of emerging technologies via the use of evidence-based mechanisms: was the data cherry-picked? Was the method obfuscated, or was there any manipulation of evidence? To avoid the risk of regulatory failure, it is necessary to discuss and thoroughly understand what counts as evidence and what can be considered policy-relevant. Otherwise, even the best policy proposals may fail to deliver adequate results in practice.

An evidence-based approach is as good as the type of evidence gathered. The quality of that evidence depends on the quality and reproducibility of the method used and its collection. The first challenge we have faced has to do with the amount of information that we could gather given the constraints of the project. These limitations are coupled with the fact that the field of robotics largely lacks such methodologies as a recent survey of state of the art acknowledges (see, e.g., ISO/TR 23482-1:2020 Application of ISO 13482:2014 safety-related test methods). The problem with lower-limb exoskeletons and robotics, in general, is the lack of agreed methods for evaluating certain parts of such developments, mainly the psychological, but also other aspects such as trust, dignity, and privacy. That is the case, for instance, with the strategies most marketed exoskeletons employ for dealing with potential falls. The lack of a settled methodology has led to several strategies whose effectiveness and risks remain unclear (He et al., 2017).

The second missing step between experimentation and policymaking that we have noticed is the lack of indicators for determining the existence of a hazard and the subsequent need for addressing the issue. In other words, identifying what aspects constitute a safety hazard and which aspects should be evaluated and safeguarded are extremely difficult without field knowledge (Koops, 2010). On top of that, there is nothing like the idea of serving patients’ interests in technology regulation. In robotics, for example, ISO 13482:2014 does not define personal care (Fosch-Villaronga, 2019), and it has many confusing categories that suggest avoiding compliance with the medical device regulation rather than ensuring users’ overall safety (Fosch-Villaronga, 2016; Salvini et al., 2021). However, regulations not only serve the goal of safety, and if they do, safety has many dimensions, not only physical (Kilovaty, 2021; Martinetti et al., 2021). As these devices become more ubiquitous and incorporate AI, other concerns with regard to privacy, discrimination, and even cyber-security arise. The findings published by the LIAISON project highlight this issue and stress that although each of the standards investigated within the context of the project, including ISO 13482:2014, is concerned with physical safety requirements, the legislative system includes many other fundamental rights to be protected and factors to be taken into account to ensure safety to the fullest extent of the word (Fosch-Villaronga and Mahler, 2021). In this line of thought, insights from the broader community of stakeholders within this project have indicated that standards should shift from mono-impact to multiimpact (Fosch-Villaronga, 2019). In this sense, they should include factors related to ethics, environmental sustainability, liability, accountability, privacy, data protection, and psychological aspects, further stressing the need for a multidisciplinary multistakeholder approach in crafting policies aimed at framing new technologies.

Moreover, the early stages of PROPELLING have shown that proposals for harnessing experimentation should also be mindful of regulatory bottlenecks. For institutions and instruments to keep an adequate connection with the stream of technological innovation, these methods should also suit the procedures and practices behind regulatory development (Brownsword, 2017), which can generally be divided into public and private policymaking initiatives. Public policymaking is generally “a system of laws, regulatory measures, courses of action, and funding priorities concerning a given topic promulgated by a governmental entity or its representatives.” ² There are examples of public policymaking activities within new technologies and robotics at the European level, resulting in a wide variety of EU-wide measures, particularly Directives and Regulations and proposals (HLEG AI, 2019a; European Commission, 2021). Nevertheless, the EU struggles to release technology-savvy, sector-specific guidelines (Fosch-Villaronga, 2019). Moreover, public policymaking is often outdated and tech-neutral, and legal responsiveness does not always follow technological development timely or at all as a consequent step (Collingridge, 1980; Marchant et al., 2011; Newlands et al., 2020).

To address this inability of regulators to keep up with the fast pace of innovation and propose regulatory actions matching state of the art and the foreseeable impacts such emerging technologies may have, private actors have developed private standards, such as the ISO standards, to mitigate the ethical and legal risks and concerns posed by robotics. ISO standards are developed within national groups involving different access levels and influence on the system. In theory, they claim they involve experts from all over the world gathered within technical committees that join together consumer associations, academia, NGOs, and government, although they do not proactively seek engagement with affected users. In a way, they allow technology to work seamlessly and establish trust so that markets can operate smoothly, thereby providing a shared language to measure and evaluate performance. However, the development of ISO standards in practice lacks a consensus-based approach because they lack a representation of all the voices affected by their development (Fosch-Villaronga, 2019).

5.2. Lessons learned

Despite these difficulties, testbeds represent an experimental, cocreative approach to innovation policy that aims to test, demonstrate, and advance new sociotechnical arrangements under real-world conditions and including the users that are truly affected by the development of robotic devices. In the simplest sense thus, testbeds are controlled experimental spaces that facilitate a kind of performance or hypothesis testing under presumably realistic conditions (Engels et al., 2019.). Hence, they set a promising environment for developing generalizable benchmarks and testing methods for robotics and emerging technologies. They provide an adequate landscape for moving toward agreed yardsticks and experiments. In this sense, testbeds can serve as data generators for better policymaking, broadly understood as regulations, laws, technical standards, and other instruments used to achieve defined goals. Moreover, they set the stage for discussing adequate experiments, methods, and performance indicators for appraising safety to be replicable across domains.

The second lesson is the importance of creating shared data repositories (SDRs) and connecting them to policymaking. Accessing and processing the information gathered in the databases EUROBENCH aims to create could be valuable for informing policymaking (Fosch-Villaronga et al., 2018; Fosch-Villaronga and Golia, 2019). These SDRs can take the form of databases compiling the results of reproducible experiments and risk assessments, along with related robot legislation/regulation collected over time and across many projects (Fosch-Villaronga et al., 2018; Fosch-Villaronga and Golia, 2019). Documenting and formalizing these processes (in lessons learned) would allow the regulatory framework to have a grounded knowledge and understanding of what characteristics and regulatory needs new robot technologies have, and this knowledge could be highly valuable for future developers and policymakers.

In this vein, developing replicable experiments and agreed yardsticks, and complementing them with data repositories accessible to regulators is the path forward to evidence-based standard-making. In future work, PROPELLING aims to map the results of conducted experiments and compare the retrieved data with those from other subjects that have undergone similar protocols in other FSTP projects to strengthen the validity of the observations and results. If users go through the same protocols and the test data is shared and made available to researchers, it becomes possible to use experimentation for policymaking. Linking experimentation settings with standard-making processes can speed up the creation, revision, or discontinuation of norms governing robot technology, increase their effectiveness in ensuring overall safety, and the legal certainty regarding a fast-paced and changing environment like that of robotics. This process can shed light on what needs regulatory attention for adequate robot governance and make robot oversight more precise and concrete, allowing for easier compliance and virtually not wasting the potential testing zones to generate relevant knowledge for policymaking.

6. Conclusion

Gathering information is vital for framing technology adequately, and experimentation serves as an appealing source of data on emerging developments for this purpose. However, harnessing that information largely depends on whether that evidence comes from replicable and generalizable methods that fit the regulatory process. Research on those aspects in the field of regulatory interventions for robotics and modern technologies currently remains in its infancy.

This article introduced PROPELLING as a stepping stone toward ideating practical ways to use benchmarking ecosystems such as the H2020 EUROBENCH as data generators for robot policymaking. The project focuses on a limited set of issues to portray the promises and challenges of using evidence to develop policies for emerging technologies. The idea, however, is not limited to wearable robots or technical standards like ISO 13482:2014. The ultimate goal is instead to show the potential of experimentations for informing policymaking for different technologies. In other words, the idea behind PROPELLING is using science to help the law keep up with the pace of technology development. This process can make interventions more precise and concrete, allowing for easier compliance and virtually not wasting the potential testing zones to generate relevant knowledge for policymaking (Fosch-Villaronga and Golia, 2019).

Igniting a conversation on the limitations and promises of evidence-based regulation is crucial for steering EU regulatory discussions to more practical and definite regulatory reform proposals. In this vein, policies will become responsive to ongoing developments in the field of new technologies.