Legal and Technical Feasibility of the GDPR’s Quest for Explanation of Algorithmic Decisions: of Black Boxes, White Boxes and Fata Morganas

Abstract

Understanding of the causes and correlations for algorithmic decisions is currently one of the major challenges of computer science, addressed under an umbrella term “explainable AI (XAI)”. Being able to explain an AI-based system may help to make algorithmic decisions more satisfying and acceptable, to better control and update AI-based systems in case of failure, to build more accurate models, and to discover new knowledge directly or indirectly. On the legal side, the question whether the General Data Protection Regulation (GDPR) provides data subjects with the right to explanation in case of automated decision-making has equally been the subject of a heated doctrinal debate. While arguing that the right to explanation in the GDPR should be a result of interpretative analysis of several GDPR provisions jointly, the authors move this debate forward by discussing the technical and legal feasibility of the explanation of algorithmic decisions. Legal limits, in particular the secrecy of algorithms, as well as technical obstacles could potentially obstruct the practical implementation of this right. By adopting an interdisciplinary approach, the authors explore not only whether it is possible to translate the EU legal requirements for an explanation into the actual machine learning decision-making, but also whether those limitations can shape the way the legal right is used in practice.

I. Introduction

“Algorithm (noun). Word used by programmers when they do not want to explain what they did”.¹ This ubiquitous joke has become increasingly popular and notoriously bears a grain of truth. Machine learning algorithms, nowadays pervasively used for various kinds of decision-making, have been traditionally associated with difficulties related to their explanation. Understanding of the causes and correlations for algorithmic decisions is currently one of the major challenges of computer science, addressed under an umbrella term “explainable AI (XAI)”.² While there is no standard definition of this notion, the core goal of XIA is to make Artificial Intelligence (AI) systems more understandable to humans. Generally, XIA “refers to the movement, initiatives and efforts made in response to AI transparency and trust concerns, more than to a formal technical concept”.³

However, the explainability of algorithmic decisions has attracted not only the attention of computer scientists, but recently also of lawyers. The advent of the General Data Protection Regulation (GDPR)⁴ brought the question of algorithmic transparency and algorithmic explainability to the centre of interdisciplinary discussions about AI-based decision-making. Hitherto, numerous authors have argued that the GDPR establishes the right of data subjects to an explanation of algorithmic decisions.⁵ As the first legal act touching upon this issue, the GDPR seems to go to hand in hand with the XAI in exploring the (im)possibility of providing such explanations. The GDPR is thus becoming increasingly important also for XAI researchers and algorithm developers, since the introduction of the legal requirement for understanding the logic and hence explanation of algorithmic decisions entails also the requirement to guarantee the practical feasibility of such explanations from a computer science perspective.

From a legal perspective, the core incentive to guarantee explanations of algorithmic decisions is to enable the data subject to understand the reasons behind the decision and to prevent discriminatory or otherwise legally non-compliant decisions.⁶ Differently, in terms of computer science, the incentive to gain insight into the algorithms goes far beyond a potential legal requirement to provide explanations for algorithmic decisions. Rather, the incentives relate to both ethical and technical considerations. Being able to explain an AI-based system may help to make algorithmic decisions more satisfying and acceptable, to better control and update AI-based systems in case of failure, to build more accurate models, and to discover new knowledge directly or indirectly.⁷ For example, Lee Sedol was able to defeat AlphaGo in only one game where he used “a very innovative move that even the machine did not anticipate”.⁸ This new knowledge, gained through the use of AlphaGo, has hence benefited the community of players of this game.⁹ However, was this knowledge gained because the researchers were able to scrutinise AlphaGo’s reasoning processes like a glass box? Or was this knowledge gained only through an external (and maybe incorrect) description of what the researchers inferred about AlphaGo’s decisions?

These questions illustrate only some of the unresolved issues relating to the explainability of algorithmic decisions. Both technical obstacles as well as legal limits, in particular the secrecy of algorithms, could potentially obstruct the practical implementation of the GDPR’s right to explanation. Some authors therefore propose to build and use inherently explainable models.¹⁰ Even though it can be argued that a legal requirement for explainability exists, putting this requirement into practice could potentially be seen as an elusive fata morgana that appears on the horizon, yet is in reality intangible and unreachable. Against this backdrop, this paper aims to examine the legal and technical feasibility of explainability of algorithmic decisions and to determine whether the GDPR’s right to explanation is a right that can be implemented in practice or rather merely a dead letter¹¹ on paper.

This article is composed of five parts. After the introductory part, the second part seeks to explain the core notions used in this article and the scope of application of the right to explanation from the GDPR. The third part of the article aims to relate different types of AI systems (automated/autonomous and supervised/unsupervised) with the right to explanation. The fourth part explores how the right to explanation can be implemented from both a technical and a legal perspective. The final part concludes the article and offers a schematic overview over different factors that might impact the feasibility to explain algorithmic decisions.

II. Setting the scene: algorithmic explainability between law and computer science

1. The requirement for algorithmic explainability in the GDPR

On the legal side, the question whether the GDPR provides data subjects with the right to an explanation in case of automated decision-making has been the subject of a heated debate. In this vein, certain authors firmly oppose such a construction of this right, given that it is not expressly provided by the GDPR,¹² save in one of its recitals.¹³ Other authors claim the opposite: for them, the right to explanation seems to be a real and tangible right, unequivocally provided by the GDPR.¹⁴ Yet others, including one of the authors of this article,¹⁵ reach the right to explanation through interpretation of various GDPR provisions.¹⁶ Finally, certain scholars claim that the right to explanation is an inappropriate remedy given the difficulties of implementing this right on machine learning algorithms.¹⁷

We do not aim to restate a detailed account of arguments in favour of the existence of the right to explanation in the GDPR,¹⁸ as one of us has provided this analysis elsewhere.¹⁹ To summarise these findings, we submit that the right to explanation in the GDPR should be a result of interpretative analysis of several GDPR provisions jointly. More precisely, we believe that the GDPR provisions on the right to information and access that require the controller to provide the data subject with the “meaningful information about the logic involved” in case of “existence of automated decision-making”,²⁰ should be read together with Article 22 GDPR that regulates automated decision-making, as well as with Recital 71, which epitomises explanation as one of the safeguards in case of such decisions.²¹ One of the core arguments is that the right to explanation enables the data subject to effectively exercise her other safeguards, notably the right to contest the automated decision and to express her point of view.²² In consequence, a clearer understanding of the reasons behind the automated decision can allow the data subject to point out irregularities in decision-making, such as unlawful discrimination.²³

It is important to recall that the GDPR does not provide for a right to explanation for all algorithmic decisions, but only for those that have a legal or significant effect.²⁴ For all other algorithmic decisions, the data subject can exercise other rights provided for by the GDPR, such as the right to information²⁵ or the rights of access,²⁶ but not the right to explanation. For example, in case of a simple targeted advertisement for shoes, the data subject will not have the right to require from the data controller information on why she was shown this particular ad for this particular brand, but she will be able to require from the controller information about the data collected about her.²⁷

It is equally important to clarify that the current EU legal regime limits the right to explanation only to decisions based on personal data. This is a consequence of the fact that algorithmic decision-making is regulated by the EU data protection legislation²⁸ and not by a more general legislation encompassing broader regulation of algorithmic decisions. However, in computer science, decisions with machine-learning algorithms are not confined only to those based on personal data; rather, these algorithms often deploy either a mixed dataset of personal and non-personal data and sometimes even purely non-personal datasets.²⁹ Non-personal data is either data not relating to natural persons (for example, data concerning companies or environment) or data regarding natural persons that does not allow for their identification (for example, anonymised data).³⁰ In mixed datasets, where personal and non-personal data can be easily separated, the GDPR, including the right to explanation,³¹ applies only to the part containing personal data, whereas the Regulation on the free flow of non-personal data³² applies to the non-personal part of the dataset.³³ If such a separation is not possible, the safeguards from GDPR apply to the entire mixed dataset.³⁴ However, decisions based purely on non-personal data are left in a legal vacuum; not only does the Regulation on the free flow of non-personal data not regulate decision-making based on such data, neither does it provide for the possibility of explanation of decisions based on non-personal data.³⁵

We submit that, in certain circumstances, decisions based on non-personal data would equally necessitate an explanation. The underlying reason is that a legal or similar significant effect on a data subject³⁶ can be produced not only by decisions based on personal data; such an effect can occur also in decisions based on non-personal data. Imagine an autonomous vehicle (AV) collecting purely non-personal data, such as information about the weather, road conditions, distance from other vehicles and tyre grip. On an icy road, the AV miscalculates the braking distance and causes an accident in which a passenger is severely injured. Such an accident undoubtedly has at least significant, if not also legal,³⁷ effects on that passenger. Let us use another example: in precision agriculture, an algorithm might wrongly assess the amount of pesticides needed for agricultural products, thus causing debilitating illness in humans consuming those products. This, too, would lead to a significant effect on these persons. Since none of these decisions is based on the processing of personal data, and hence does not fall under the EU data protection regime, the affected persons do not have the possibility to request the explanation of the reasons behind the decisions. Given a significant impact that certain decisions of this kind can have on natural persons, it would be more appropriate for the legislator to provide for a broader regulation of algorithmic decisions and to ensure a right to explanation whenever persons are legally or significantly affected regardless of the type of data on which the decision is based.

2. Algorithmic explainability: discussing the notions

Before embarking on a discussion about the feasibility of the right to explanation from a computer science and a legal perspective, it is appropriate to provide an elucidation of certain concepts in order to situate the right to explanation in the broader context of AI-based decision-making. In the remainder of this article, instead of “AI-based system”, the term “artificial agent” will be used.³⁸ This term is a common metaphor used in computer science to situate software and robots under the same concept, depending on the models, their ability to reason and decide on the action to execute, taking into account different pieces of information. While the GDPR refers to automated decision-making,³⁹ AI experts distinguish between automated, autonomous and algorithmic decision-making and processes. While the goal is the same for automation and autonomy, that is, to perform actions without the need for human intervention, they differ considerably in their features. An automated process is a software or hardware process which executes a predefined sequence of actions without human intervention.⁴⁰ For example, an automated university selection procedure can function in a way to average all candidates’ school marks, then to sort all candidates in decreasing order based on their average mark, and finally to assign admission rank based on this sorting. Therefore, save for machine failures, automated decisions are fully predictable. In this sense, such decisions can be fully explainable, as far as the system’s specifications and the situation in which the decision was made are known. For example, the university admission decision made by an automated system can be explained by the circumstance that the student’s average mark was higher than that of the non-admitted candidates.

Differently, autonomous decision-making entails that the algorithmic procedure behind the decision is computed by the agent, and relies only on a high-level goal defined by a human. Hence, autonomy emphases the capacity to compute which decisions must be made in order to satisfy a formalised goal, and therefore autonomy is the central notion in the design of artificial agents.⁴¹ For example, making the previous university selection procedure autonomous may consist in training a neural network to identify the best students based on all their school records. Here the combination of variables, thresholds and aggregations is not explicitly defined by a human but computed by the machine. Therefore, an artificial agent with autonomous decision-making capacities⁴² has the capacity “to act according to its own goals, perceptions, internal states, and knowledge, without outside intervention”.⁴³ However, autonomy is still bounded: the actions are indeed limited by the amount of information the agent has, by the time available for computation, and by the expressive power⁴⁴ of the language used to specify the decision-making process. Although a human being always specifies the goals themselves or decides how those goals are set, autonomous decisions are more difficult to explain since autonomy allows the machine to decide how to reach the goals, or learn criteria.

Algorithmic decision-making, for its part, is an overarching notion, encompassing both automated and autonomous decision-making. It means that a given decision is made (partly or completely) with the help of an algorithm; this algorithm may be either automated or autonomous and based or not based on AI techniques.

The analysis above demonstrates that the GDPR, by referring only to automated decision-making, does not take stock of these definitions. From a computer science perspective, it would be more sensible if this legal act deployed the overarching term algorithmic, rather than automated, decision-making. If the GDPR truly covered only automated decisions, the practical implementation of the right to explanation would face very few technical obstacles, since automated decision-making processes are based on predefined step-by-step procedures. However, the examples used to substantiate this decision-making in the GDPR – such as predicting aspects of data subject’s performance at work or her economic situation⁴⁵ – fall into the category of autonomous decision-making. Consequently, this legal act in fact seeks to apply to the entire domain of algorithmic decision-making, including decisions made by autonomous agents. The deployment of the latter systems might obstruct the effective application of the law since these systems allow the machine to decide on or learn almost all steps of the decision-making without the a priori knowledge of the user. This opacity can have an impact not only on the implementation of the legal requirement for explainability, but also on determination of accountability⁴⁶ and liability⁴⁷ for the performance of these systems.

3. The multi-faceted nature of explanations

In order to better understand the meaning of explanations in computer science, it is equally important to discuss the notions of traces, explanations, interpretations and justifications as understood in computer science, even though there is a certain degree of disagreement among computer scientists as to the precise meaning of these notions.

All computer programs (thereby artificial agents) can provide execution traces, which show what statements are being executed as the program runs.⁴⁸ Those traces can be analysed by a human expert to understand how an agent, being automated or autonomous, made a given decision. For example, a logic-based system can provide a complete formal proof showing how a given decision will allow a given goal to be reached. While this approach can be useful, such traces are difficult to use for non-expert humans. Thus, it might be preferable to rely on another kind of information, called interpretations. Interpretations are descriptions of an agent’s operations “that can be understood by a human, either through introspection⁴⁹ or through a produced explanation”.⁵⁰ Unlike traces, interpretations are tailored to be readable and understandable not only by experts but also by users. Interpretations can be divided into two sub-categories, depending on what is expected by the users or the designers: explanations and justifications.⁵¹

An explanation describes how a given decision was made, whereas a justification describes why a given decision was a “correct”⁵² one.⁵³ In other words, justifications describe why decisions are “reasonable” (providing the meaning of this word has been defined) knowing the applicative domain,⁵⁴ while explanations exhibit causations and correlations that led to the decision.⁵⁵ It is important to remark that the limit between explanations and justifications is blurred. For instance, Fox et al consider that answering the question “why is what you propose to do more efficient/safe/cheap than something else?” can serve as an explanation,⁵⁶ while it is clearly a justification as defined above. Moreover, computer scientists do not agree on a general and formal definition of explanations, preferring to characterise them through questions. For instance, Doshi-Velez et al state that explanations must answer at least one of the following questions:⁵⁷

1. 1. What were the main factors in the decision?

2. 2. Would changing a given factor have changed the decision?

3. 3. Why did two different similar-looking inputs produce different decisions?

However, neither do computer scientists agree on a finite set of questions. For instance Fox et al consider “why didn’t you do something else?” and “why can’t you do that?” as a means to provide explanations.⁵⁸

Therefore, regarding explanations, this article rests upon the premise that it is challenging or even impossible to define explanations in the abstract, without taking into account the decision-making model, the kind of data used, the desired understandability, or the kind of questions we ask regarding the decision.⁵⁹ From a computer science perspective, this complexity and multi-faceted nature of explanations led several authors to provide various classifications of explanations,⁶⁰ where the notion of “explanations” has various meanings, mostly depicting different methods to reach an explanation.⁶¹ For example, according to Adadi and Berrada, models can be inherently interpretable (meaning that a model provides its own explanation); the explainability⁶² can be either global or local⁶³ (understanding either the logic of the entire model as opposed to explaining why a model made a specific decision); or explanations can be model-specific or model-agnostic (limited to a specific model or independent of a type of model).⁶⁴ Among model-agnostic explanations are visualisations (exploring patterns in a neural unit),⁶⁵ knowledge extractions (computing an interpretable model that approximates a non-interpretable one),⁶⁶ influence methods (methods that change the input and detect how much the changes impact the performance of a model) and example-based explanations (among which counterfactual explanations⁶⁷ have recently gained popularity).⁶⁸ To that, we could add natural language explanations that explain, in natural language or visually, the relationship between the input and output of the model.⁶⁹

As can be seen from this brief overview of the literature, the methodology as to how explanations are reached often also determines their scope and type. Nevertheless, a common thread shared by all these various approaches towards explanations is a desire to make algorithmic decisions understandable to humans, ether experts or non-experts. When humans – and we could add non-experts in particular – desire an explanation for a decision, they in principle seek to understand the “reasons or justifications for that particular outcome, rather than” having a general insight into “inner workings or the logic of reasoning behind the decision-making process”.⁷⁰ Therefore, several authors argue that computer science, by fostering “explanations” as simple models that approximate a more complex one, often misses a true nature of human-like explanations⁷¹ that could be useful for a non-expert user directly affected by an algorithmic decision. Humans are not seeking for a model but rather for contrastive explanations (eg not why a decision D was made, but rather why D was made instead of another decision Q) as well as causal attribution instead of likelihood-based explanations.⁷² We could argue that causality is also one of the main characteristics of the GDPR-based right to explanation because it seems that it is precisely the understanding of causality that would enable the data subject to effectively challenge the grounds for an algorithmic decision. However, while symbolic and Bayesian artificial intelligence proposed many approaches to model and reason on causality,⁷³ the recent advance in machine learning – that leads to the strong resurgence of XIA – is more about learning correlations.⁷⁴ Consequently, as XAI is currently more focused on machine learning, recent explanation methods touch more upon correlations rather than causality.

At the same time, several works⁷⁵ have shown that an explanation is considered futile if the human has to make too great a cognitive effort to interpret it. Therefore, many authors consider that explanations must rely on symbolic approaches,⁷⁶ must be able to be evaluated,⁷⁷ and must help the addressee to understand why a decision was taken to enable her to contest the decision.⁷⁸

Without providing a definition of the notion of explanation in abstracto, in the next section of this article, we delve into different possible explanations, depending on the types of artificial agents, their interactions with humans, and the stages of algorithmic decision-making process. Therefore, in the remainder of this article, we discuss how different kind of agents and the legal requirement in GDPR may lead to a greater degree of feasibility for some explanations compared to others.

III. Different types of agents, different types of explanations

As mentioned above, the nature of artificial agents may have an impact on the nature and feasibility of the explanations that may be provided. In this part of the paper, we analyse this impact more closely. More specifically, in Section III.1, we explain how agents’ behaviours can be described; in Section III.2 we distinguish different types of artificial agents with respect to how they interact with a human. Finally, in Section III.3, we demonstrate that a decision-making process is composed of different steps, which call for different kinds of explanation.

1. Explainability as a question of point of view: external and internal descriptions

The nature of explanations that may be provided about an artificial agent depends on what we know about it. Therefore, it can be claimed that explanations always depend on a point of view. The process of engineering an artificial agent involves a designer who designs it. Once the agent is designed, its behaviour can be observed by an observer. The agent can then be described by a set of properties that are understandable by the observer, by the designer, and sometimes by the agent itself. The notion of observer is particularly important, since some properties that may be observed do not have any practical and direct implementation in the agent as built by the designer. Thus, an artificial agent can be described from two perspectives: an external description and an internal description.⁷⁹ On the one hand, the external description is the description of the artificial agent based on a set of properties used by the observer.⁸⁰ This description is objective in the sense that it is built from the functioning of the agent without knowing what the designer had in mind and how the agent is really implemented. On the other hand, the internal description is the description of the agent based on a set of properties used by the designer, which may for instance be described by the source code of the agent, the fine-tuned parameters within the algorithms, or the weights learned by a neural network.⁸¹ This description is subjective because it relies on how the designer intended to implement the properties.

It is important to note that the internal and external descriptions of an artificial agent do not necessarily match. It is indeed possible for an observer to explain the behaviour of an artificial agent in a way that is completely different from how its internal architecture behaves in reality. This difference can be illustrated by a well-known example from the AI field. In 1948, the cybernetician William Grey Walter built a couple of tortoise robots, named Elmer and Elsie.⁸² Those robots were embedded with light sensors and a small light placed on the front that switched on when no light is detected. Their behaviour was based on a simple automaton that made them walk towards light if detected, and wandering randomly in search for light in other cases. When placed in front of a mirror, the tortoise robots were attracted by the reflection of their own light, making the light continuously switch off and on. The alternation of light switching on and off made the robots shake as their motors alternated quickly between moving towards the mirror and wandering randomly.⁸³ Walter noticed that the robots appeared to be “dancing” when “recognizing themselves in the mirror”.⁸⁴ It shows, as stated by Louise Barrett, that “without the knowledge of how [the robots] were put together, it would be natural to attribute more internal complexity than was warranted, as well as to overestimate the cognitive demands of the tasks that they could perform, like ‘recognizing’ themselves in the mirror”.⁸⁵ Consequently, from a right to explanation perspective, it is of the utmost importance to know whether the explanations provided about the behaviour of a given system are based on internal or external descriptions.

Grounding explanations on external descriptions means that it is only necessary to observe the behaviour of the system, and to analyse it afterwards. This is particularly relevant in cases where the internal description cannot be accessed, for instance if the source code of the assessed system is patented. Moreover, explanations based on external descriptions allow unintended behaviours or side effects to be better taken into consideration. Indeed, external descriptions rely on properties that hold in the eye of the observer, and those properties may have been unconsidered by the designer. For example, let us consider a medical assistant agent deciding what kind of medication a patient needs. Its decision may be explained based on the internal criteria implemented within the agent (eg the agent proposed this medication because it minimises the risks for the patient given her health state). This explanation is based on an internal description. However, from an external point of view, an explanation may be grounded on other criteria that were not implemented in the agent and that the agent was not intended to consider (eg the agent proposed this medication because it minimises the hospital’s financial cost).

The example above shows that explanations based on external descriptions have also drawbacks. Indeed, explaining a decision in not just about giving an explanation, but it is also about giving a correct explanation. Human beings might be misled by these external descriptions – in particular end users who do not have computer literacy. In this sense, explanations based on internal description can be more accurate as they are based on elements that are actually implemented in the assessed system. Table 1 summarises the advantages and the drawbacks of both types of descriptions in the perspective of explaining an algorithmic decision.

Table 1: Explanations based on external and internal descriptions

2. Interactions as explainability factors: supervised and unsupervised agents

The explainability of an agent, that is the capacity to explain its actions, can be further impacted depending on whether the decision is made by a supervised or by an unsupervised agent. In the case of a supervised agent, the human being is an operator who interacts with the agent (eg a robotic agent such as a drone) to make it achieve its functions. An example of a supervised agent is given by Coppin and Legras:⁸⁶ a military operator supervises a bio-inspired⁸⁷ unmanned aerial vehicle (UAV) swarm in order to perform various missions. In this case, the operator is trained to use the agents through setting high-level goals, eg commanding the swarm to watch or avoid an area, or to intercept a target, or directly taking the control of the UAV. Since the operator is trained to understand how the agents behave, it may be easier for her to explain the decision taken by the agent. However, as highlighted by Bathaee,⁸⁸ this task might nevertheless be challenging, because correctly explaining a supervised agent’s decision requires the correct assessment to be made of how the operator influenced the agent’s decision.

Another degree of complexity is added when the supervised agent is also an autonomous agent. Indeed, several types of supervised agents – called agents with adaptive autonomy, with adjustable autonomy or agents in mixed initiative – may be considered according to how the artificial agent, the human operator or both entities can take control of the artificial agent’s behaviour.⁸⁹ Adaptive autonomy means that the artificial agent can take the control from the operator, whereas adjustable autonomy means that the operator can take control whenever she wants.⁹⁰ Finally, mixed initiative means that both the operator and the artificial agent can take over authority.⁹¹ Consequently, in terms of explainability, it is first important to know who (either the artificial agent, or the human operator) had control, in order to explain a given decision.

Differently, in case of an unsupervised agent, the human that interacts with the agent is a user. She uses the functions of the agent (eg a search agent on the Internet), mostly not knowing how they are implemented. An example of unsupervised agents are socially interactive agents such as conversation bots (eg Apple’s Siri) where the user makes requests to an artificial agent in natural language.⁹² Since the users interacting with the agent are not trained to supervise the decisions taken by the agent, the “level” of explanations that an untrained user is capable of understanding may be different to the “level” understood by a trained operator. The level of expertise of the user is relevant also with regard to previously analysed external and internal descriptions. Indeed, it is easier to assume that an operator can better understand an explanation based on an internal description than a non-expert user.

3. Explainability in different stages of a decision-making process

As shown in Section II.3, explanations are multi-faceted. Considering the nature of the objects to be explained, the literature focuses on two kinds of explanations: global explanations that explain the whole system, and local explanations that explain a given decision.⁹³ However, this distinction does not take into account that the algorithmic decision-making process is not a monolithic process. Each decision involves several steps, ranging from data acquisition to the final decision, and, in the context of logic-based expert systems, it has been highlighted that an explanation showing the reasons behind each step of the decision-making process is more acceptable for users.⁹⁴ Traditionally, a decision-making process of an artificial agent is divided into four steps:

1. 1. the agent perceives raw data through its sensors;

2. 2. the agent uses a situation awareness module to transform those data into a situation representation in which it must make a decision;

3. 3. the agent uses a decision module to compute with respect to its situation representation which decision is the best according to its goals;

4. 4. the agent acts according to the decision.

Evidently, if we desire a complete explanation of a given decision, we need an explanation for each step: first, an explanation of how the agent assesses the situation with the perceived data; second, explanation of factors impacting the decision with respect to its goal; third, explanation of each factor involved in the decision-making process; and fourth, explanation of the final “product” of the decision. Explaining only step 4 may not be sufficient, as each step depends on the previous one. An explanation may require an understanding of why given data leads the artificial agent to a given decision. For example, let us consider a university admission system that uses an artificial agent to decide whether a given student is allowed to enrol, based on her academic dossier. Suppose the following explanation is given at the decision module step: “the artificial agent decided to not enrol the student as her chances of academic success are lower than the other students, and her Humanities’ marks are below a critical threshold”. Here, the different criteria used to weigh the decision are given: the student’s chances of academic success, and a threshold on the marks. However, we can wonder on what elements the student’s chances of academic success and this threshold are based on? In this case, a complete explanation would help to understand how the artificial agent assessed the student.

IV. The right to explanation: technical and legal feasibility

In the previous sections, we discussed certain difficulties that an explanation of algorithmic decision-making processes might face both from computer science as well as legal perspectives. We demonstrated that any explanation should also take into account different types of agents and different steps of the decision-making process. In this section, we discuss whether these different types of explanations are feasible from a technical and a legal perspective. We review different technical approaches aimed at providing explanations and we investigate their legal feasibility, focusing in particular on intellectual property obstacles and the secrecy of algorithms.

1. Technical feasibility

Many of the surveys of XAI techniques focus on explaining machine learning systems, without putting the focus on the general perspective of artificial agents.⁹⁵ They distinguish external systems (also called post-hoc or black-box approaches), which are able to analyse an artificial agent and reflexive systems (also called intrinsic, interpretable, constructive or by design approaches), which allow artificial agents to provide explanations by themselves.⁹⁶ While each approach has its own advantages, they all have drawbacks: they are not fit for all types of artificial agents and cannot provide all kinds of explanations. Therefore, this section is not structured around different kinds of explanation but rather based on what is needed to compute those explanations. Such a structure highlights the technical requirements for each kind of explanation.

a. External explanation systems

External⁹⁷ explanation systems are devoted to analysing an artificial agent and proposing explanations. Such an approach has several advantages. First, it makes the explanation system distinct from the artificial agent, allowing the use of the same explanation system for different agents. Second, it can be used on agents that are unable to provide high-level representations of their behaviours. Indeed, such explanation systems can be used both on automated systems and autonomous artificial agents, whatever their inner decision-making processes are (eg logic-based rules, machine learning techniques). Two approaches can be considered, depending on whether the explanation system has access to an internal description of the agent:⁹⁸ the white-box approach, where analysis of the code or specification of the agent is possible, and black-box approach, where there is no knowledge of the agent’s code.

1. 1. Verification and validation techniques may be used as a white-box testing technique which inspects the agents in order to infer general properties such as invariance (what always happens) and liveness (what will happen).⁹⁹ The main advantage of this approach, called property checking,¹⁰⁰ is being able to provide pre-computed external descriptions for the users (eg a given banking decision-support system will never grant a loan to somebody with less than a given annual income). As stated by Fey and Drechsler, “these properties summarize the functionality of a design in a different way and thus explain the behaviour”.¹⁰¹ However, such approaches can only deal with agents whose internal description is known, and rely either on source code analysis or (more conveniently) formal specifications. For example, these approaches are not appropriate to explain the decisions of artificial agents based on neural networks, as the most important part in those agents’ decisions are based on learned parameters, and not the algorithm. Moreover, those external white-box approaches often suffer from a high computational complexity.¹⁰²

2. 2. Black-box testing systems may be used to probe the artificial agent by simulating different inputs and observing the results,¹⁰³ in order to infer either local explanations, or counter-factual faithfulness, or decision trees. Local explanation consists in computing a simpler decision model locally by sampling possible inputs.¹⁰⁴ This model is then used to highlight the main factors of the original decision-making process. For example, one major black-box algorithm explaining the predictions of a classifier is LIME¹⁰⁵ (Local Interpretable Model-agnostic Explanations) and its variants SPLIME,¹⁰⁶ DLIME¹⁰⁷ or LEAFAGE.¹⁰⁸ While those algorithms rely on machine learning techniques, other algorithms relying on principal component analysis¹⁰⁹ or Shapley value computation¹¹⁰ exist. Counter-factual faithfulness (or counter-factual explanations) consist in computing how an input would have needed to be changed in order to produce a desired result. It can then be used to show the impact of a given factor on the decision. From a computational point of view, apart from fixing a desired output, known black-box algorithms to compute counter-factual explanations consist in probing the inputs. For instance, the Growing Sphere¹¹¹ method samples the inputs around the input for which a counterfactual explanation must be computed, while Watcher et al¹¹² sample the inputs with an optimisation procedure that allow to find a counterfactual explanation which minimises the difference with the input to be explained.

Another approach, decision decompilation, consists in building logic-based programs from the input and results of the probed artificial agents, such as a system that generates easily explainable decision trees,¹¹³ or LISP-like¹¹⁴ programs, which interpret how traces change with time.¹¹⁵ For example, Guidotti et al proposed LORE (Local Rule-based Explanations), which provide logic-based local explanations with a decision rule associated to a set of counterfactual rules.¹¹⁶ In general, black-box testing is a very interesting approach to deal with a broad range of agents. For instance, it is very well adapted to provide explanations for neural networks. However, it only provides high-level links between inputs and outputs, namely an external description. It sees the decision-making as a whole and cannot consider explicitly each step in the process. Moreover, it is difficult for such approaches to distinguish a controlled usage of randomness¹¹⁷ in a decision-making process from simple chaotic behaviours.

b. Reflexive explanation systems

A second way to provide explanations is to use artificial agents embedded with reflexive systems. These are agents which are designed for a given goal but which can also reason on their own behaviour and thus can produce formalised explanations.¹¹⁸ This approach is complementary to external explanation systems as it produces explanations based on a fine-grained knowledge of the agent’s internal description, and is clearly fitted to this latter. To this end, artificial agents embedded with a reflexive explanation system satisfy a reflexivity property, meaning their internal description must be able to reason by itself and produce external descriptions. Two approaches can be considered, depending on whether the reflexivity is implemented within the agent’s reasoning module or is implemented in addition to the reasoning module:

1. 1. Reflexive systems added to the reasoning module are often systems which translate the agent’s traces into natural language, providing an ex ante explanation through interpretable traces.¹¹⁹ More sophisticated systems try to highlight causal links between the decisions and the data the agent used when the decision was made. For example, some works propose an explanation module which inspects the agent’s hierarchy of tasks (eg a sequence of actions “to achieve A, the agent needs to achieve B, then C”) and explains a given action through the underlying goal and the current beliefs at the time the decision was taken (eg “the agent decided B in order to achieve A because B is needed to achieve A and the agent believed that B was possible”).¹²⁰ Others propose a similar system for translating optimal policies¹²¹ in human-interpretable sentences: the artificial agent can communicate not only about data it uses to decide but also about likelihoods of possible outcomes or expected rewards.¹²²

2. 2. Similarly, as in the previous approach, reflexive systems built within the agent’s reasoning module aim to translate the agent’s traces into natural language. Unlike the previous approach, built-in reflexive systems aim to produce explanations during the reasoning process and not ex ante. To this end, the explanation system must be interlinked with the reasoning module and provide arguments. For example, Briggs and Scheutz proposed a deontic logic-based explanation module for the special case where an artificial agent can decide to reject a human user request.¹²³ General reasons to reject the request are hard-wired in the reasoning process (eg in the context of the autonomous university admission agent: “the agent cannot enrol more than a given number of students”) and concrete explanations are generated when those rules are violated (eg “your academic dossier ranks you on 211th position while we can only enrol 200 students”). In the context of machine learning, Ding¹²⁴ and Rudin¹²⁵ advocate the use of neural logic networks whose combination of weights explicitly represent logical rules. Another way to interlink explanations and reasoning is to use formal argumentation.¹²⁶ This technique represents pro and contra arguments about a set of possible decisions and computes acceptable decisions with respect to different semantics (eg a decision can be made if – and only if – there always exists an argument against all arguments against the decision). All arguments form a hierarchical structure where users may search for more detailed explanations, For example, Čyras et al proposed to explain decisions, determined by humans or by machines indifferently, by modelling argumentative disputes between two fictitious disputants arguing, respectively, for or against the decision in need of explanation.¹²⁷

c. Classification of different explanation systems

In conclusion, both self-explainable artificial agents, either built on top or built within, as well as external explanation systems, either white-box testing or black-box testing approaches, have advantages and drawbacks, summarised in Table 2.

Table 2: Advantages and drawbacks of different explanation systems

As shown by Table 2, if technical constraints prevent the use of external white-box explanation systems or reflexive modules, then the right to explanation is limited to external black-box approaches. Unfortunately, the external black-box approaches cannot explain the different steps of a decision-making process. Indeed, only the output, namely the final step of the decision-making process, is explained in terms of inputs. How the inputs are used to produce internal representations is outside the scope of the explanations, as well as the fact the decision may have been influenced by human supervision in case of supervised systems.

The other types of explanation approaches also have their limits. Both reflexive approaches are limited to agents specially designed for such approaches. Moreover, those agents are best fit when they rely on logic-based mechanism. Consequently, while it seems interesting to design such an agent from a scientific point of view, it seems difficult to oblige designers to comply with such constraints. Lastly, the external white-box approach, while not relying on an agent especially designed for it, suffers from two limitations: it needs access to the source code or a formal specification of the agent’s behaviour; and it does not provide explanations in itself but only shows some general properties that may be used for explanation afterwards.

Finally, not all approaches are tailored to deal with multi-agent systems. Such systems are composed of several (even hundreds of) artificial agents which may be heterogeneous in their design, and their global behaviour is the result of the local interactions between all agents.¹²⁸ A concrete example of such a system is a web service composition system: a global service (eg a trip management service) is an agent that uses several small services, also agents, dynamically chosen with respect to some quality metrics (eg cost or response time).¹²⁹ To use another example, a network of autonomous vehicles is also a multi-agent system. Due to the distributed and decentralised properties of such systems, providing proofs and analysis is more difficult. Moreover, those systems are generally open, meaning that agents are able to join or leave the system. Such agents can be heterogeneous (in architecture, designers and owners) and can act on behalf of different users. In this context, several new questions are raised. How to provide explanations while some part of the decision is heavily influenced by the decisions made by the other agents? Assuming the agents can provide explanations, how to use and combine them to provide a global explanation? How to deal with agents unable to provide explanations? Those questions currently remain open in the context of the future massive interconnected multi-agent systems.

2. Legal feasibility

As stated at the beginning of this article, numerous scholars have argued that the GDPR establishes or at least should establish the right to explanation.¹³⁰ Recognising the legal right to explanation of algorithmic decisions might be seen as a panacea that clearly and easily enables an in-depth understanding of such decisions. However, the ease of such a claim could be compared to an elusive fata morgana. Not only does the existence of the right to explanation not guarantee its smooth practical implementation, due to concerns related to technical feasibility, as explained in the previous section; the effective understanding of reasons behind an algorithmic decision might also be obstructed by legal obstacles. Specifically, trade secrets or other confidential information can stand in the way of algorithmic transparency. Below we examine the legal factors that can impact the effective exercise of the right to explanation.

a. Trade secrets: a genuine obstacle?

Trade secrets,¹³¹ which are distinct from intellectual property rights,¹³² can potentially stand in the way of effective exercise of the right to explanation.¹³³ As stipulated by the Trade Secrets Directive,¹³⁴ a trade secret is information which is secret, has commercial value due to its secrecy and has been kept secret by reasonable steps of the information holder,¹³⁵ such as an undisclosed know-how and business information.¹³⁶ Algorithms can certainly fall within this definition¹³⁷ and have been effectively covered by trade secrets in practice. Indeed, national case law has already recognised such protection¹³⁸ and some Member States specifically offered the possibility of acquiring trade secrets over technology when transposing the Trade Secrets Directive into national law.¹³⁹ On a global scale, the Amazon recommendation system, the Instagram algorithm for publication diffusion or Google’s search algorithms are among the most well-known examples of trade secrets.¹⁴⁰ In fact, the core algorithm of Google’s search engine, the PageRank algorithm, based on a randomised procedure to diagonalise a matrix which represents the relationships between web pages, is widely known and has been used in several other algorithms, such as an e-reputation algorithm called EigenTrust.¹⁴¹ However, Google’s precise modalities to determine the relationships between pages, the optimisations built into the search system and the parameters used to detect such manipulations are not revealed.¹⁴² For example, it is unknown how different criteria are weighted, such as the number of links, the traffic on the pages or the structure of the pages’ source code.

The importance of trade secrets and the resulting tension with data subject’s rights to access and information is also recognised by the GDPR. Article 23 GDPR allows for these rights to be limited for the protection of “the rights and freedoms of others”¹⁴³ and one of the recitals specifically requires that the right to access should not “adversely affect” the trade secrets of the controller.¹⁴⁴ For its part, the Trade Secrets Directive allows for a suspension of a trade secret “for the purpose of protecting a legitimate interest recognised by Union or national law”;¹⁴⁵ such a legitimate interest could potentially be providing an explanation of an algorithmic decision to a data subject.¹⁴⁶ However, a trade secret should also not affect data subjects’ rights, in particular the right of access to the “personal data being processed”.¹⁴⁷ From a legal perspective, it therefore remains rather unclear which set of rules should take precedence in case of conflict of trade secrets with data subjects’ rights. Malgieri and Comandé argue, based on a systematic interpretative method, that data protection rights should take precedence over trade secret rules.¹⁴⁸ However, such a solution might not correspond to the purpose-based analysis; if GDPR always prevailed over trade secrets, the latter could never be protected when providing an explanation of an algorithmic decision to the data subject.

In this contribution, we would like to offer an alternative solution to the legal conundrum of prevalence of the two sets of rules. The core of our argument is that, even though an algorithm is protected by a trade secret, explaining the decision based on this algorithm does not necessarily need to encroach upon this secret.¹⁴⁹ First, we believe that the extent to which a trade secret and the underlying functioning of an algorithm are revealed largely depend on the scope of the explanation. If the explanation requires the revelation of the entire underlying logic of the algorithm, the trade secret would obviously be revealed. If, on the other hand, the explanation aims to divulge, for example, only the main factor influencing a decision, the trade secret would remain untouched.¹⁵⁰ In cases where an algorithm is protected by the trade secret, the right to explanation (albeit in its more limited version) could be safeguarded through the balancing of different interests.

Second, revealing trade secrets on algorithms would depend also on who probes and tests the algorithm to reach its explanation. If the company itself holding the trade secret on the algorithm provides the data subject with a textual explanation as to the factors impacting the decision, this might not mean that the trade secret would be unlawfully revealed. Moreover, if an algorithm protected by a trade secret is probed by a court, such probing should equally be allowed as long as the further non-disclosure of a trade secret by anyone involved in legal proceedings is respected.¹⁵¹ For example, in a procedure before the court to determine the existence of discrimination by an algorithm protected by a trade secret, the latter could be revealed provided that anyone involved in the procedure keeps this information confidential.

Third, the explanation could be further obtained through reverse engineering of a protected algorithm in the public domain.¹⁵² According to the Trade Secrets Directive, if a product has “been made available to the public” and the revelation of trade secret is a consequence of “observation, study, disassembly or testing” of this product, the trade secret is considered to have been acquired lawfully.¹⁵³ By way of example of Google’s search algorithm, it is legally allowed to reverse engineer this algorithm, even though technically this is a difficult endeavour given frequent changes to the algorithm.¹⁵⁴ Furthermore, this provision of the Trade Secrets Directive equally seems to allow for “black-box testing” or, in other words, “external explanation systems” based on the black-box approach as presented in section IV.1 of this article. Black-box testing means that another algorithm will probe the tested algorithm in order to build a model of the latter, and use this model to provide an explanation. For example, an algorithm can be probed to identify the influence of a given variable¹⁵⁵ or to try to identify whether the outcome is discriminatory.

b. Classified information

Classified information on the level of the EU¹⁵⁶ or its Member States can potentially be one of the biggest obstacles for algorithmic transparency. The EU distinguishes between different levels of classified information, ranging from “EU top secret” to “EU restricted”.¹⁵⁷ The level of classification depends on the degree of harm that revealing of such information can cause to the interests of the EU or its Member States.¹⁵⁸ In certain cases, it is in the interest of the public authorities not to reveal exactly why a certain decision was taken.¹⁵⁹ For example, if military forces use algorithms to plan a sensitive and targeted military operation, and if the modalities of this operation constitute classified information, it could be justified that such algorithms not be revealed to the public. It is important, however, that labelling of a particular information as classified is not abused with the intention to keep the algorithms intransparent. It is expected that the modalities of algorithmic decision-making would need to be part of a broader content of classified information. In other words, it is difficult to see how an algorithm by itself could be designated as “classified” without the classified nature of the information for which it is being used.

The desire of public authorities to keep algorithms secret should not, therefore, be equated with the classified nature of information of which these algorithms form part. For example, the government in the Netherlands has been using secret algorithms to profile its citizens and categorise them on the basis of the risk they might pose, in particular the risk for tax fraud and allowance fraud.¹⁶⁰ The secrecy of these algorithms has been widely debated in the media and even led to a legal action against the Dutch government for non-compliance with privacy rights.¹⁶¹ Consequently, the Dutch court declared this risk indication system, known under the abbreviation SyRI, as incompatible with the right to privacy.¹⁶² Similarly, the French algorithm for university admission remained (unjustifiably) secret until a high school association managed to obtain the information about its modalities.¹⁶³ However, to our knowledge, the use of these algorithms did not seem to fall under “classified” information. Therefore, contrary to the classified information, these “unclassified” algorithms may be revealed. In responding to the question whether they should be revealed, different interests have to be balanced: the right to privacy and data protection, the prevention of fraud, or – in the case of France – the secrecy of deliberation for university admission.

c. Patents and copyrights¹⁶⁴

It is submitted that patent protection does not stand in the way of effective exercise of the right to explanation. Since patent law is not harmonised in the EU, save for a few exceptions,¹⁶⁵ this legal domain remains to be covered by the European Patent Convention (EPC)¹⁶⁶ to which all EU Member States are parties. The EPC allows for patenting of “computer-implemented inventions”,¹⁶⁷ provided that the software is new, involves an inventive step and allows for industrial application.¹⁶⁸ However, such patentability does not extend to algorithms where the inventive step criterion would be fulfilled only if the algorithm possessed “technical features”.¹⁶⁹ In this regard, the European Patent Office (EPO) confirmed that an algorithm is essentially a mathematical method which should not be considered an invention, except if it serves a technical purpose which would give it a technical character.¹⁷⁰ This understanding is in line with the general definition of an algorithm, which can be seen as “a set of steps to accomplish a task that is described precisely enough that a computer can run it”.¹⁷¹ The nature of a mathematical method of an algorithm could be portrayed with an example of a procedure to build a house or a more complex procedure to train a neural network. Based on this line of reasoning, the EPO for example refused to award patents for an algorithm for automatic document classification,¹⁷² an optimisation algorithm determining a travel route in navigation system,¹⁷³ and an algorithm aimed at finding relationships between items.¹⁷⁴ According to the EPO, in order to patent a computer algorithm, the “programmer must have had technical considerations beyond ‘merely’ finding a computer algorithm”.¹⁷⁵ Even though Edsger W Dijkstra is considered to have “invented” the algorithm nowadays used in most¹⁷⁶ of the modern GPS systems,¹⁷⁷ this “invention” still does not fulfil the criteria of patentability because the algorithm remains a mathematical procedure. In consequence, since algorithms as mathematical methods are currently not patentable under the EPC,¹⁷⁸ the patent protection cannot truly create obstacles for understanding and explaining of algorithmic decision-making.

The absence of copyright protection of algorithms in the EU leads to a similar result.¹⁷⁹ While the Directive on the legal protection of computer programs¹⁸⁰ expressly allows for copyright protection of those programs, this is not the case for algorithms underpinning them. In line with the TRIPS agreement¹⁸¹ and the WIPO copyright treaty,¹⁸² the said Directive protects only “the expression of a computer program” and not “ideas and principles” behind those programs.¹⁸³ Therefore, if algorithms comprise such ideas and principles, they cannot enjoy copyright protection,¹⁸⁴ an approach which is largely in line with some other non-European jurisdictions.¹⁸⁵ Even though the Court of Justice of the EU (CJEU) has not yet expressly decided on this issue, its obiter dictum in SAS Institute ¹⁸⁶ seems to have implicitly confirmed this reasoning. By referring to the equivalent recital of the directive previously in force¹⁸⁷ and pointing out that the copyright protection extends only to the source or object code of the program, the CJEU not only confirmed that algorithms cannot be subject to copyright protection, but also that the algorithm and the code need to be clearly distinguished.¹⁸⁸ Despite that, some national jurisprudence nevertheless seems to recognise this possibility, as demonstrated by the recent decisions of an Italian administrative court referring to a copyright protection of algorithms.¹⁸⁹

Within the system of copyright protection, it is important to distinguish algorithms from source code and software. A source code is a particular implementation of an algorithm in a particular programming language, whereas an algorithm is a “reliable definable procedure for solving a problem” which is independent of the language and the implementation.¹⁹⁰ Software is the “final product” of the software engineering process, ie the functionalities through an executable program and the packaging, such as the name of the software, the data used as parameters, the appearance of the user interface and the documentation.¹⁹¹ The distinction between these concepts can be illustrated with an example of a facial recognition software based on neural networks technology. The procedure used to train the data structure within the neural network is the algorithm (for instance a back-propagation algorithm¹⁹² which plays the same role as Dijkstra’s algorithm for pathfinding systems¹⁹³). The particular implementation of both the data structure and the procedure (for example in C++ language) is the source code. All that remains is the software, such as the data used to train the network, the particular values of the neurons once the network has been trained, the interface to interact with a user or the documentation. Therefore, a software and a source code can be copyrighted, but not an algorithm that is merely an abstract idea underpinning the software and the source code.

Furthermore, even if an algorithm forms part of a copyrighted computer program, this does not mean that the algorithm itself is protected by copyright and that its functioning cannot be revealed. The Directive on the legal protection of computer programs allows the user of a computer program “to observe, study or test the functioning of the program in order to determine the ideas and principles”¹⁹⁴ – that is, also algorithms – behind the program. In terms of the right to explanation, a user of a computer program could determine the operation of the algorithm and, if its technical features allow for it, determine the factors that played a role in the algorithmic decision-making.¹⁹⁵

V. Conclusion: feasibility of the right to explanation

Numerous technical and legal factors can shape the type and scope of explanation and information offered to the person affected by an algorithmic decision. These factors are usually intertwined and impact the final possibility of explanation of an algorithmic decision. In conclusion, therefore, we seek to bring the discussion on the interplay of those factors together and offer a final overview of different types of explanation in correlation with factors that impact the feasibility of the explanation. Based on the analysis in this article, we classify explanations into six different types summarised in Table 3: property checking (see Section IV.1.a.1), interpretable traces (see Section IV.1.b.1) local explanation (see Section IV.1.a.2), counter-factual faithfulness (see Section IV.1.a.2), decision decompilation (see Section IV.1.a.2), and providing arguments (see Section IV.1.b.2). Each of these types of explanations bears different characteristics and varies depending on the scope of explanation provided, the time it can be provided, the system it can be applied to and the way it is applied on the system.

Table 3: Feasibility of explanations of algorithmic decisions

As far as we are aware, from the computer science perspective, and more specifically XAI, there are no general impossibility results. An impossibility result is a formal proof showing that a given problem cannot be solved by a given computational model.¹⁹⁶ Therefore, XAI methods present scientific limitations for the type of explanation they can provide, depending on which information should be provided to the data subject and depending on the time when the explanation is provided. These technical constraints are depicted in the rows “Meaning of explanation” and “Time of issue”. The “Meaning of explanation” category refers to the information that can be provided when trying to explain an algorithmic decision, ranging from a simple verification of properties to more elaborate arguments that are used to support such a decision. The “Time of issue” category indicates when the explanations can be given: either in advance (ex ante) or after the algorithmic decision has been taken (ex post).¹⁹⁷ Furthermore, different types of systems (supervised/unsupervised) might be suitable to provide a certain type of information, as depicted in the column “Best fitted systems to explain”. The explanations further depend on different technical methods to provide such explanations (“Best fitted technical method”).

The “Feasibility to overcome legal obstacles” column depicts how difficult it is to effectively provide an explanation where legal obstacles are present, such as trade secrets or classified information over the algorithm or the source code. Generally speaking, external white-box methods necessarily need access to the source code or specifications. More specifically, in the case of property checking, trade secrets and classified information can indeed constitute an obstacle to reach an explanation as this method requires access to the source code in order to allow for extraction of properties of both the algorithm and the program. Therefore, if the source code is subject to a trade secret, this might prevent the effective explanation of an algorithmic decision, unless one of the methods proposed earlier in this article is applied. As elaborated above in Section IV.2, observing, studying, disassembling or testing of a public product does not violate the trade secret.¹⁹⁸ Nevertheless, as also mentioned above, such reverse engineering does not always lead to satisfactory results.

Differently, in case of interpretable traces, the danger of infringing a trade secret or classified information is minor. Because a module is designed to provide a simple textual explanation of the execution traces left by the algorithm taking a decision, no direct access to the latter algorithm or the source code is necessary. However, this algorithm or this source code must be especially designed to produce those explanations, which is difficult to be legally enforced. On the contrary, explanations using external black-box methods (local explanation and counter-factual faithfulness¹⁹⁹) do not need a special design of the program. There is also no need to analyse the code of the program, since the program itself is probed or tested.

In cases of decision decompilation, which is a kind of reverse engineering of the decision-making system, access to the code or the algorithm is equally not necessary. Decision decompilation functions in a comparable fashion to local explanation, since both approaches rely on probing the system by giving it inputs and observing the outputs. However, differently from local explanation, decision decompilation does not focus on one main factor impacting a decision, but rather builds a simple model (such as a decision tree) that approximates the logic of the decision.

Finally, the explanation that provides arguments in favour and against a decision would not really face legal obstacles in terms of trade secrets or classified information. Since the program generates arguments that can be used as explanations, the question of these types of legal obstacles would not really arise. However, as in the case of interpretable traces, the system that is able to provide arguments needs a special design of the program, which is difficult to legally enforce.

The column “Degree of correspondence with GDPR explanation” aims to explain which type of explanation is best fitted to correspond to the scope of explanation required by the GDPR. As discussed in this article, the explanation given to the data subject should enable her to understand the reasons behind the decision, in order for her to be able to contest such decision.²⁰⁰ In practice, this means that the explanation should be able to be translated into natural language and be understandable to the average data subject without expertise in computer science.²⁰¹

When it comes to property checking, it is quite apparent that this approach does not correspond to the abovementioned requirements. Not only does this type of explanation focus on providing the logic of the program rather than concrete reasons underpinning a decision, its result is also rather complex and understandable mostly to experts. Differently, interpretable traces can be easily understood by non-experts because the explanation provided by this method is rather simple. This method is suitable for decisions with clear pre-determined criteria on the basis of which a decision is made. For example, this method could explain a decision of a non-complex university admission system that uses the average high-school grade as a benchmark for admission. However, this approach cannot be used for more complex machine-learning systems. For example, it cannot explain how a credit rating system based on a neural network makes decisions because such network can merge in a single value both postal codes and birthdates, which is semantically difficult to interpret.

Furthermore, local explanations are suitable in terms of GDPR requirements. Local explanations probe the system to determine the correlations between input and output as well as to extract the main factors of the decision.²⁰² While this approach individualises the explanations, it can be quite time consuming if the system is probed for every decision of every individual. Counter-factual faithfulness functions in a similar fashion, with the difference being that it evaluates how a change in a particular factor influences the decision. It may be used, for instance, to assess the fairness of a decision, based on how each factor within a given specific input influences the decision. Therefore, this type of explanation is useful in terms of the GDPR.²⁰³

The method of decision compilation may be also useful because it approximates the whole logic of the decision, instead of focusing only on the main factors of the decision independently. Finally, providing arguments in favour and against a decision could, at least theoretically, be a helpful method to explain more complex algorithmic decisions, for example decisions made in public administration based on previous similar precedents. This method could also be used as a decision support in legal decision-making more generally, as it allows for the weighing of various arguments. However, the models providing for such explanations are rather difficult to build, which hinders their wider use in practice.²⁰⁴

The analysis in this article demonstrates that algorithmic decision-making, in particular the question of the right to explanation, necessitates a broader and clearer policy approach. Specifically, it must be clarified which type of explanation is required by the legal rules (GDPR), whether explanations need to be provided also for decisions based on non-personal data and whether those explanations can allow for flexibility depending on the model used.

Moreover, XAI methods also present different needs in terms of access to the systems to be explained. While the attention of the recent literature has been very much on “opening” black boxes, an alternative option, as Rudin suggests, could lie in building and using more inherently interpretable models.²⁰⁵ This idea follows the same line of reasoning as using reflexive explanation methods, which may produce arguments but require a special design of artificial agents. However, this idea depends not only on the general possibility of using certain models for certain types of decisions, but also on the willingness of industry and public administration that deploy algorithmic decision-making to choose one model rather than another.²⁰⁶ Indeed, it might seem disproportionate to (legally) oblige those stakeholders to use a specific model in order to increase the explainability of their decisions. Hopefully, further research in both law and computer science will help dissipate the current fata morganas around the right to explanation, and allow for clearer insights into the reasons and causes behind algorithmic decision-making.