Addressing Algorithmic Errors in Data-Driven Border Control Procedures

Abstract

The gradual digitization of EU migration policies is turning external borders into AI-driven filters that limit access to fundamental rights for people from third countries according to risk indicators. An unshakeable confidence in the reliability of technological devices and their ability to predict the future behaviour of incoming foreigners is leading towards the datafication of EU external frontiers. What happens if the supposedly infallible algorithms are wrong? The article aims to understand the consequences of algorithmic errors on the lives of migrants, refugees and asylum seekers arriving in the European Union. This contribution investigates the socio-political implications of deploying data-driven solutions at the borders in an attempt to problematize the techno-solutionist approach of EU migratory policies and its fundamental rights impact on affected individuals.

A. Introduction

The European Union (EU) and its Member States increasingly rely on data-driven technologies to perform border control procedures and manage incoming migration flows. The progressive ratification ¹ of EU external frontiers aims to supervise the movements of third-country nationals in the aftermath of the Schengen agreement ² and the consequent abolition of internal borders between EU Member States. ³ Implementing digital frontiers with data-driven risk assessment procedures aims to deflect external safety and stability menaces, thus preventing those who represent a threat from entry. The increasing digitalization of migration management influences how the EU and its Member States assess international protection requests from incoming third-country nationals. Consequently, borders are gradually becoming data collection and elaboration sites that serve social sorting processes for security purposes. ⁴ More specifically, the EU is gradually implementing an interoperable network of large-scale IT systems collecting all the relevant information retrieved from non-EU citizens entering EU territory. ⁵ The Schengen Information System (SIS II) provides public security authorities with data on people and objects passing through EU frontiers from abroad. ⁶ The Visa Information System (VIS) permits sharing of visa information between Schengen States. ⁷ The VIS aims to prevent visa shopping episodes and irregular migration through biometric matching for identification and verification purposes. The “Eurodac” is a crucial feature in managing EU asylum applications. ⁸ It processes the fingerprints of asylum seekers and irregular migrants who have passed through EU frontiers, preventing the duplication of asylum requests. The Entry/Exit System is currently under development; it will register through electronic means the time and place of entry and exit of third country-nationals to calculate the length of their authorized stay. ⁹

The EU and the Member States are looking toward technological solutions with an unwavering amount of trust in their capabilities to analyze the surrounding context and forecast possible developments. More specifically, EU migratory policies adopt data-realism assumptions, according to which data present truthful representations of reality. ¹⁰ EU digital border policies move from this confidence in the descriptive capability of data to build complex and interlinked infrastructures that collect and process personal data of different categories from third-country nationals entering EU territory. ¹¹

Artificial Intelligence (AI) technologies could be crucial to elaborating on such information. In general terms, modern machine-learning algorithms identify hidden patterns between large amounts of “raw” information and produce diagnostic outcomes based on processed data. ¹² In other words, these algorithms study the statistical inferences between training data (information used by the software to develop reasoning structures) to find correlations between a specific input and the resulting output. ¹³ Well-functioning AI software can accurately identify the patterns retrieved from the training datasets in new circumstances. ¹⁴ The resulting statistical model should be able to forecast the outcome of a future prompt. Regarding border control and migration management goals, AI-powered software could verify the identity of already known persons and assess unknown individuals against risk indicators to evaluate threats to EU security and internal stability. ¹⁵

The deployment of AI algorithms could benefit migration management operations, including the fast and efficient analysis of asylum applications and more expedited access to available information to make policy choices. According to a technical report, ¹⁶ AI technologies are crucial features of large-scale IT systems that perform several functions, which include IT infrastructure and service management, protection from cyberattacks, and optimization of data processing performances. The soon-to-be-released European Travel Information and Authorization System (ETIAS), ¹⁷ a visa waiver pre-evaluation framework for visa-exempt foreigners entering the EU territory, will conduct AI-driven risk assessment procedures to detect incoming security threats. This system will process data through AI algorithms against risk indicators to evaluate if the third-country nationals under investigation could threaten EU stability and safety. However, in the migration context, AI technologies raise risks for the rights of vulnerable people, including the migrants themselves. The recent proposal for a European Regulation on Artificial Intelligence (the AI Act) considers the use of AI systems in the field of migration to be a potential high risk to the health, safety, and fundamental rights of the people concerned (Annex III, Art. 7). ¹⁸ Thus, the legislative draft requires appropriate risk management procedures (Art. 9) to implement high-risk AI solutions. However, the AI Act does not provide any legal remedies in case of algorithmic errors.

This Article interrogates the EU regulatory framework on its adequacy in protecting the fundamental rights of people affected by misleading and erroneous algorithmic outcomes at the EU’s external frontiers. The aim is to understand the legal consequences and policy implications of algorithmic errors in the context of EU migratory policies. The first part of this contribution deals with the notions and types of different AI flaws and addresses their potential causes. The second section analyzes how algorithmic errors could infringe on the fundamental rights framework. More specifically, this part of the Article highlights how incorrect AI outputs could exacerbate the vulnerabilities of migrants, refugees, and asylum seekers. Furthermore, the analysis explains the shortcomings of the existing legal remedies at the EU level to challenge misleading AI outcomes in migration management procedures due to an unclear accountability framework. The last section of the contribution formulates a few concluding remarks that suggest possible solutions to the problems raised by algorithmic errors. More specifically, the Article argues for a shift when implementing AI solutions at the borders; migrants should be considered as humans with specific rights, not data that requires processing.

B. Algorithmic Errors and their Potential Causes

Artificial Intelligence is a general term that indicates a vast array of technologies and software deployed for an ever-growing variety of scopes. Therefore, providing a legal definition of algorithmic errors that could fit within different contexts is challenging. Scott and Yampolskiy assess errors as static conditions that may lead to AI failure, resulting in the algorithmic system’s inability to perform its functions in the surrounding circumstances. ¹⁹ Insofar as border control is concerned, this Article considers algorithmic errors when AI-driven frontiers deliver misleading and erroneous outputs. Thus, it wrongly identifies migrants and results in poorly performing risk-assessment operations. Moving from these assumptions, I will make a few considerations about the limits of modern AI technologies.

Machine learning technologies use complex mathematical functions to establish the correlation between input and output according to processed datasets. AI algorithms provide human observers with a statistical explanation of the relationship intercurring between two or more elements. However, such technologies cannot describe the logical causation between input and output. ²⁰ Correlations do not necessarily infer a cause-effect relationship between such objects. ²¹ Policy choices can rely on falsely predicting algorithmic outcomes.

AI algorithms can process vast amounts of data because they previously compressed such information. Data compression mechanisms may imply the loss of differences and peculiarities within the reality under analysis.

More specifically, such algorithms interpret data relevant to a specific subject against categories and benchmarks elaborated on with information about other elements. ²² Insofar as risk assessment procedures are concerned, these AI-powered programs make predictions about a person based on group data. Consequently, subjects not belonging to the statistical majority represent an anomaly to the AI software. ²³ Regarding migration management purposes, digital borders may become data-driven filters ²⁴ that limit access to fundamental rights according to potentially discriminatory factors such as race, ethnicity, language, nationality, and religion. ²⁵

Machine learning programs may fall within so-called feedback loops, which occur when algorithmic outputs misinterpret the surrounding reality and influence the implementation of training datasets to update and develop algorithms. ²⁶ Misleading outputs may become erroneous inputs for other machine-learning routines. Algorithms are overfitting when they rely too much on training data and only look for exact matches while disregarding similarities and correlations between elaborated information. ²⁷ On the contrary, underfitting problems occur when AI software cannot identify patterns from the training data and fit them within the surrounding reality. ²⁸

Data misinterpretation could produce erroneous algorithmic outcomes and propagate and amplify biases. This term indicates one reason to choose a specific data generalization instead of another, thus losing the inherent complexities of the reality under analysis. ²⁹

Pasquinelli proposes a tripartite classification of biases that could influence the working routine of machine learning algorithms: ³⁰ i) world biases indicate how the algorithms reproduce and propagate prejudices and inequalities already occurring in the real world. Datasets represent reality, including its stereotypes; ii) data biases occur during elaborating and implementing training datasets. Marcuse explained through the concept of technological rationality how the deployment of specific technologies could influence the rationality of society. ³¹ Deploying ever more sophisticated and advanced technologies shapes the state and its citizens’ relationship. ³² Data processing activities are not neutral operations but adopt peculiar perspectives and social hierarchies to analyze surrounding circumstances. Using unreliable taxonomies and data categorizations could portray an inaccurate view of reality, disregarding data peculiarities and features. Foucault explained the normative potentialities of data, addressing them not as a matter of fact but as a matter of concern; ³³ iii) algorithmic biases propagate the above-mentioned discrepancies through computational failures and information compression mechanisms.

Therefore, the highest data quality standards are crucial in implementing reliable and accurate AI-driven technologies. The General Data Protection Regulation (GDPR) states reliability and accuracy as founding principles of data processing activities (Art. 5(1)(d)). ³⁴ However, EU information databases in the area of Freedom, Security and Justice have been suffering from data quality issues for a long time. ³⁵ Spelling errors, mistakes in translating names into the Latin alphabet, the recording of erroneous and misleading birth certificates, technical failures, and lack of training in dealing with digital infrastructures are only a few reasons to explain low-quality data within such border management information systems. ³⁶ AI algorithms receiving data inputs with such flaws will logically deliver unreliable outcomes.

Data processing aims to give a truthful representation of the reality under scrutiny. However, deciding what is the correct picture of reality is a choice that has policy and legal implications. ³⁷ Accepting a specific margin of error, intended as the difference between reality and its representation delivered by data elaboration operations, could have fundamental rights and consequences worth mentioning.

C. Algorithmic Errors and Fundamental Rights Consequences

Erroneous algorithmic results can dramatically affect the lives of migrants, refugees, and asylum seekers, exacerbating their vulnerabilities behind misplaced confidence in the efficiency and reliability of data-driven frontiers. The widespread use of technological solutions to assist border control may produce incorrect AI results that infringe on fundamental rights and freedoms, resulting in unfair treatment: that is, the right to life and liberty, the right to privacy, and principles of non-discrimination. ³⁸ In addition, the inherent opaqueness of AI algorithms may make identifying possible errors difficult, which prevents affected individuals from challenging biased administrative decisions.

I. Algorithmic discrimination and false prophecies

“We are black and border guards hate us. Their computers hate us too.” Adissu, an Eritrean migrant living undocumented in Brussels, describes how EU border guards treated him and his fellow migrants. ³⁹ Technological devices in border management risk repeating and amplifying systemic prejudices and discriminatory attitudes already present in modern society.

As explained before, technologies influence society. More specifically, socio-political, economic, and cultural variables can influence the working routine of AI-powered devices. A study shows how a decision-making algorithm used in several US hospitals usually shows that White patients are more likely to be potential users of social programs that address complex health issues. ⁴⁰ The researchers found that the algorithm assigned lower risk values to Black people despite having the same medical issues as White patients. AI software calculated how much a single person spent over the year in health expenses to evaluate appropriate risk scores on the assumption that fewer expenses indicate fewer health needs. However, a close examination of the dataset showed that Black patients exhibited far more severe health issues than White subjects despite the same healthcare expenditures. According to the data, Black people spent an average of $1,800 (US) less per year compared to White individuals with identical health problems. The lack of trust in the medical system and racial discrimination from healthcare personnel are but a few symptoms of a discriminatory attitude that impacts Black patients. Thus, the algorithm has introjected and reproduced systemic racism that prevents Black people from accessing medical services for Black people.

Notwithstanding these concerns, the EU places great trust in the reliability of algorithmic outcomes, relying on technological neutrality beliefs and disregarding any political implications of data processing activities for border management purposes. ⁴¹ As an example of this dangerous techno-solutionism tendency, ⁴² the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-Lisa) is considered as “noise data” to be eliminated from their info-systems due to migrants’ unwillingness to share information about themselves. ⁴³ Such an approach disregards how data processing operations impact the affected individuals.

Insofar as border control procedures are concerned, AI-powered frontiers could exacerbate the inherent vulnerabilities of people on the move, preventing them from accessing their fundamental rights. The working mechanisms of machine learning algorithms could propagate discriminatory biases, reinforcing prejudicial treatments against migrants, refugees, and asylum seekers. Profiling activities and risk assessment procedures targeting these categories may be considered unreliable data or a result of misinterpretations of reality. As explained before, feedback loops might mislead future diagnostic activities. Biased algorithms could lead to self-fulfilling algorithmic prophecies. ⁴⁴ In other words, algorithms could label specific individuals or minority groups as potential security threats, thus justifying additional security controls against them for no other reason than algorithmic outcomes. ⁴⁵ Such a trend could produce networked discrimination phenomena, according to which discriminatory treatments originate from previous biases. ⁴⁶ Restrictions on freedom of movement, actions of deprivation of liberty, and surveillance procedures may be justified by incorrect algorithmic results, upon which the EU bases its migration policy due to its unshakeable confidence in the reliability and impartiality of technology.

II. The algorithmic black box and its consequences on procedural rights

Machine learning algorithms find hidden patterns between a given input and a specific output. More specifically, AI-powered solutions elaborate on complex mathematical functions that find statistical correlations between data using their vast computational power, which is way beyond human capability. An additional layer of complexity is due to the mutating nature of the algorithms themselves. More specifically, such algorithms adapt their working routine to the mutating circumstances, developing and improving their diagnostic capabilities according to data inputs subject to concept drifts. ⁴⁷

The term “black box barrier” indicates the impossibility for external human observers to identify, understand, and replicate the reasoning patterns chosen by machine learning algorithms. ⁴⁸ The rationale behind a specific output remains obscure behind the curtains of technological complexities.

Explainable Artificial Intelligence is a highly debated issue in legal scholarship. ⁴⁹ Implementing the principle of explanation within the algorithmic working routine would ensure meaningful human control over the AI-driven decision-making process. ⁵⁰ Figuring out the reasons behind a specific algorithmic output would make human actors morally accountable for its implementation.

Insofar as AI-powered devices for public affairs are concerned, explainable algorithms comply with the right to good administration, raising an obligation for governments and public offices to justify their decisions (Art. 41 EU Charter of Fundamental Rights – EUCFR). Understanding the reasoning behind algorithmic outcomes would make it possible to question and challenge legal and administrative measures based on such AI outputs, thus complying with the right to an effective remedy and a fair trial (Art. 47 EUCFR). The inherent opaqueness and undetectability of AI algorithms prevent the identification of errors and biases, thus permitting the propagation of incorrect algorithmic results.

The importance of algorithmic outputs in the decision-making process of human operators (for example, border guards, administrative courts, etc.) is ever-growing. Human decision-makers may suffer from automation bias and unreasonably regard algorithmic outputs as unquestionably correct, ⁵¹ thereby failing to consider the risks of flaws and errors in the working routine of the algorithms themselves. Due to this worrying tendency, the affected individuals may face additional difficulties in challenging judicial and administrative decisions. Who should be held responsible? The human judge or the algorithm?

Agents working in the field of migration management may also lack the appropriate knowledge to evaluate the correctness of the algorithmic information they have access to in their decision-making processes. Therefore, the EU and its Member States should provide their migration management agents with appropriate training opportunities. Furthermore, standardized guidelines on the role of algorithmic outcomes in international protection request assessments would ensure the applicants’ rule of law safeguards, including legal certainty and procedural predictability.

D. The Lack of Appropriate Legal Remedies to Challenge Algorithmic Errors and their Consequences

Migrant people face significant difficulties accessing justice to protect their rights from abuses by the EU and its Member States in data-driven border control procedures. More specifically, interested individuals may struggle to find suitable reasons to appeal adverse migration decisions based on data processing activities due to the inherent opaqueness of the algorithmic working routine. It may be technically impossible to detect AI mistakes and identify the subjects responsible for them. Insofar as EU digital borders are concerned, the interoperable nature of databases operating in the Areas of Freedom, Security and Justice causes additional complexities from a procedural perspective. More specifically, these IT systems contain data from several sources and serve various purposes. These variables may prevent the identification of the subject accountable for having entered unreliable data within the networked digital archives.

The Court of Justice of the European Union has jurisdiction over the European Border and the Coast Guard Agency (Frontex) and its data-driven border control operations. According to EU Regulation 2019/1896 (Frontex Regulation), the European Border Surveillance System (EUROSUR) should function as a surveillance framework and communication network for the exchange of information and operational cooperation between Frontex and EU Member States to improve situational awareness and increase reaction capabilities for border management aims (Art. 18). More specifically, EUROSUR has to monitor, detect, identify, track, and intercept any unauthorized border crossing of EU external frontiers (Art. 19). Frontex, acting as a Eurosur coordinator, can take advantage of numerous surveillance tools such as drones, cameras, and sensors to collect crucial data to provide the EU Commission and its national coordination centers with a detailed and updated analysis of the pre-frontier areas and the EU’s external borders (Arts 26–27). In spite of the above considerations, existing legal remedies at the EU level may fall short of efficiently addressing data analysis errors made by Frontex (and their consequences). According to the CJEU, interested subjects could request the annulment of unclear, insufficiently detailed, or unsubstantiated reasons that caused the EU to act (Art. 263 TFEU). ⁵² However, Nicolosi rightly points out that annulment actions against Frontex operations could be in vain. ⁵³ Frontex border management duties take the form of factual conduct, which does not often require the adoption of legally binding acts and is thus not compliant with the requirements prescribed by Art. 263 TFEU. Insofar as the scope of this contribution is concerned, AI-powered risk assessment procedures, digital pushbacks, and other data-driven forms of border control actions may not be amenable to annulment action. An action for damages (Art. 340 (2) TFEU) may provide shortcomings in protecting individuals from the consequences of algorithmic errors. They may have difficulty proving that the harmful conduct is attributable to Frontex because of the opacity of the algorithms. In addition, the coexistence of several actors cooperating for the same goals could make it impossible to distinguish between the EU and its Member States’ competences (and related responsibilities). ⁵⁴

The European Commission recently released a proposal for a directive on adapting non-contractual civil liability rules to Artificial Intelligence, introducing the so-called “fault-based liability” approach (Recital 7), according to which claimants must prove the defendants’ prior guilty behavior before holding them liable. ⁵⁵ The draft would allow interested subjects to request developers of high-risk AI products to disclose relevant evidence to prove alleged harmful events deriving from their products (Art. 3). Furthermore, a rebuttable presumption of causality applies when claimants can demonstrate defendant failures in complying with the AI act norms (Art. 4).

Notwithstanding these measures, migrants face a difficult journey to obtain damages due to harmful algorithmic events. ⁵⁶ Migrants, refugees, and asylum seekers do not have the means to break through the algorithmic Black barrier, identify those potentially responsible for border control and surveillance activities, and demand the necessary evidence from them to prove their responsibility for the harm caused by AI-driven border operations.

E. Concluding Remarks

This contribution demonstrates the inadequacy of the current EU legal framework in protecting affected individuals from the consequences of algorithmic errors that occur at the EU’s external borders. Several technical and legal factors contribute to such a critical situation. On the one hand, the inherent opaqueness of the mechanisms by which the AI-powered software operates could prevent human observers from timely identifying and fixing algorithmic errors. Such AI programs continually change their cognitive processes to adapt to mutating external circumstances, making it technically difficult to detect AI errors and their causes. Furthermore, the networked nature of digital infrastructures at EU borders may favor the propagation of misleading AI outcomes.

On the other hand, the EU legal framework struggles to keep pace with rapidly evolving AI technologies. The lack of a proper legal definition of algorithmic errors testifies to the difficulties of integrating such an ever-changing world within legal norms and regulations. This crucial shortcoming also relies on the “technosolutionist” approach of EU policies that move responsibilities from human actors to technologies. ⁵⁷

The unwavering confidence in the reliability and impartiality of AI outcomes shown by EU policymakers and regulators prevents the problematization of the accuracy of AI-driven devices operating for public governance goals. According to a document on implementing biometric identification mechanisms within the Entry/Exit System, ⁵⁸ the European Commission accepts a margin of error of 0.1% for false positive matches (biometric matches not indicating the subject under exam) and 1% for missed identification outcomes. Such negligible percentages may extend to a significant number of individuals when addressing vast amounts of data (a 1% margin of error in 100,000 searches may affect 1,000 people). Tolerating specific margins of error implies political consequences: the EU and its Member States consider the migration context as an ideal laboratory to experiment on technological solutions without considering the effects that such technologies have on people on the move. ⁵⁹

The deployment of data-driven technologies is gradually transforming Eu external frontiers in social sorting sites that classify migrants accordingly to their data on the bases of policies of control and mistrust. The progressive “datafication” process of EU external frontiers is transforming borders into social sorting sites that address migrants as data to be processed to create their “data double” to manage based on policies of control and mistrust. ⁶⁰ Data-driven borders contribute to identifying individuals labelled as risks to EU stability and security, thus justifying restrictive migration policies because of assumptions of technological reliability. This contribution rejects the “dataism” paradigm, ⁶¹ believing in data as faithful representations of reality. Moving from these thoughts, a “dataistic” approach assumes that data processing activities can predict the future behaviors of targeted individuals. This Article suggests addressing digital border control policies from a data justice perspective, bringing into discussion the neutrality and reliability of data. Such a point of view aims to understand the sociopolitical dynamics that influence data elaboration practices and their products. Insofar as the scope of this Article is concerned, a data justice approach investigates the mutual relationship between algorithmic errors and the surrounding circumstances. In other words, it aims to problematize AI’s mistakes by understanding how these failures impact interested individuals while considering sociopolitical influences as possible causes of misleading algorithmic outputs.

Regarding data-driven border control practices, this Article also proposes a few solutions to mitigate the consequences of AI mistakes. First and foremost, clear governance frameworks should be prerequisites for deploying AI-powered technologies at EU external frontiers. This measure would help to attribute the responsibilities of the alleged wrongful conduct, thus overcoming algorithmic opaqueness. In addition, clear accountability schemes would provide interested individuals with indications on how to be held responsible for their rights and put data-driven borders under democratic scrutiny. Periodic checks should oversee the persistence of necessity and proportionality requirements to justify digital border control mechanisms. According to Art. 263 (5) TFEU, acts establishing EU bodies and agencies may lay down specific requirements that allow potentially interested individuals to bring actions against the measures producing legal effects against them. Moving from these considerations, introducing complaint procedures that address EU acts by relying on algorithmic errors could be crucial to ensure people on the move receive the highest fundamental rights protections and rule of law safeguards in the AI era.