Part I - Automated Banks

1 AI in the Financial Sector Policy Challenges and Regulatory Needs

Teresa Rodríguez de las Heras Ballell

1.1 Setting the Scene: AI in the Financial Sector

The progressive, but irrepressible, automation of activities, tasks, and decision-making processes through the systematic, pervasive application of AI techniques and systems is ushering in a new era in the digital transformation of the contemporary society and the modern economy.¹ The financial sector, traditionally receptive and permeable² to technological advances, is not oblivious to this process of intense and extensive incorporation of AI, for multiple purposes and under variegated forms.³ The advantages and opportunities that AI solutions offer in terms of efficiency, personalisation potential, risk management, and cost reduction have not gone unnoticed by the financial sector. On the contrary, the characteristics of AI systems seem to perfectly fit in with the features of financial services and to masterly address their most distinctive and challenging needs. Thus, the financial industry provides a receptive and conducive environment to the growing application of AI solutions.

Despite the spotlight on AI, the fact that AI solutions are usually applied, implemented, and incorporated in the financial activity in synergetic combination with other transformative and emerging technologies should not be disregarded. These are technologies such as big data, Internet of Things (IoT), cloud computing, distributed ledger technology (DLT), quantum computing, platform model, virtual reality, and augmented reality⁴ that are synchronously present in the market, with similar levels of technical maturity,⁵ commercial viability, and practical applicability. In fact, the multiplying effects triggered by such a combination of sophisticated technological ecosystems largely explain the perceived disruptive nature of AI and its actual impact.

With very diverse uses and applications, AI has penetrated financial markets across the board in an increasingly visible way.⁶ Its alliance with analytical and predictive processing of big data by financial institutions⁷ is perhaps the most telling dimension of a profound transformation of the industry, business strategies, risks, and operations.⁸

The perception of their usefulness⁹ and, above all, of the timeliness and desirability of their increasingly pressing incorporation has been encouraged by markedly different competitive conditions, precisely because of the impact of technology on market architecture and exceptional circumstances arising from the pandemic.¹⁰ Indeed, this process of intense digital migration has altered the structure and conditions of competition in the market with the opening of new niches for the emergence of innovative fintech firms¹¹ and the sweeping entry of Big Tech in the financial services sector. The essential function of financial markets as mechanisms for the efficient allocation of savings to investment can take many different forms. Technological innovation has endowed the sector with new architectures¹² on a continuum that shifts from platform models¹³ based on a centralised structure to decentralised or distributed models¹⁴ – to varying degrees – that DLT¹⁵ allows to articulate.¹⁶

Changes in market architecture and opportunities for the provision of new services and intermediation in the distribution of new financial assets and products have driven the emergence of new market players – crowdfunding platform operators, aggregators, comparators, robo-advisers, algorithm providers, social trading platform operators, and multilateral trading system operators – encouraged by low barriers to entry, promising business opportunities, cost reduction, and economies of scale.

In this new landscape, complex relationships of cooperation¹⁷ and competition¹⁸ are established between entrants and incumbents.¹⁹ The presence of new players in the market – offering complementary or instrumental services, creating new environments and channels of communication and intermediation, and adding value to traditional services and products – challenges the traditional scope of regulation and the classical limits of supervision.²⁰

On the other hand, mobility restrictions, with closures, confinements, and limitations on travel aimed at containing the spread of the Covid-19 pandemic from the first quarter of 2020, although temporary, have turned the opportunity of digital banking into a survival necessity and even an obligation, in practice, for the proper provision of service and customer care. In a fully digital context for all customer interactions and operations, the use of AI for optimisation, personalisation, or recommendation is key. The processing of increasing amounts of data requires automated means. At this forced and exceptional juncture, many digitalisation initiatives have been prioritised to meet the needs of the changed circumstances. A bank that has completed its digital migration is in a very favourable and receptive position for AI solutions.

This trend, as a response to market demands, is met with increasing regulatory attention seeking to unleash the possibilities and contain the risks of AI. The European Union (EU) provides a perfect illustration. Efforts to define a harmonised regulatory framework for the market introduction, operation, and use of AI systems under certain prohibitions, requirements, and obligations crystallised in the proposed Regulation known as the AI Act.²¹ From a sectoral perspective, the European Banking Authority (EBA) had already advocated the need to incorporate a set of fundamental principles to ensure the responsible use and implementation of safe and reliable AI in the banking sector.²² Indeed, promoting safe, reliable, and high-quality AI in Europe has become one of the backbones of the EU’s digital strategy as defined in the strategic package adopted on 19 February 2020. The White Paper on AI²³ and the European Commission Report on Safety and Liability Implications of AI, the Internet of Things and Robotics²⁴ define the coordinates for Europe’s digital future.²⁵ The Ethics Guidelines for Trustworthy AI prepared by the independent High-Level Expert Group on AI in the European Union,²⁶ which takes the EBA as a reference, marked the first step towards the consolidation of a body of principles and rules for AI – explainability, traceability, avoidance of bias, fairness, data quality, security, and data protection. But the legal regime for the development, implementation, marketing, or use of AI systems requires incorporating other rules found in European legislation, in particular, the recently adopted Regulations on digital services and digital markets – Digital Services Act (DSA)²⁷ and Digital Markets Act (DMA),²⁸ or in some of the forthcoming instruments related to AI liability. Even so, it does not result in a coherent and comprehensive body of rules relating to the use of AI systems in the banking sector. It is necessary to compose a heterogeneous and plural set of rules that derive from sectoral regulations, result from the inference of general principles, apply standards from international harmonisation instruments, or project the rules on obligations, contracts, or liability through more or less successful schemes based on functional equivalence and technological neutrality.²⁹

The aim of this chapter is to follow this path, which starts with the observation of a growing and visible use of AI in the financial sector, moves into the regulatory and normative debate, and concludes with a reflection on the principles that should guide the design, development, and implementation of AI systems in decision-making (ADM) in the sector. To this end, the chapter is structured as follows. First, it explores the concept of an AI system, considering definitions proposed in the EU, especially in the AI Act, and the interaction of this term with other related terms such as ADM (Section 1.2.1). The various applications of AI in the financial sector in general and in the banking sector in particular are then explored (Section 1.2.2). This provides the conceptual basis for analysing the regulatory framework, including existing and emerging standards, applicable to AI systems, and concludes (Section 1.3) with a proposal of the main principles that should guide the design, implementation, and use of AI systems in the financial sector (Section 1.4).

1.2 Concept and Taxonomy: AI System and ADM

The digital transformation is generating an intimate and intense intertwining of various technologies with socioeconomic reality. This implies not only recalibrating principles and rules, but also terminology and concepts. The legal response must be articulated with appropriate definitions and concepts with legal relevance that adequately grasp the distinctive features of technological solutions without falling into a mere technical description, which would make the law irremediably and forever obsolete in the face of technological progress. The law would rather opt for a functional categorisation, which understands the functions without prejudging the technological solution or the business model.

1.2.1 AI Systems: Concept and Definition

In the European legislation, whether in force or pending adoption, references to automation appear scattered and with disparate terminology. Both in the General Data Protection Regulation (GDPR),³⁰ or in the Digital Services Act references to automated individual decisions, algorithmic decisions, algorithmic content recommendation or moderation systems, algorithmic prioritisation, or the use of automatic or automated means for various purposes can be spotted. But there is no definition or explicit reference to ‘AI’ in the said texts. It is the future, and still evolving, AI Act that expressly defines ‘AI systems’, for the purposes of the regulation, in order to delimit its material scope of application.

The initial definition of AI system for the purposes of the proposed instrument in the European Commission’s proposal was as follows: artificial intelligence system (AI system) means ‘software that is developed using one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations or decisions that influence the environments with which it interacts’ (Art. 3.1 AI Act).

With this definition, AI systems are defined on the basis of two components. First, the qualification as learning systems and thus separated from more traditional computational systems. That is, the fact that they employ or are developed using ‘AI’ techniques, which the AI Act would define in an Annex I, subject to further extension or modification, and which currently includes: machine learning strategies, including supervised, unsupervised, and reinforcement learning, employing a wide variety of methods, including deep learning; logic and knowledge-based strategies, especially knowledge representation, inductive (logic) programming, knowledge bases, inference and deduction engines, expert systems and (symbolic) reasoning; statistical strategies, Bayesian estimation, search and optimisation methods. Second, the influence on the environment with which they interact, generating outcomes such as predictions, recommendations, content, or actual decisions. Behind this definition lies the assumption that it is precisely the ‘learning’ capabilities of these systems that largely determine their disruptive effects³¹ (opacity, vulnerability, complexity, data dependence, autonomy) and hence the need to reconsider the adequacy of traditional rules. This is, in fact, the reasoning that leads to rethink the rules of liability and thus assess their adequacy in the face of the distinctive features of AI as proposed in the report published on 21 November 2019, titled Report on Liability for Artificial Intelligence and other emerging technologies.³² And it was issued by the Expert Group³³ on New Technologies and Liability advising the European Commission.³⁴

Along the same lines, the Commission adopted two related proposals in 2022: proposal for a directive on adapting non-contractual civil liability rules to artificial intelligence³⁵ and proposal for a directive of the European Parliament and of the Council on liability for defective products.³⁶

However, the wording of this definition for AI systems in the Commission’s proposal has been subject to significant reconsideration and might still evolve into its final wording. The compromise text submitted at the end of November 2021 by the Slovenian Presidency of the European Council (Council of the European Union, Presidency compromise text, 29 November 2021, 2021/0106(COD), hereafter simply Joint Undertaking) proposed some changes to this definition.³⁷ The text, in its preamble, explains that the changes make an explicit reference to the fact that the AI system should be able to determine how to achieve a given set of pre-defined human objectives through learning, reasoning, or modelling, in order to distinguish them more clearly and unambiguously from more traditional software systems, which should not fall within the scope of the proposed Regulation. But also with this proposal, the definition of an AI system is stylised and structurally reflects the three basic building blocks: inputs, processes, and outputs.

Yet, a subsequent version version of the compromise text³⁸ of the AI Act offers another definition that refines the previous drafting and provides sufficiently clear criteria for distinguishing AI from simpler software systems. Thus, AI system means a system that is designed to operate with a certain level of autonomy and that, based on machine and/or human-provided data and inputs, infers how to achieve a given set of human-defined objectives using machine learning and/or logic- and knowledge-based approaches, and produces system-generated outputs such as content (generative AI systems), predictions, recommendations, or decisions influencing the environments with which the AI system interacts. Some key elements of the initial definition are preserved or recovered in this recent version that finally narrows down the description of ‘learning systems’ to the definition to systems developed through machine learning approaches and logic- and knowledge-based approaches.

The definition is still evolving. In the latest compromise text³⁹ the new definition of AI system is ‘a machine-based system that is designed to operate with varying levels of autonomy and that can, for explicit or implicit objectives, generate outputs such as predictions, recommendations, or decisions, that influence physical or virtual environments’.

Also, the European Parliament Resolution on liability for the operation of artificial intelligence systems⁴⁰ referred expressly to AI systems and formulated its own definition (Art. 3.a).⁴¹ This Resolution contains a set of recommendations for a Regulation of the European Parliament and of the Council on civil liability for damage caused by the operation of AI systems. The proposal has not been adopted. Instead, the Commission proposed the abovementioned tandem of draft Directives, that follow a substantially different approach aimed to revise the Defective Product Liability rules so as to accommodate AI-driven products and to alleviate the burden of proof in fault-based liability scenarios on damages caused by AI systems.

The rest of the regulatory texts do not explicitly refer to AI, although they contain rules on algorithms, algorithmic systems of various types, automation or automatic decision-making. Thus, as mentioned above, the GDPR, the DSA, the DMA or, among others, the P2B Regulation⁴² refer to algorithmic rating, algorithmic decision-making, algorithmic recommendation systems, algorithmic content moderation, algorithmic structures, automated profiling, or a variety of activities and actions performed by automated means. They include rules related to algorithms, such as disclosure, risk assessment, accountability and transparency audits, on-site inspections, obtaining consent, etc. As the definition of AI systems proposed by the AI Act reveals, recommendations, decisions, predictions, or other digital content of any kind, as well as actions resulting from the system in or in relation to the environment, are natural and frequent outputs of AI systems. Consequently, regulatory provisions that in some way regulate algorithmic processes and decision-making by automated means in a variety of scenarios and for a variety of purposes are also relevant for the construction of the regulatory framework for AI in the European Union.

Provided that the AI system falls under the scope of application of the proposed AI Act, an AI system may be subject to the AI Act as well as to other rules depending on the specific purpose, the purpose for which it is intended or the specific action. As an illustration, if the system is intended to produce recommendations by a very large banking platform, the DSA (Art. 27) – applicable to any online platform – applies, or if the system is intended for profiling, the GDPR (Art. 22) would be relevant.

In conclusion, understanding the complementarity between the various legal texts that directly or indirectly address the use of AI systems for a variety of purposes and from a range of legal perspectives is fundamental to composing the current and future regulatory framework for AI, as discussed below.

1.2.2 Current and Potential Uses of AI in the Financial Sector

With varying degrees of intensity, AI systems are used transversally in the banking sector along the entire front-line-mid-office-back-office value chain. For customer service and interaction, AI systems offer extraordinary possibilities for personalisation, recommendation and profiling, account management, trading and financial advice (robo advisers), continuous service via chatbots and virtual assistants, and sophisticated Know Your Customer (KYC) solutions.⁴³ In the internal management of operations, AI solutions are applied in the automation of corporate, administrative and transactional processes, in the optimisation of various activities, or compliance management. For risk management, AI solutions are projected to improve fraud prevention mechanisms, early warning and cybersecurity systems, as well as being incorporated in predictive models for recruitment and promotion. Another interesting use⁴⁴ of advanced analysis models with machine learning is the calculation and determination of regulatory capital. Significant cost savings are estimated⁴⁵ if these models are used to calculate risk-weighted assets.

Acknowledging this transversal and multipurpose use allows to anticipate some considerations of interest and relevance for legal analysis. It can be seen that automation has an impact on decision-making processes, actions, or operations of a diverse nature, which will be decisive in determining at least three elements.

First, the applicable regulatory regime – for example, whether it is used to automate compliance with reporting rules, to prevent fraud, to personalise customer offers, or to handle complaints via a chatbot. Second, the possible liability scenarios – for example, whether algorithmic biases and data obtained from social media for the credit scoring and creditworthiness assessment system could lead to systematic discriminatory actions. Third, the transactional context in which it is used – for example, in consumer relations with retail customers, in relations with the supervisor, or in internal relations with employees or partners.

The benefits deriving from the use of automation and AI and the expected gains from systematic and extensive application are numerous.⁴⁶ Algorithm-driven systems provide speed, simplicity, and efficiency in solving a multitude of problems. Automation drastically reduces transaction costs, enabling services that would otherwise be unprofitable, unaffordable, or unviable to be provided on reasonable and competitive terms. Cost reduction explains, for example, the burgeoning sector of robo-advisers⁴⁷ that have expanded the market beyond traditional financial advisers with appreciable benefits for consumers by diversifying supply, increasing competition, and improving financial inclusion.⁴⁸ Such expansion has facilitated financial advice to small investment and low-income investors on market terms.

ADM systems can therefore perform automated tasks and make mass decisions efficiently (high-frequency algorithmic trading, search engines, facial recognition, personal assistants, machine translation, predictive algorithms, and recommender systems). The use of automated means is critical for the large-scale provision of critical services in our society that would otherwise be impossible or highly inefficient (search, sorting, filtering, rating, and ranking).

However, the expansive and growing use of algorithms in our society can also be a source of new risks, can lead to unintended outcomes, have unintended consequences, or raise legal concerns and social challenges of many different kinds. ADM may be biased or discriminatory⁴⁹ as a result of prejudiced preconditions, based on stereotypes or aimed at exploiting user vulnerabilities, inadequate algorithm design, or an insufficient or inaccurate training and learning data set.⁵⁰ The automation of ADM makes bias massive, amplified and distorted, and easily gain virality. In a densely connected society such as ours, virality acts as an amplifier of the harmful effects of any action. Negative impacts spread rapidly, the magnitude of the damage increases, and the reversibility of the effects becomes less likely and increasingly impractical. The incorporation of decision and learning techniques into increasingly sophisticated AI systems adds to the growing unpredictability of future response. This leads to greater unpredictability and unstoppable complexity that is not always consistent with traditional rules and formulas for attribution of legal effects and allocations of risk and liability (infra 3.1.2.2 and 3.1.2.3).

1.3 An Initial Review of the Policy and Regulatory Framework in the European Union

The use of AI systems for decision-making and the automation of tasks and activities in the financial sector does not have a comprehensive and specific legal framework, either across the board or in its various sectoral applications.

The legal and regulatory framework needs to be assembled by the interlocking of legal provisions from various instruments and completed by the inference of certain principles from rules applicable to equivalent non-automated decisions. The application of the principle of functional equivalence (between automated and non-automated decisions with equivalent functions) guided by technological neutrality makes it possible to extract or extrapolate existing rules to the use of AI systems. However, as argued in the final part of this chapter, this effort to accommodate existing rules to the use of different technologies, under a non-discrimination approach on a medium basis, presents difficulties due to the distinctive characteristics of AI systems, thus compromising legal certainty and consistency. It is therefore suggested that a set of principles be formulated and a critical review of regulation be conducted to ensure that the European Union has a framework that provides certainty and encourages the responsible use of AI systems in the financial sector.

1.3.1 The Expected Application of the Future AI Law to the Uses of AI in the Financial Sector

The (future) AI Act is based on a risk-based classification of AI uses, applications, and practices, to which a specific legal regime is attached: prohibition, requirements for high-risk systems, transparency, and other obligations for certain low-risk systems. The classification of AI systems is not done on the basis of the employed technology but in conformity with the (intended, actual, reasonably expected) specific uses or applications. This means that there is no explicit sectoral selection, but certain practices can be identified with preferred sectoral uses such as creditworthiness assessment, and automated credit rating determination.

1.3.1.1 Prohibited Practices under the AI Act and Their Relevance to Financial Activity

The prohibited practices under Article 5 of the AI Act does not at a first sight naturally embrace the expected uses of AI in the financial sector, but, to the extent that they are defined on the basis of certain effects, they cannot be fully ruled out and should be taken into consideration as red lines. Thus, for example, a personalised-marketing AI system that uses subliminal techniques to substantially alter behaviour in a way that may cause physical or psychological harm (Art. 5.a) or a loan offering and marketing system that exploits any of the vulnerabilities of a group of people based on age, physical or mental disability, or a specific social or economic situation (Art. 5.b).

As initially drafted, although slightly nuanced in subsequent versions (in the Joint Undertaking), the scenarios for the use of biometric identification systems or the assessment or classification of the trustworthiness of natural persons according to their social behaviour or personality are less likely to cover AI applications in the financial sector. The reason is because the prohibition is linked to their use by (or on behalf of) public authorities, or in publicly accessible spaces for law enforcement purposes (although there are still scenarios in which they could apply, such as precisely banks, mentioned in Recital 9 as ‘publicly accessible spaces’). These requirements, questioned for being excessively restrictive, would leave outside the scope of prohibited use of the application in a private space – of an institution – of biometric recognition systems or even assessment systems (social scoring) that could be implemented to accompany a creditworthiness assessment or to profile the eligibility of applicants for banking products. Therefore, in the latest compromise text (14 June 2023), these restrictive criteria have been deleted. The prohibition is extended now to (Article 5.1.d) the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces.

While the potential impact of the AI Act’s on prohibitions of certain practices in the financial sector appears limited, the likelihood of these being systems classified as high risk is certainly much higher.

1.3.1.2 High-Risk Systems in the AI Act

Annex III of the AI Act provides a list of AI systems, related to eight areas (pursuant to the most recent version of the compromise text), defined by their use, purpose or aim, which will very easily reflect frequent applications of AI in the financial sector: systems for the remote biometric identification of natural persons (1.a Annex III), systems for recruitment or selection of natural persons or for making decisions on promotion or termination of employment relationships or assignment of tasks and monitoring and evaluation of performance (4.a and b Annex III), and, directly and obviously, systems for assessing the creditworthiness of natural persons or establishing their credit rating – with the exception of AI systems used for the purpose of detecting financial fraud – (5.b Annex III).

Confirming the application of the AI Act to certain uses of AI systems proposed by a financial institution will mean it being subject to certain more intensive requirements if it is qualified as high risk (Art. 8 AI Act). These are essentially audit, risk assessment and management, data governance (training, validation, and testing), technical documentation, event logging, cybersecurity, and transparency and reporting obligations to which financial institutions are by no means neither oblivious nor unfamiliar. They respond to a regulatory strategy of supervision and risk management that is well known in regulated sectors such as the financial sector. In fact, the need to avoid duplications, contradictions, or overlaps with sectoral regulations has been taken into account in the AI Act, in particular in relation to the financial sector already subject to risk management, assessment, and supervision obligations similar to those envisaged in the future Regulation – (see Recital 80, and Articles 17.3, 18.2, 20.2, 29.4, 61.4, 62.3). In this regard, the AI Act articulates some solutions to ensure consistency between the obligations of credit institutions under Directive 2013/36/EU⁵¹ when they employ, operate, or place on the market IA systems in the exercise of their activity.

1.3.2 Principles and Rules for the Use of AI Systems in Decision-Making

However, the eventual application of the AI Act does not exhaust the regulatory framework of reference for the use of AI systems in the financial sector, nor, in fact, does it resolve a good number of questions that the implementation and subsequent operation of such systems in the course of their activity will generate. To this end, and for this reason, it is essential to explore other regulatory instruments and to discover legal avenues to answer a number of important questions. First, to what extent can automated systems be used with full functional equivalence for any activity, decision, or process without prior legal authorisation? Second, to what extent are decisions taken or assisted by AI systems attributed to the financial institution operating the system? Third, who bears the risks and liability for damage caused by the AI systems used?

1.3.2.1 On the Principle of Non-discrimination for the Use of AI in Decision-Making

Neither the AI Act nor, in principle, any other regulation expressly enables the use of AI systems to support decision-making or to automate specific tasks, processes, or activities.⁵² Occasionally and incidentally, reference to automation is found in some texts, even simply in the recitals – Regulation (EU) 2020/1503/EU⁵³ to automatic investment in par. 20 – without further specification or development in the legal provisions. In other cases, this possibility is confirmed because reference is made to ‘with or without human intervention’ or ‘by automatic means’, as in the DSA – Art. 3(s) on recommender systems, Art. 16(6) on means of notification and action, Art. 17(3) on the statement of reasons. And in other cases, an express limitation to full automation of a decision-making process such as complaints handling on a platform – DSA, Art. 20(6) – is provided for.

Within this regulatory context, the question on the admissibility, validity, and enforceability of the use of AI systems must be approached on the basis of two backbone principles: the principle of functional equivalence and the principle of non-discrimination and technological neutrality. These principles lead to a positive and enabling initial response that allows the use of AI systems for decision-making or to assist in decision-making, to automate tasks, processes, and activities in a general way and without the need for prior express legislative recognition. There is no reason to deny this functional equivalence or to generally discriminate against the use of AI systems under analogous conditions. Subject-specific limits or sector-specific regulatory requirements might in practice restrict certain applications in the financial sector, but the basic rule is the feasibility of using AI in any activity and for any decision-making.

Naturally, the implementation of an IA system will require ensuring that the automated process is in compliance with the rules applicable to the same process, situation, or transaction if it were not automated. AI systems have to be designed, implemented, and operated in such a way that they comply with the rules that would apply to the legal nature of the decision or activity and, therefore, also to its regulatory treatment in the financial field. If the marketing of certain financial products is automated through a digital banking application, it should be ensured that the legal requirements for pre-contractual information are met. If an automated robo-adviser system is implemented, the requirements for financial advice must be met, if it is indeed categorised as such.

Despite the apparent simplicity of this principle of non-discriminatory recognition of AI, its effect is intense and powerful. It constitutes, in practice, a natural enabler for the multiple and intensive integration of AI in any area of financial activity, as a principle. As long as compliance with the rules and requirements applicable to the action or the equivalent non-automated process can be ensured, AI can be employed to make or assist in making any decision.

1.3.2.2 On the Attribution of Legal Effects

The particular complexity in the chain of design, development, implementation, and operation of AI systems with a set of actors involved, very often without prior agreement or coordination among them, raises a legal question of indisputable business relevance: to whom the legal effects, and thus the risks of a decision, or an action resulting from an automated process, are attributed.

Although this issue can be interpreted as a single attribution problem from a business perspective, from a legal point of view, it is useful to distinguish between two different, albeit related, issues.

First is the question to whom the decision – any decision with contractual relevance (offer, acceptance, modification, renegotiation, termination) – or the resulting action – commercial practice, compliance with supervision request – is to be attributed. That is, if a bank implements an application that incorporates a credit scoring system leading to the automated granting or refusal (without human intervention in each decision) of consumer credit applications, assessing creditworthiness and the decision to accept or deny the credit request are attributable to the bank. Thus, if the credit is granted, the bank is the counterparty to the resulting credit contract; whereas, if it is unjustifiably denied, discriminating against certain groups, the bank would be the offender, violating, for example, the right not to be discriminated against. Similar reasoning would apply to the use of an AI system in an employee recruitment or promotion programme, or to a fraud detection and prevention system.

This attribution of legal effects is based on the formulation of the concept of ‘operator’. This concept proposed by the Report on Liability for Artificial Intelligence and Other Emerging Technologies and subsequently taken over by the European Parliament Resolution of 2020⁵⁴ is based on two factors: control and benefit. Thus, the operator will be the centre of imputation of the legal effects insofar as it controls (or should be able to control) the risks of operating an AI system that it decides to integrate into its activity and, therefore, benefit from its operation.

This attribution of legal effects to the operator also has another important consequence. The operator cannot hide behind the automated or increasingly autonomous nature of the AI system used in order not to assume the consequences of the action or decision taken, nor can the bank consider attributing such effects to other actors involved in the life cycle of the AI system. Thus, for example, it cannot be attributed to the developer of the system, the distributor, or the provider of the data per se and vis-à-vis the bank customer. This is without prejudice to the possibility for the operator (the bank) to bring subsequent actions or remedies against these actors. However, the operator is who assumes the legal – legal or contractual – effects vis-à-vis the affected person concerned (customer).

Second, a question arises as to who should bear the risks and liability for damage caused by the operation of AI systems, as expounded below.

1.3.2.3 On Liability for Damage Caused by the Operation of AI Systems

The operation of an AI system can cause a wide range of damages. In certain sectors, substantial property damage and personal injury can be anticipated (autonomous vehicles, drones, home automation, care robots). Their applications in financial activities are linked to systemic risks, threats to economic stability and financial integrity, or cyclical responses and market shocks. But their malfunctioning can also simply cause massive data loss, disrupt access to services and products, generate misleading messages to customers about the status of their accounts, recommend unsuitable investments according to risk profile, or result in non-compliance with certain obligations vis-à-vis supervisory authorities. The use in rankings, recruitment services, content filtering, or virtual assistants for complaint handling opens the door to a far-reaching debate on their impact on fundamental rights and freedoms – freedom of expression, the right not to be discriminated against, the right to honour, and personality rights – but also on the competitive structure of the market or on the fairness of the commercial practices. Hence, the approach adopted by the proposed AI Act in Europe is based on the identification of certain AI practices, uses, or applications which, due to their particular risk or criticality, are prohibited, qualified as high risk and therefore subject to certain obligations and requirements, or subject to harmonised rules regulating their introduction on the market, their putting into service, and their use.⁵⁵

However, in the face of such potentially negative effects, the fundamental question is whether, beyond the adoption of specific rules for AI systems aimed at controlling their use and mitigating their negative effects, traditional legal liability regimes are adequately equipped to manage the risks and effectively resolve the conflicts arising from such situations in complex technological environments.

In this respect, the European Union faces important legislative policy choices. First, to assess whether a thorough reform of the product liability regime⁵⁶ is necessary to accommodate AI systems.⁵⁷ The questions are manifold: are AI systems products?, is a decision of the AI system that causes damage necessarily the result of a defect?, and do the provisions of the Directive work adequately in the face of an AI system that has been updated since it was put on the market? Second, to consider whether it is appropriate to establish a harmonised liability regime specific to AI, as suggested in the abovementioned Parliament Resolution,⁵⁸ and if so, whether it should be an operator’s liability and whether the distinction between strict liability for high-risk systems and fault-based liability for the rest is appropriate. The Proposal of the Commission in 2022⁵⁹ departs from the route initiated by the Parliament in 2020 as it proposes a Directive instead of a Regulation, and it puts forward a minimum and complementary harmonisation to national rules on (fault-based) (non-contractual) civil liability in a targeted manner with rules on specific aspects of fault-based liability rules at Union level.

1.4 Concluding Remarks: Principles for the Responsible Use of AI in Decision-Making

The principle of non-discrimination against the use of AI systems in any activity and for any decision-making enables intense and extensive automation in the banking (financial) sector through the implementation of AI solutions. Within this favourable and automation-friendly framework, compliance with the regulatory requirements demanded by the nature of the sectoral activity (law-compliant AI systems) must nevertheless be ensured and some specific limitations must be added which, by reason of their use or purpose (e.g. credit scoring, recruitment and promotion, biometric recognition), the future AI Act could prohibit or subject to certain obligations. To the extent that these AI systems are also employed to provide recommendations, personalise offers, produce rankings, or moderate content, additional rules (DSA, DMA, GDPR) could apply if they are used by financial institutions that have transformed their business model into an online platform.

Even so, there is neither a compact and coherent set of principles capable of guiding automation strategies nor a comprehensive body of rules that would provide full legal certainty for the implementation of AI systems in the banking sector. The highly distinctive characteristics of AI do not always make an application of existing rules under a technology-neutral and functional equivalence approach fully satisfactory, nor are the existing rules always feasible or workable in the AI context. Therefore, there are calls in the European Union for the complementation of the legal framework with other specific principles to crystallise a body of rules suitable for AI. The EBA also advocated for this strategy at sectoral level.

The formulation of ethical principles is certainly a starting point, but the integration of AI systems in the course of an economic activity, throughout the transactional cycle and for business management requires a clear framework of duties and obligations. This is the endeavour that policymakers in the European Union and internationally must face now.⁶⁰ It is necessary to specify how AI systems should be designed, implemented, and commissioned to satisfy the principles of traceability, explainability, transparency, human oversight, auditability, non-discrimination, reasoned explanation of decisions, and access to a review mechanism for significant decisions. It will be key to understand how the provisions of the future AI Act interact with contract law and liability rules,⁶¹ to what extent the classification of an AI system as high risk under the AI Act could imply the application of a strict liability regime (as previously proposed under the Parliament’s resolution scheme, even if this approach has not been followed by the recent Commission’s proposals for Directives), or what effects the failure to articulate a human-intervention mechanism under Art. 22 GDPR would have on the validity and effectiveness of an automated decision based on profiling, or what implications the failure of the bank operator to comply with the requirements of the AI Act would have on the validity and the enforceability of the contract or on the eventual categorisation of certain bank practices as unfair commercial practices.It is essential for financial firms, referred to in this book as Automated Banks, to be provided with clear and coherent rules for the use and implementation of AI systems in decision-making. The law must be developed in combination with, and accompanied by, detailed (technical) standards, best practices, and protocols progressively and increasingly harmonised in the financial sector.

2 Demystifying Consumer-Facing Fintech Accountability for Automated Advice Tools

Jeannie Paterson , Tim Miller , and Henrietta Lyons

2.1 Introduction: Money, Power, and AI

As the authors of this book recognise, money and power are intimately linked. For most consumers, access to banking services, credit, and a saving plan for retirement are necessary – although not sufficient – requirements for a stable, meaningful, and autonomous life. Conversely, financial hardship may have considerable impact on not only the financial but also the emotional well-being of consumers.¹ There are many causes of financial hardship, including high levels of personal debt, reliance on high-cost credit, lack of access to mainstream banking services, and unexpected circumstances such as unemployment or ill health.² Additionally, consumers are sometimes subject to fraudulent, deceptive, and dishonest practices, which can escalate their financial problems.³ Moreover, many consumers find that they lack the time or skills to manage their day-to-day finances, select optimal credit products, or invest effectively for the future.⁴

So where does AI⁵ – the third theme of this book – sit in this schema? The growing capacity of AI and related digital technologies has contributed to a burgeoning interest in the potential for financial technology (‘fintech’) to transform the way in which traditional banking and financial services are provided.⁶ Governments across the globe have promoted the capacity of AI informed fintech to improve market competition and consumer welfare,⁷ and have introduced initiatives to support the development of innovative fintech products within their jurisdictions.⁸ Fintech products are increasingly being used by the financial services sector for internal processes, decision-making, and interactions with customers.⁹

Inside financial institutions, fintech products are assisting in fraud detection, cybersecurity, marketing, and onboarding new clients.¹⁰ Fintech products are being developed to automate financial services firms’ decisions about lending, creditworthiness, and pricing credit and insurance.¹¹ In a consumer-facing role, fintech products are being used for communicating with customers, such as through chatbots (generative or otherwise),¹² and in providing access to financial products, for example, loan¹³ or credit card online applications.¹⁴ Fintech products are being developed to provide credit product comparisons for consumers looking for the best deal.¹⁵ However, the most common forms of consumer-facing fintech are, at the time of writing, financial advice tools¹⁶ primarily for investing and budgeting.¹⁷

Consumer-facing fintech generally, and automated financial advice tools specifically, are often promoted as benefiting consumers by assisting them to make better decisions about credit, savings, and investment, and by providing these services in a manner that is more cost-effective, convenient, and consistent than could be provided by human advisers.¹⁸ These features undoubtedly hold attractions for consumers. However, in our opinion, the allure of AI, and its financial market equivalent of fintech, should not be allowed to overshadow the limitations of, and the risks of harm inherent in, these technologies. As this book makes clear, whether used by governments or private sector firms, AI and automated decision-making tools raise risks of harm to privacy, efficacy, bias, and perpetrating existing power hierarchies. Albeit on a different scale, consumer-facing fintech, such as automated financial advice tools, carry many of the same kinds of risks, which equally demand regulatory attention and best practice for good governance. There has been little assessment of whether automated financial advice tools are effective in achieving improving the financial well-being of consumers. It is also unclear whether and to what extent such tools are equitable and inclusive, or conversely amplify existing bias or patterns of exclusion in financial services and credit markets.

Some of the potential risks of harm to consumers from automated financial advice tools will be addressed by existing law. However, we argue that there is a need to move past the commercial, and indeed political, promotion of ‘AI’ and ‘fintech’ to understand their specific fields of operation and demystify their scope. This is because the use of AI in this equation is not neutral or without friction. Automated advice tools raise discrete and unique challenges for regulatory oversight, namely opacity, personalisation, and scale. We therefore suggest, drawing on the key principles propounded in AI ethics frameworks, that the effective regulation of automated financial advice tools should require greater transparency about what is being offered to consumers. There should also be a regulatory commitment to ensuring the outputs of such tools are contestable and accountable, having regard to the challenges raised by the technology they utilise.

This chapter explores these issues, beginning with an overview of automated financial advice, focusing on what are currently the most widely available tools, namely ‘robo’ investment advice and budgeting apps. We discuss the risks of harm raised by these uses of AI and related technologies, arising from uncertainty about the quality of the service provided, untrammelled data collection, and the potential for bias, as well as the need for a positive policy focus on the impact of such tools on goals of equity and inclusion. We review the guidance provided by regulators, as well as the gaps and uncertainties in the existing regulatory regimes. We then consider the role of principles of transparency and contestability as preconditions to greater accountability from the firms deploying such tools, and more effective oversight by regulators.

2.2 Aspiration and Application in Consumer-Facing Fintech

The term ‘fintech’ refers to the use of AI and related digital technologies to deliver financial products and services.¹⁹ The AI used to deliver fintech products may include natural language processing in front-end interfaces to communicate effectively with clients and statistical machine learning models to make predictions that inform financial decision-making. ‘Consumer-facing’ fintech refers to the use of fintech to provide services to consumers, as opposed to use by professional investors, business lenders, or for back-room banking processes. As already noted, perhaps the most prominent form of fintech service offered to consumers, as opposed to informing the internal processes of financial institutions, is automated financial advice, primarily about investing and budgeting.

The aims of most fintech products are to allow services to be delivered at scale, reducing human handling of information, and, in the case of consumer-facing fintech, benefiting consumers. Automated financial advice tools typically purport to offer a low-cost option for financial advice derived from insights from consumer data and statistical analysis and provided through an accessible interface using state-of-the-art processing to identify and respond to consumers’ financial aims. The commonly stated aspiration of governments and regulators in supporting the development of these and other fintech products is to promote innovation and to provide low-cost, reliable, and effective financial services to consumers.²⁰ Some fintech providers express aspirations to be more inclusive and empower ordinary people to participate in the financial and banking sectors.²¹

There are undoubted attractions in such aspirations.²² The majority of consumers do not seek financial planning advice,²³ probably because it is perceived as being too expensive.²⁴ Yet many consumers find financial matters difficult or confusing. This is due to a combination of factors, including low financial literacy, limits on time, and the impact of behavioural biases on decision-making. In principle, automation should allow financial services providers to lower the cost and improve the consistency of advice,²⁵ as well as providing the convenience of an on-demand service.²⁶ Additionally, by using consumers’ own data, automated financial advice tools have the potential to be uniquely tailored to those consumers’ individual circumstances.²⁷ Indeed, this is one of the premises behind Australia’s consumer data right, which aims to give consumers control over their data to promote innovation and competition in the banking sector.²⁸

Currently, the two main kinds of automated financial advice tools are robo-advisers and budgeting apps.²⁹ Though these tools will no doubt evolve, they provide a simpler, less personalised service than might be envisaged by the ‘AI’ label commonly attached to them.

2.2.1 Robo-Advisers

Robo-advisers³⁰ provide ‘automated financial product advice using algorithms and technology and without the direct involvement of a human adviser’.³¹ In principle, robo-advice might cover automated advice about any topic relevant to financial management, such as budgeting, borrowing, investing, superannuation, retirement planning, and insurance. Currently, most robo-advisers provide automated investment advice and portfolio management.³²

Typically, robo-advice services begin with consumers answering a questionnaire about their goals, expectations, and aptitude for risk. An investment profile for consumers is derived from this information, based on their goals and capacity to bear risk. An algorithm matches consumers’ profiles with an investment portfolio available through the advisory firm to produce an investment recommendation.³³ Should a consumer choose to follow the advice and invest in that portfolio, many robo-advisers will also manage the portfolio on an ongoing basis, keeping it within the parameters recommended for the consumer. Consumers generally pay a fee for the service provided by the robo-adviser, often a percentage of the amount invested, with minimum investment amounts required to access the service.

Robo-advice is sometimes described as ‘trading with AI’.³⁴ This language might be thought to suggest specialised insights into the stock market uniquely tailored to consumers’ needs and arrived at through sophisticated machine learning models. The practice is more straightforward. At the time of writing, robo-advisers do not rely on state-of-the-art AI technology, such as using neural networks to process data points and make predictions about stock market moves, or link individual profiles to unique investment strategies. As Baker and Dellaert explain, the matching process will be based on ‘a model of how to optimise the fit between the attributes of the financial products available to the consumer and the attributes of the consumers who are using the robo-advisor’.³⁵ The robo-adviser will typically build the consumer profile based on the entry questionnaire and match this with an investment strategy established using financial modelling techniques and based on the investment packages already offered by firm. The process will usually have been automated through some form of expert system – a hand coded application of binary rule identified by humans. Ongoing management of the consumer’s portfolio will be done on a similar basis, often using exchange-traded funds (ETFs) that ‘require no or less active portfolio management’.³⁶

Unlike human financial advisers, robo-advice tools typically do not provide budgeting or financial management advice to consumers.³⁷ Their recommendations are limited to the kinds of investment that will match consumers’ investment profiles. Robo investment advisers do not provide advice on matters of tax, superannuation, asset management, or savings, and they do not yet have the capacity to provide this more nuanced advice.³⁸ Sometimes robo-advice tools are used in conjunction with human financial advisers who will provide a broader suite of advice. Automated budgeting tools are also increasingly available on the market.

2.2.2 Budgeting Tools

Budgeting tools allow consumers to keep track of their spending by categorising expenses and providing dashboard-style visualisations of spending and saving.³⁹ Some banks offer budgeting tools to clients, and there are many independent service providers. Some neo-banks have, additionally, consolidated their brand around their in-built budgeting tools.⁴⁰

As with robo-advisers, automated budgeting tools collect information about consumers through an online questionnaire. Budgeting tools also typically require consumers to provide access to their bank accounts, in order to scrape transaction data from that account,⁴¹ or alternatively rely on data-sharing arrangements.⁴² Based on this information, the services provided by budgeting tools include categorising and keeping track of spending; providing recommendations about budgeting; and monitoring savings.⁴³ In some cases, the tools will transfer funds matching consumers’ savings goals to a specific account, provide bill reminders, make bill payments, monitor information about credit scores, suggest potential savings through various cost-cutting measures or identifying alternative service providers.⁴⁴ Additionally, automated budgeting tools may provide articles and opinion pieces about financial matters, such as crypto, non-fungible tokens (NFTs), or budgeting.⁴⁵ Some budgeting tools have a credit card option,⁴⁶ and at least one is linked to a ‘buy now-pay later’ provider.⁴⁷

Automated budgeting tools often describe their service as relying on AI.⁴⁸ Again, however, they do not, as might have been expected from this terminology, typically provide a personalised plan for saving derived from insights from multiple data points relating to consumers. They may use some form of natural language processing to identify spending items. Primarily, somewhat like robo-advisers, they rely on predetermined, human-coded rules for categorising spending and presenting savings. Most budgeting tools are free, although some charge for a premium service. This means that the tools are funded in other, more indirect ways, including through selling targeted advertisements on the app, fees for referrals, commissions for third-party products sold on the app, the sale of data (usually aggregated), and in some cases a percentage of the savings where a lower cost service or provider is identified for consumers.⁴⁹

2.3 Regulation and Risk in Consumer-Facing Fintech

This brief survey of available automated financial advice tools aimed at consumers suggests that they are operating with a fairly narrowly defined scope and using relatively straightforward digital processes. The tools may evolve into the future to make greater use of such state-of-the-art AI, such as using generative AI for providing general advice to consumers. However, even in their current form, the tools pose risks of harm to consumers that are more than fanciful, and similar to those raised by AI generally. The risks arising from AI are becoming increasingly well recognised, including poor efficacy, eroding privacy, data profiling, and bias and discrimination.⁵⁰ These risks are also inherent in consumer-facing fintech and automated financial advice tools. Moreover, we suggest they are only partially addressed by existing law. While financial services law commonly imposes robust obligations on those providing financial advice, those obligations may not squarely address the issues arising from the automated character of the advice, particularly issues of bias. Additionally, some automated advice tools, such as budgeting apps, may fall outside of these regimes. It is therefore worth considering these issues in more detail.

2.3.1 Quality of Performance

One of the notable features of automated financial advice is that consumers are unlikely to be able to scrutinise the quality of the service provided. Consumers will typically turn to automated advice tools because they lack skills in the relevant area, be it investing or budgeting. This lack of expertise makes it difficult for them to assess the quality of the advice they receive.⁵¹ There is not a lot of information for consumers in selecting between different tools, as compared to standard consumer goods. While some rankings of automated financial advice tools have emerged, these often focus on ease of use – the interface, syncing with bank data, fees charged – rather than the quality of the advice provided,⁵² and some ranking reviews include sponsored content.⁵³ Accordingly, at least at this point in time, automated financial advice tools may be very much a credence good – for which assertions of quality are all that is available to consumers. Unless the advice provided by the tools is patently bad, it may not be apparent that the poor quality of the automated process is to blame, as opposed to other external factors. Indeed, without a point of comparison, which is effectively excluded by the personalised nature of the service, it may be difficult for consumers to identify poor quality advice at all.

There is currently little academic research on the extent to which consumers are well-served by automated financial advice tools, particularly when weighted against possible costs in terms of data-sharing.⁵⁴ There have been a number of concerns raised in the literature about how well the tools may function. Although robo-advisers may operate in a manner that is more objective and consistent than human financial advisers,⁵⁵ this does not mean they operate free from the influence of commissions, which may be coded into their advisory process. It is unclear to what extent the recommendations provided by automated financial advice tools are personalised to consumers, as opposed to being generic or based on broad target groupings. Additionally, concerns have been raised about the relatively small number of investment options actually held by robo-investment advisers.⁵⁶ While automated budgeting tools may assist consumers by providing an accessible, straightforward, and visual way of monitoring spending,⁵⁷ this does not necessarily translate into long-term savings⁵⁸ or improved financial literacy.⁵⁹ It is further possible that one of the main functions of at least some budgeting apps is to obtain consumers’ attention in order to market other financial services, such as credit cards, as well as the opportunity for the providers to profit from the use or sale of consumer data for marketing and data analytics.⁶⁰

In consumer transactions – particularly those that are complex, hard for consumers to monitor, or which carry the risk of high impact harms – reliance is usually placed on regulators to take ‘ex ante’ measures for ensuring that the products supplied to consumers are acceptably safe and reliable. Financial services regulators in jurisdictions such as Australia, the United Kingdom, the European Union, and the United States of America have responded to the rise of robo-advisers by affirming that the existing regulatory regime applies to this form of advice.⁶¹ Financial services providers are typically subject to an array of statutory conduct obligations, which overlap, albeit imperfectly, with their fiduciary duties arising under general law.⁶² These statutory duties require firms to manage conflicts of interest,⁶³ act in their clients, best interests,⁶⁴ ensure the suitability of the advice provided,⁶⁵ and take reasonable care in proving the advice.⁶⁶ These obligations should, in principle, assist in addressing concerns about the quality of the service provided by robo-advisers.⁶⁷ Nonetheless, some uncertainties remain, including, for example, whether the category-based approach deployed by robo-advisers fits with statutory requirements for personalised advice that is suitable for the individual consumer.⁶⁸

Regulators have additionally stated they expect firms providing robo-advice to have a ‘human in the loop’, in the sense of a person with ‘an understanding of the technology and algorithms used to provide digital advice’ and who are ‘able to review the digital advice generated by algorithms’.⁶⁹ Recommendations for a human overseeing the automated advice leave open the question of what that human should be monitoring – is it merely compliance with existing law applying to the giving of advice, or should there be other considerations taken into account, arising from the automated character of the advice?

In terms of the issue of automation, regulators have focused on the informational aspects of the process. They have emphasised that firms providing automated advice should give consideration to the way in which the information on which the advice is based is collected from consumers so as to ensure it is accurate and relevant, especially because there is no human intermediary to pick up possible discrepancies or errors. Regulators have also advised firms to take care in the way the advice is framed and explained, given the potential for misunderstanding and error in an automated process.⁷⁰ Issues of information gathering and reporting are important but they are only part of the challenge presented by automation for consumer protection law and policy. Moreover, they tend to represent a very individualised response to the risks of harm to consumers relying on automated financial advice, focusing on what consumers need to provide and understand, as opposed to the substance of the process through which advice is provided.

Notably, there is typically no specific law or regulatory guidance that applies to automated budgeting tools, which do not involve financial services. These tools will be subject to general consumer protection regimes, which typically prohibit misleading conduct, and mandate reasonable care and skill in the provision of services.⁷¹ Uncertainties about the application of existing law to automated advice give rise to the question of whether other kinds of regulatory mechanisms are required to complement sector-specific or general consumer protection law in order to address the risks of harms that are specific to the use of AI and related digital technologies. In answering this question, we suggest that, at minimum, the risks around data collection and bias need to be considered.

2.3.2 The Data/Service Trade-Off

Automated financial advice tools operate on the core premise that consumers necessarily hand over data to obtain the service. A firm may be using consumer data for the dual purposes of providing advice and making a return for itself, such as through promoting other products for a commission on sales, up-selling add-on products for a fee, or on-selling the data for profit.⁷² This behaviour is particularly apparent in the case of budgeting apps, which are typically free. As already noted, these services earn income through in-app advertising, fees, and commissions for referrals and potentially through selling aggregated consumer data, as well as targeted advertising. Notably, the privacy terms of automated budgeting tools commonly allow the collection of a wide range of consumer data and the use of that data for a number of purposes, including improving the service and related company group services, marketing, and, in aggregated form, sharing with third parties.⁷³

Data protection and privacy law impose obligations on the collection and processing of data.⁷⁴ However, the key requirements of notice and consent typically found under these regimes may easily be met in automated advice contexts because the exchange is at the heart of the transaction. Consumers provide their data in order to obtain the advice they need. While consumers may be unaware of how much information they are handing over, there is some evidence that consumers, particularly younger consumers, are prepared to trade data for cheaper, more efficient financial services.⁷⁵ However, to the extent consumers are ill or under-informed about the quality of the service being provided by automated advice tools, the data-for-service bargain may look thinner than they might have at first thought.⁷⁶ Under the fintech service model, consumers provide personal data to obtain a personalised and cost-effective service but have few objective measures as to the quality of what is actually being provided.

2.3.3 Bias and Exclusion

In discussing legal and regulatory responses to the growing influence of AI and related technologies, much attention has rightly been given to their role in amplifying surveillance, bias and discrimination.⁷⁷ The technologies may use personal data to profile consumers, which in turn allows firms to differentiate between different consumers and groups with a high degree of precision, leading to risks of harmful manifestations of targeted advertising, or differential pricing.⁷⁸ Bias and error are particular concerns in firms’ use of AI technologies for decision-making, including in decisions about lending,⁷⁹ credit,⁸⁰ or insurance.⁸¹ Automated lending decisions and credit scoring might be more objective than human-made decisions and might benefit cohorts that have previously been disadvantaged by human prejudice.⁸² But there is no guarantee this is the case, and indeed the outcomes may be worse for these groups. Differential treatment of already disadvantaged groups – such as minoritiy or low-income cohorts – may already be embedded in the practices and processes of the institution. To the extent this data is used in credit-scoring models or to inform automated decisions, historical unequal treatment may be amplified⁸³ or distorted.⁸⁴ Unequal treatment may, moreover, be difficult to identify or address where it is based, not directly on protected attributes, but on proxies for those attributes found in the training data.⁸⁵

Bias may also be embedded in automated advice tools used by consumers. For example, a robo-advice tool might exhibit bias by treating a person who takes time off work for childrearing as going through a period of precarious employment or being unable to hold down steady employment. An automated budgeting tool might exhibit bias by characterising products for menstruation as discretionary spending, instead of essentials. There are complex technical and policy decisions to be made in identifying and responding to the risks of unacceptable bias in automated financial advice tools.⁸⁶ Consumer protection and financial services law have not traditionally have not been central to this process, which is primarily the domain of human rights law. However, decisions based on historical prejudice may be unconscionable or unfair, contrary to consumer protection law. Certainly, in the United States, the Federal Trade Commission has indicated that discriminatory algorithms would fall foul of its jurisdiction to respond to unfair business practices.⁸⁷

A related issue concerns financial exclusion. Fintech innovators and government initiatives to encourage innovation often refer to an aspiration of promoting inclusion and overcoming exclusion.⁸⁸ There are few findings on the extent to which this aspiration is achievable. There are plausible reasons why automated advice tools may fail to assist, or assist adequately, consumers already excluded from mainstream financial or banking services, or consumers who have had less engagement with the mainstream banking system, such as where they are ‘not accessing or using financial services in a mainstream market in a way that is appropriate to their needs’.⁸⁹ Financially excluded consumers might not be offered meaningfully relevant advice tools because there is no relevant or useful data about them or because they are unlikely to be sufficiently profitable for financial services providers to develop products suited to them. These consumers may also find that the models on which the advisory tools are based are inaccurate when applied to their circumstances.

For example, investment tools may be of little value to consumers struggling to make ends meet and with no savings to invest. The models used by automated budgeting tools may have a poor fit with consumers living on very low incomes and for whom cutting back on discretionary spending is not an option available. In these circumstances, the tools will do little to improve equity, leaving unrepresented groups without advice, or relevantly personalised advice. Moreover, there may be a real risk of harm. Inept recommendations may subject consumers to harms of financial over-commitment or lull inexperienced consumers into a false sense of financial security. At a more systematic level, the availability of automated advice tools for improving financial well-being may feed into longstanding liberal rhetoric about the value of individual responsibility, as opposed to government initiatives for improving overall financial well-being.

It is possible to envisage services that would be useful to financially excluded consumers or consumers experiencing financial harshi, such as for example, advice on affordable loans and other services.⁹⁰ Emma Leong and Jodi Gardner point to proposed uses of Open Banking in the United Kingdom to provide tools that assist with better managing fluctuating incomes.⁹¹ The United Kingdom Financial Conduct Authority notes there are some apps on the market providing legal aid and welfare support advice.⁹² These kinds of initiatives are likely to require a deliberate policy decision to initiate rather than arising ‘naturally’ in the market.⁹³ This is because there would seem to be little commercial incentive for firms to invest in tools specifically tailored to low-income or otherwise marginalised consumers from whom there is little likelihood of ongoing lucrative return to the firm, without government support.

2.4 New Regulatory Responses to the Risks of Automated Financial Advice

Automated financial advice tools illustrate the continuing uncertainties in regulating consumer-facing fintech and AI informed consumer products. We have seen that regulators will need to adapt existing regimes to the new ways in which services are being provided to consumers, which requires attention not only to the risks in providing advice but in the automation of advice. We further suggest that regulators need to be cognisant of the ways in which the AI and digital technologies informing the tools raise unique challenges for regulation. Opacity is a key concern in any regulatory response to making AI systems more accountable.⁹⁴ Automated financial advice tools may not currently rely on sophisticated AI, in the sense of deep learning or neural networks. Nonetheless, they are for commercial (if not technical) reasons highly opaque as to the technology being utilised and how recommendations are reached. Their very purpose is to provide advice without significant human intervention and at scale, which may amplify harms of bias or error in the system.⁹⁵ The tools typically purport to provide output on factors personal to the consumer, which may make it difficult to determine whether an adverse outcome is unfortunate, a systematic error or failure of a legal duty.⁹⁶

One response to navigating the challenges of regulating consumer-facing fintech is provided by the principles of ethical AI.⁹⁷ Principles of AI ethics are sometimes criticised as too general to be useful.⁹⁸ The principles operate as a form of soft law – they are not legally binding and must necessarily be supplemented by legal rules.⁹⁹ However, principles of AI ethics may be effective when operationalised to apply to specific contexts and when used in conjunction with other forms of regulation. The principles provide the preconditions for responsible use of AI and automated decision tools by firms. They also provide an indication of what regulators should demand from firms deploying such technology to reduce the risk of harm to consumers.¹⁰⁰ While there are various formulations of the principles of ethical AI,¹⁰¹ key features typically include requirements for AI to be transparent and explainable,¹⁰² along with mechanisms for ensuring accountability¹⁰³ and – at least in the Australian government’s principles¹⁰⁴ – contesting adverse outcomes.¹⁰⁵

2.4.1 Transparency and Explanations

Principles of ethical AI typically require the use of such technologies to be transparent.¹⁰⁶ A starting place for transparency is to inform consumers when AI is being used in an interaction with them. Applied to automated financial advice tools, transparency must mean more than informing consumers that AI is being used to provide advice. Consumers choosing to turn to a robo-adviser or budgeting app will usually be aware of the automated character of the advice. Consumers also require transparency in the kind of technology being used to provide that advice: i.e. is it based in machine learning or a hand coded expert system. Additionally, a principle of transparency would require firms to inform consumers clearly about the scope of the service that is being provided, including the limitations of the technology in terms of personalised or expert advice.¹⁰⁷ If the advice provided is generalised to broadly defined categories of consumers, then this should be made clear, to counter consumers’ expectations of a unique and personal experience.

To the extent that consumers overestimate the capacities of fintech, transparency in way the advice is produced is important to ground expectations and allow scrutiny of the veracity of claims made about it. For regulators, transparency is key to overseeing the performance of the tools. Transparency is key to allowing bias or distortions in the scope of advice to be identified, scrutinised and, in some instances, rectified. Regulation can support the imperative for firms to take these ethical demands seriously, including by treating them as necessary elements of statutory obligations of suitability or best interests, and essential to ensuring that claims about the operation of the product are not misleading. For example, the process of automation, and its claims to objectivity and consistency, may make consumers overconfident about the advice and more likely to act on it.¹⁰⁸ This might suggest an obligation on firms to be scrupulously clear on the limits of what is able to provided by automated advice tools, and of the insights that can be derived from the technology being utilised.¹⁰⁹

Transparency in ethical AI is closely associated with initiatives in AI ‘explanations’ or ‘explainability’.¹¹⁰ Explanations in this sense do not lie in the details of the code. Rather, explainable AI considers the kind and degree of information that should be provided in assisting the various stakeholders in the decision or recommendation process to understand why decisions were taken or the factors that were significant in reaching a recommendation.¹¹¹ Explainable AI aims to provide greater transparency into the basis for automated decisions, predictions, and recommendations.¹¹² There are different ways in which explanations may be provided, and indeed the field of study in computer science is still developing.¹¹³ Possibilities include the use of counterfactuals, feature disclosure scores, weightings of influential factors, or a preference for simpler models where high levels of accuracy are not as imperative.¹¹⁴ Overall, however, a requirement for explanations would assist in scrutinising the basis of the recommendations produced through automated financial advice tools.

For lawyers, suggesting that a core element in the regulation of automated financial advice tools should focus on requirements related to transparency/explanations may seem a surprising aspiration.¹¹⁵ Disclosure as a consumer protection strategy has increasingly fallen out of favour, particularly in the regulation of financial services and credit. The insights into decision-making from behavioural psychology have shown that mere information disclosure does not lead to better decisions by consumers. Consumers are subject to bounded rationality which means they rely on rules of thumb, heuristics, and behavioural bias rather than information.¹¹⁶ In this light, it may be thought that any demand for greater transparency in automated financial advice tools may be of marginal utility. However, in a consumer protection context, consumers’ interests are substantially protected by regulators, and therefore transparency and explanations are relevant to both consumers seeking to protect their interests, and regulators charged with overseeing the market. Explanations should be provided in a form that is meaningful to the recipient.¹¹⁷ This means that the detail and technicality of the information provided may need to differ between consumers and regulators.¹¹⁸ In other words, the requirements should be scaled according to who is receiving the explanation.

2.4.2 Accountability

Principles of AI ethics typically require mechanisms for ensuring firms are accountable for the operation of the technologies.¹¹⁹ To have impact, accountability will require more than allocating responsibility for supervising the AI to a person. There is little worth in having a ‘human in the loop’ in circumstances where the design of the AI or automated tool means it is difficult for that person genuinely to oversee, interrogate or control the tool.¹²⁰ Accountability for automated financial advice tools should therefore require a firm to implement systematic processes for reviewing the operations and performance of the tools.¹²¹ A commitment to accountability may therefore require firms to have processes for scrutinising the data on which the AI is trained, its ongoing use, and its outputs.¹²² A model for the kind of robust approach required might be found in the audits increasingly recommended for AI used in public sector decision-making.¹²³ Such processes should aim to ensure the veracity of the tools and are a critical element in addressing and redressing concerns about bias, equity, and inclusion.¹²⁴

2.4.3 Contestability

There is little utility in requiring transparency and accountability in AI systems if there is no mechanism available to those affected by an AI or automated decision for acting to challenge an outcome that is erroneous, discriminatory, or otherwise flawed. Some formulations of AI ethical principles respond to this issue by requiring processes for contesting adverse outcomes.¹²⁵ While accountability processes should aim to be proactive in preventing these kinds of problems, contestability is a mechanism for individuals, advocates, or regulators to respond to harms that do occur.

Lyons et al. make the point that little is currently known about ‘what contestability in relation to algorithmic decisions entails, and whether the same processes used to contest human decisions … are suitable for algorithmic decision-making’.¹²⁶ Contestability for automated decisions may not be able simply to follow existing mechanisms for dealing with individual complaints or concerns. The models informing AI may be complex and opaque, thus creating challenges for review by subject domain experts who may nonetheless be unfamiliar with the technology. Additionally, scale creates a challenge. This is because one of the benefits of automated decision-making is that it can operate on a scale that is not possible for human decision-makers or advisers, and yet this makes processes for individual review potentially unmanageable.

The inquiry into what contestability requires may be different in the context of automated financial advice tools, as opposed to public sector use of automated decision-making. Consumers using automated advice tools will not be challenging a decision made about their rights to access public resources or benefits. Rather they will be challenging the advice given to them, the consistency of this advice with any representations about the tool, or compliance with any applicable regulatory regimes. Nonetheless, complexity and scale remain significant challenges. It is possible that the field of consumer protection law may have insights given its focus on both legal rights and structural mechanisms for protecting consumers’ interests in circumstances where there are considerable imbalances in power, resources, and information, which in some ways mirrors concerns around AI contestability. For example, in this context of automated financial advice tools, contestability for poor outcomes may come through the oversight provided by ombudsmen and regulators, rather than traditional litigation. These inquiries have the capacity to look at systemic errors, thus bringing expertise and capacity to review processes through which advice or recommendations are provided, rather than necessarily reopening every decision.

2.5 Conclusion

The triad of money, power, and AI collide in fintech innovation, which sees public and private sector support for using AI, along with blockchain and big data, in the delivery of financial services. Currently, the most prominent forms of fintech available to consumers are automated advice tools for investing and budgeting. These tools offer advantages of low cost, convenient and consistent advice on matters consumers often find difficult. Without discounting these attractions, we have argued that the oft-stated aspiration of automated advice financial tools in democratising personal finance should not distract attention from their potential to provide only a marginally useful service, while extracting consumer data and perpetuating the exclusion of some consumer cohorts from adequate access to credit, advice and banking. From this perspective, consumer-facing fintech provides a exemplary example of the need for careful regulatory attention being provided to the use of AI and related technologies even in seemingly low-risk contexts. Fintech tools that hold out to consumers a promise of expertise and assistance should genuinely be fit for the purpose. Consumers are unlikely to be able to monitor this quality themselves. As such, robust standards of transparency, accountability, and contestability that facilitate good governance and allow adequate regulatory oversight are crucial, even for these modest applications of AI.

3 Leveraging AI to Mitigate Money Laundering Risks in the Banking System

Doron Goldbarsht

^*

3.1 Introduction

Money laundering involves the transfer of illegally obtained money through legitimate channels so that its original source cannot be traced.¹ The United Nations estimates that the amount of money laundered each year represents 2–5 per cent of global gross domestic product (GDP); however, due to the surreptitious nature of money laundering, the total could be much higher.² Money launderers conceal the source, possession, or use of funds through a range of methods of varying sophistication, often involving multiple individuals or institutions across several jurisdictions to exploit gaps in the financial economy.³

As major facilitators in the global movement of money, banks carry a high level of responsibility for protecting the integrity of the financial system by preventing and obstructing illicit transactions. Many of the financial products and services they offer are specifically associated with money laundering risks. To ensure regulatory compliance in the fight against financial crime, banks must develop artificial intelligence (AI) about emerging money-laundering processes and create systems that effectively target suspicious behaviour.⁴

‘Smart’ regulation in the financial industry requires the development and deployment of new strategies and methodologies. Technology can assist regulators, supervisors, and regulated entities by alleviating the existing challenges of anti-money laundering (AML) initiatives. In particular, the use of AI can speed up risk identification and enhance the monitoring of suspicious activity by acquiring, processing, and analysing data rapidly, efficiently, and cost-effectively. It thus has the potential to facilitate improved compliance with domestic AML legal regimes. While the full implications of emerging technologies remain largely unknown, banks would be well advised to evaluate the capabilities, risks, and limitations of AI – as well as the associated ethical considerations.

This chapter will evaluate compliance with the Financial Action Task Force (FATF) global standards for AML,⁵ noting that banks continue to be sanctioned for non-compliance with AML standards. The chapter will then discuss the concept of AI, which can be leveraged by banks to identify, assess, monitor, and manage money laundering risks.⁶ Next, the chapter will examine the deficiencies in the traditional rule-based systems and the FATF’s move to a more risk-oriented approach, which allows banks to concentrate their resources where the risks are particularly high.⁷ Following this, the chapter will consider the potential for AI to enhance the efficiency and effectiveness of AML systems used by banks, as well as the challenges posed by its introduction. Finally, the chapter will offer some concluding thoughts.

3.2 Enforcement and Detection: The Cost of Non-compliance

The FATF sets global standards for AML, with more than 200 jurisdictions committed to implementing its recommendations.⁸ It monitors and assesses how well countries fulfil their commitment through legal, regulatory, and operational measures to combat money laundering (as well as terrorist financing and other related threats).⁹ Pursuant to the FATF recommendations, banks must employ customer due diligence (CDD) measures.¹⁰ CDD involves the identification and verification of customer identity through the use of other sources and data. Banks should conduct CDD for both new and existing business relationships.¹¹ They have a duty to monitor transactions and, where there are reasonable grounds to suspect criminal activity, report them to the relevant financial intelligence agency.¹² Banks must conduct their operations in ways that withstand the scrutiny of customers, shareholders, governments, and regulators. There are considerable consequences for falling short of AML standards.

In a 2021 report, AUSTRAC, Australia’s financial intelligence agency, assessed the nature and extent of the money laundering risk faced by Australia’s major banks as ‘high’. The report highlighted the consequences for customers, the Australian financial system, and the community at large.¹³ It drew attention to impacts on the banking sector – including financial losses, increased compliance costs, lower share prices, and increased risk of legal action from non-compliance – as well as reputational impacts on Australia’s international economic security.¹⁴

In this climate of heightened regulatory oversight, banks continue to be sanctioned for failing to maintain sufficient AML controls. In 2009, Credit Suisse Group was fined US$536 million for illegally removing material information, such as customer names and bank names, so that wire transfers would pass undetected through the filters at US banks. The violations were conducted on behalf of Credit Suisse customers in Iran, Sudan, and other sanctioned countries, allowing them to move hundreds of millions of dollars through the US financial system.¹⁵ Also in 2009, Lloyds Banking Group was fined US$350 million after it deliberately falsified customer information in payment records, ‘repairing’ transfers so that they would not be detected by US banks.¹⁶ In 2012, US authorities fined HSBC US$1.9 billion in a money laundering settlement.¹⁷ That same year, the ING Bank group was fined US$619 million for allowing money launderers to illegally move billions of dollars through the US banking system.¹⁸ The Commonwealth Bank of Australia was fined A$700 million in 2017 after it failed to comply with AML monitoring requirements and failed to report suspicious matters worth tens of millions of dollars.¹⁹ Even after becoming aware of suspected money laundering, the bank failed to meet its CDD obligations while continuing to conduct business with suspicious customers.²⁰ In 2019, fifty-eight AML-related fines were issued worldwide, totalling US$8.14 billion – more than double the amount for the previous year.²¹ Westpac Bank recently agreed to pay A$1.3 billion fine – an Australian record – for violating the Anti-Money Laundering and Counter-Terrorism Financing Act 2006. Westpac had failed to properly report almost 20 million international fund transfers, amounting to over A$11 billion, to AUSTRAC, thereby exposing Australia’s financial system to criminal misuse.²² In 2020, Citigroup agreed to pay US$400 million fine after engaging in what US regulators called ‘unsafe and unsound banking practices’, including with regard to money laundering.²³ The bank had previously agreed to a US$97.4 million settlement after ‘failing to safeguard its systems from being infiltrated by drug money and other illicit funds’.²⁴ The severity of these fines reflects the fact that non-compliance with AML measures in the banking industry is unacceptable to regulators.²⁵ More recently, AUSTRAC accepted an enforceable undertaking from National Australia Bank to improve the bank’s systems, controls, and record keeping so that they are compliant with AML laws.²⁶

The pressure on banks comes not only from increased regulatory requirements, but also from a marketplace that is increasingly concerned with financial integrity and reputational risks.²⁷ A bank’s failure to maintain adequate systems may have consequences for its share price and its customer base. Citigroup, for example, was fined in 2004 for failing to detect and investigate suspicious transactions. The bank admitted to regulators that it had ‘failed to establish a culture that ensured ongoing compliance with laws and regulations’. Within one week of the announcement by regulators, the value of Citigroup shares had declined by 2.75 per cent.²⁸

It is, therefore, in the best interests of the banks themselves to manage risks effectively and to ensure full compliance with the domestic legislation that implements the FATF recommendations, including by retaining senior compliance staff.²⁹ Despite the high costs involved, banks have largely expressed a strong commitment to improving their risk management systems to protect their own integrity and that of the financial system – as well as to avoid heavy penalties, such as those detailed above.³⁰ Yet, while banks continue to invest in their capabilities in this area, they also continue to attract fines. This suggests that the current systems are inadequate for combating financial crime.

The current systems rely on models that are largely speculative and rapidly outdated.³¹ Fraud patterns change constantly to keep up with technological advancements, making it difficult to distinguish between money laundering and legitimate transactions.³² But while emerging technologies can be exploited for criminal activity, they also have the potential to thwart it.³³ AI has proven effective in improving operational efficiency and predictive accuracy in a range of fields, while also reducing operational costs.³⁴ Already, some banks have begun using AI to automate data in order to detect suspicious transactions. Indeed, AI could revolutionise the banking industry, including by improving the banking experience in multiple ways.³⁵

3.3 Leveraging AI for AML

AI simulates human thought processes through a set of theories and computerised algorithms that execute activities that would normally require human intellect.³⁶ It is, in short, the ability of a computer to mimic the capabilities of the human mind. The technology uses predictive analytics through pattern recognition with differing degrees of autonomy. Machine learning is one of the most effective forms of AI for AML purposes.³⁷ It can use computational techniques to gain insights from data, recognise patterns, and create algorithms to execute tasks – all without explicit programming.³⁸ Standard programming, in contrast, operates by specific rules that are developed to make inferences and produce outcomes based on input data.³⁹ Machine learning initiatives allow AML systems to conduct risk assessments with varying levels of independence from human intervention.⁴⁰ Deep learning, for example, is a form of machine learning that builds an artificial neural network by conducting repeated tasks, allowing it to improve the outcome continuously and solve complex problems by adapting to environmental changes.⁴¹ Although there are many machine learning techniques, AI has four main capabilities for AML purposes: anomaly detection, suspicious behaviour monitoring, cognitive capabilities, and automatic robotic processing.⁴² The effectiveness of these capabilities depends largely on processing power, the variability of data, and the quality of data, thus requiring some degree of human expertise.

The processes involved in AI can be broadly grouped into supervised and unsupervised techniques. Supervised techniques use algorithms to learn from a training set of data, allowing new data to be classified into different categories. Unsupervised techniques, which often operate without training data, use algorithms to separate data into clusters that hold unique characteristics. Researchers maintain that algorithmic processes have the potential to detect money laundering by classifying financial transactions at a larger scale than is currently possible – and with greater accuracy and improved cost-efficiency.⁴³

3.4 The Shift to a Risk-Based Approach

One of the most significant obstacles for banks seeking to meet their compliance obligations is the difficulty of appropriately detecting, analysing, and mitigating money laundering risks – particularly during CDD and when monitoring transactions.⁴⁴ Currently, transaction monitoring and filtering technology is primarily rule-based, meaning that it is relatively simplistic and predominantly focused on automated and predetermined risk factors.⁴⁵ The system operates as a ‘decision tree’, in which identified outliers generate alerts that require investigation by other parties. Thus, when a suspicious activity is flagged, a compliance officer must investigate the alert and, if appropriate, generate a suspicious matter report.⁴⁶

In order to minimise the costs and time required to investigate suspicious transactions, it is essential to detect them accurately at the first instance.⁴⁷ In rule-based systems, the task is made all the more difficult by the high false positive rate of the alerts, which is believed to be above 98 per cent.⁴⁸ If risk assessment in low-risk situations is overly strict, unmanageable numbers of false positive identifications can cause significant operational costs.⁴⁹ Conversely, if risk assessments are too lax, illicit transactions can slip through unnoticed.⁵⁰ These static reporting processes make it difficult to analyse increasingly large volumes of data, making them impractical on the scale required by banks. It has thus become necessary for banks to choose between the efficiency and the effectiveness of their AML processes.

Moreover, the rule-based systems rely on human-defined criteria and thresholds that are easy for money launderers to understand and circumvent. The changing patterns of fraud make it difficult for rule-based systems and policies to maintain their effectiveness, thus allowing money laundering transactions to be misidentified as genuine.⁵¹ AML systems are designed to detect unusual transaction patterns, rather than actual criminal behaviour. Rule-based systems thus have the potential to implicate good customers, initiate criminal investigations against them, and thereby damage customer relationships – all without disrupting actual money laundering activities. This is because the systems were designed for a relatively slow-moving fraud environment in which patterns would eventually emerge and be identified and then incorporated into fraud detection systems. Today, criminal organisations are themselves leveraging evolving technologies to intrude into organisational systems and proceed undetected.⁵² For example, AI allows criminals to use online banking and other electronic payment methods to move illicit funds across borders through the production of bots and false identities that circumnavigate AML systems.⁵³

According to the FATF, implementing a risk-based approach is the ‘cornerstone of an effective AML/CFT system and is essential to properly managing risks’.⁵⁴ Yet many jurisdictions continue to use antiquated rule-based systems, leading to defensive compliance. To keep pace with modern crime and the increasing volume and velocity of data, banks need a faster and more agile approach to the detection of money laundering. They should reconsider their AML strategies and evolve from traditional rule-based systems to more sophisticated risk-based AI solutions. By leveraging AI, banks can take a proactive and preventive approach to fighting financial crime.⁵⁵

3.5 Advantages and Challenges

3.5.1 Advantages

New technologies are key to improving the management of regulatory risks. Banks have begun exploring the use of AI to assist analysts in what has traditionally been a manually intensive task to improve the performance of AML processes.⁵⁶ In 2018, US government agencies issued a joint statement encouraging banks to use innovative methods, including AI, to further efforts to protect the integrity of the financial system against illicit financial activity.⁵⁷ The United Kingdom Financial Conduct Authority has supported a series of public workshops aimed at encouraging banks to experiment with novel technologies to improve the detection of financial crimes.⁵⁸ AUSTRAC has invested in data analysis and advanced analytics to assist in the investigation of suspicious activity.⁵⁹ Indeed, developments in AI offer an opportunity to fundamentally transform the operations of banks, equipping them to combat modern threats to the integrity of the financial system.⁶⁰ And, where AI reaches the same conclusions as traditional analytical models, this can confirm the accuracy of such assessments, ultimately increasing the safeguards available to supervisors.⁶¹ Although machine learning remains relatively underutilised in the area of AML, it offers the potential to greatly enhance the efficiency and effectiveness of existing systems.⁶²

3.5.1.1 Improved Efficiency

Incorporating AI in AML procedures can reduce the occurrence of false positives and increase the identification of true positives. In Singapore, the United Overseas Bank has already piloted machine learning to enhance its AML surveillance by implementing an AML ‘suite’ that includes know-your-customer (KYC), transaction monitoring, name screening, and payment screening processes.⁶³ The suite provides an additional layer of scrutiny that leverages machine learning models over traditional rule-based monitoring systems, resulting in real benefits. In relation to transaction monitoring, the recognition of unknown suspicious patterns saw an increase of 5 per cent in true positives and a decrease of 40 per cent in false positives. There was a more than 50 per cent reduction in false positive findings in relation to name screening.⁶⁴

AI has the capability to analyse vast volumes of data, drawing on an increased number of variables. This means that the quality of the analysis is enhanced and the results obtained are more precise.⁶⁵ At the same time, utilising AI in AML can increase productivity by reducing staff work time by 30 per cent.⁶⁶ By combining transactional data with other information, such as customer profile data, it is possible to investigate AML risks within days. In contrast, traditional methods that review isolated accounts often require months of analysis. Additionally, banks can use AI to facilitate the live monitoring of AML standards, which can also improve governance, auditability, and accountability.⁶⁷ Overall, the use of machine learning has resulted in a 40 per cent increase in operational efficiency, reinforcing the notion that investment in AI initiatives may have positive implications for the reliability of AML processes.⁶⁸

3.5.1.2 Reduced Compliance Costs

By leveraging AI, banks have an opportunity to reduce costs and prioritise human resources in complex areas of AML.⁶⁹ It has been estimated that incorporating AI in AML compliance procedures could save the global banking industry more than US$1 trillion by 2030⁷⁰ and reduce its costs by 22 per cent over the next twelve years.⁷¹ The opportunities for cost reduction and improved productivity and risk management offer convincing incentives for banks to engage AI and machine learning to achieve greater profitability.⁷² With increased profits, banks could further improve the accuracy of AML systems and, in the process, advance the goals of AML.⁷³

3.5.1.3 Increased Inclusiveness

Digital tools have the potential to increase financial inclusion, promoting more equitable access to the formal financial sector.⁷⁴ Customers with less reliable forms of identification – including First Nations peoples and refugees – can access banking services through solutions such as behavioural analytics, which reduces the burden of verification to one instance of customer onboarding. Utilising AI makes banks less reliant on traditional CDD, offering enhanced monitoring capabilities that can be used to manage verification data.⁷⁵

3.5.2 Challenges

Despite the growing recognition of the potential for AI to improve the accuracy, speed, and cost-effectiveness of AML processes, banks remain slow to adopt these technologies due to the regulatory and operational challenges involved.⁷⁶ Significant hurdles to wider adoption persist and these may continue to stifle innovations in AML compliance.

3.5.2.1 Interpretation

The difficulty of interpreting and explaining the outcomes derived from AI technologies is among the main barriers to securing increased support for these tools.⁷⁷ The Basel Committee on Banking Supervision has stated that, in order to replicate models, organisations should be able to demonstrate developmental evidence of theoretical construction, behavioural characteristics, and key assumptions; the types and use of input data; specified mathematical calculations; and code-writing language and protocols.⁷⁸ Yet artificial neural networks may comprise hundreds of millions of connections, each contributing in some small way to the outcomes produced.⁷⁹ Indeed, as technological models become increasingly complex, the inner workings of the algorithms become more obscure and difficult to decode, creating ‘black boxes’ in decision-making.⁸⁰

In the European Union, the increased volume of data processing led to the adoption of the General Data Protection Regulation (GDPR) in 2016.⁸¹ The GDPR aims to ensure that the data of individuals is protected – particularly in relation to AML procedures, which often collect highly personal data.⁸² With respect to AI and machine learning, Recital 71 specifies that there is a right to obtain an explanation of the decision reached after algorithmic assessment. Because regulated entities remain responsible for the technical details of AI solutions, fears persist concerning accountability and interpretability where technologies cannot offer robust transparency.⁸³ While the GDPR expects that internal compliance teams will understand and defend the algorithms utilised by digital tools, compliance officers working in banks require expertise and resources to do so. It may take a long period of time for even the most technologically literate of supervisors to adjust to new regulatory practices.⁸⁴ Efforts to improve the interpretation of AI and machine learning are vital if banks are to enhance risk management and earn the trust of supervisors, regulators, and the public.

3.5.2.2 Data Quality

The data utilised to train and manage AI systems must be of high quality.⁸⁵ Machine learning models are not self-operating; they require human intervention to ensure their optimal functioning.⁸⁶ In other words, machines cannot think for themselves. Rather, they merely execute and learn from their encoded programming.⁸⁷ Since machine learning is only as good as its input, it is crucial that the models used are based on relevant and diverse data.⁸⁸ Where money-laundering transactions have not previously been identified by the system, it may be difficult for machine learning to detect future instances.⁸⁹ Moreover, false positives would be learned into the system if the training data included them.⁹⁰ Therefore, it is essential that data quality is monitored on an ongoing basis to ensure thorough data analysis and regular data cleansing. This serves to highlight the vital importance of vigilant human collaboration in the technological implementation of AI to ensure that models are well maintained and remain effective.⁹¹

3.5.2.3 Collaboration

The inexplicable nature of AI, especially machine learning processes, has sparked concerns that are exacerbated by the lack of data harmonisation between actors and users.⁹² Currently, customer privacy rules and information security considerations prevent banks from warning each other about potentially suspicious activity involving their customers. While some customers rely on a single financial services provider for all their banking requirements, criminals often avoid detection by moving illicit proceeds through numerous financial intermediaries.⁹³ The FATF has reported that intricate schemes involving complex transaction patterns are difficult and sometimes impossible to detect without information from counterparty banks or other banks providing services to the same customer.⁹⁴ Nevertheless, the FATF’s rules to prevent ‘tipping off’ support the objective of protecting the confidentiality of criminal investigations.⁹⁵

While data standardisation and integrated reporting strategies simplify regulatory reporting processes, they also raise various legal, practical, and competition issues.⁹⁶ It is likely that the capacity of banks to model will continue to be limited by the financial transactions that they themselves process.⁹⁷ Moreover, where information is unavailable across multiple entities, some technological tools may not be cost-effective.⁹⁸ On the other hand, stronger collaboration may introduce the risk of data being exploited on a large scale.⁹⁹ There is as yet no ‘model template’ in relation to private sector information sharing that complies with AML and data protection and privacy requirements. However, information sharing initiatives are being explored and should be considered in targeted AI policy developments.

3.5.2.4 Privacy

Due to the interconnectedness of banks and third party service providers, cyber risks are heightened when tools such as AI and machine learning are used and stored in cloud platforms. Concentrating digital solutions might exacerbate these risks.¹⁰⁰ These regulatory challenges reinforce the desire to maintain human-based supervisory processes so that digital tools are not replacements but rather aids in the enhancement of regulatory systems.¹⁰¹ Article 22 of the GDPR provides that subjects of data analysis have the right not to be subject to a decision with legal or significant consequences ‘based solely on automated processing’.¹⁰² The FATF also maintains that the adoption of AI technology in AML procedures requires human collaboration, due to particular concerns that technology is incapable of identifying emerging issues such as regional inequalities.¹⁰³

3.5.2.5 Bias

Although algorithmic decision-making may appear to offer an objective alternative to human subjectivity, many AI algorithms replicate the conscious and unconscious biases of their programmers.¹⁰⁴ This may lead to unfairly targeting the financial activities of certain individuals or entities, or it may produce risk profiles that deny certain persons access to financial services. For example, AI and machine learning are increasingly being used in relation to KYC models.¹⁰⁵ Recommendation 10 of the FATF standards requires banks to monitor both new and existing customers to ensure that their transactions are legitimate.¹⁰⁶ Without the incorporation of AI, existing KYC processes are typically costly and labour-intensive.¹⁰⁷ Utilising AI can help evaluate the legitimacy of customer documentation and calculate the risks for banks where applications may seem to be fake.¹⁰⁸ The data input team should ensure that it does not unintentionally encode systemic bias into the models by using attributes such as employment status or net worth.¹⁰⁹ Transactional monitoring is less vulnerable to such biases, as it does not involve personal data such as gender, race, and religion. Nonetheless, AI and machine learning algorithms could implicitly correlate those indicators based on characteristics such as geographical location.¹¹⁰ If not implemented responsibly, AI has the potential to exacerbate the financial exclusion of certain populations for cultural, political, or other reasons.¹¹¹ The use of these digital tools may thus lead to unintended discrimination.¹¹² Such concerns are heightened by the fact that the correlations are neither explicit nor transparent.¹¹³ Therefore, regulators must remain mindful of the need to limit bias, ensure fairness, and maintain controls. The evolving field of discrimination-aware data mining may assist the decision-making processes that flow through information technology to ensure that they are not affected on unjust or illegitimate grounds.¹¹⁴ It does this by recognising statistical imbalances in data sets and leveraging background information about discrimination-indexed features to identify ‘bad’ patterns that can then be either flagged or filtered out entirely.¹¹⁵

3.5.2.6 Big Data

The term ‘big data’ refers to large, complex, and ever-changing data sets and the technological techniques that are relevant to their analysis.¹¹⁶ Policymakers and technical organisations have expressed significant concerns over the potential misuse of data.¹¹⁷ There are also apprehensions that the lack of clarity around how data is handled may lead to potential violations of privacy.¹¹⁸ In addition, there are uncertainties surrounding the ownership of data, as well as its cross-border flow.¹¹⁹ Nonetheless, the primary focus should remain on the use of big data, rather than its collection and storage, as issues pertaining to use have the potential to cause the most egregious harm.¹²⁰

3.5.2.7 Liability

The issues discussed above raise questions of liability regarding who will carry the burden of any systemic faults that result in the loss or corruption of data and related breaches of human rights.¹²¹ While artificial agents are not human, they are not without responsibility.¹²² Because it is impossible to punish machines, questions of liability are left to be determined between system operators and system providers.¹²³ This situation can be likened to a traffic accident in which an employee injures a pedestrian while driving the company truck. While the employer and the employee may both be liable for the injuries, the truck is not.¹²⁴ These issues enliven questions of causation. Will the use of AI and machine learning be considered a novus actus interveniens that breaks the chain of causation and prevents liability from being attributed to other actors?¹²⁵ The answer to this question will largely depend on the characteristics of artificial agents and whether they will be considered as mere tools or as agents in themselves, subject to liability for certain data breaches or losses. Despite the impact of automation processes on decision-making, doubts remain as to whether AI uses ‘mental processes of deliberation’.¹²⁶ Due to the collaborative nature of AI technology and human actors, it is generally assumed that AI is merely an instrument and that accountability will be transferred to banks and developers.¹²⁷ Therefore, where supervisors can be considered legal agents for the operation of artificial technology, they may incur liability on the basis of that agency relationship.¹²⁸ Alternatively, where system developers are negligent as far as security vulnerabilities are concerned, they may be liable for the harm caused by unauthorised users or cyber criminals who exploit these deficiencies.¹²⁹ Thus, supervisors and developers have a duty of care to ensure that they take reasonable steps to prevent harm or damage.¹³⁰ It is possible that, as a result of its continued advancement, machine learning may eventually be granted legal personhood. Rights and obligations would therefore belong to the technology itself, excusing operators and developers from liability.¹³¹ However, this viewpoint remains highly contested on the basis that AI does not possess ‘free will’, since it is programmed by humans and has little volition of its own.¹³² Banks must not underestimate the importance of these concerns. They should ensure that AI and machine learning are carefully implemented with well-designed governance in place so that risks and liabilities are not unintentionally heightened by the use of new technologies.¹³³ Strong checks and balances are required at all stages of the development process.¹³⁴

3.5.2.8 Costs

Banks must consider the costs of maintaining, repairing, and adapting new AI systems.¹³⁵ While AI models have the potential to improve the cost-efficiency of AML compliance, it may be difficult for banks – especially smaller institutions – to budget for high-level AI solutions.¹³⁶ Moreover, there are associated indirect costs that require firms to invest in additional funding – for example, updating existing database systems to make them compatible with new AI solutions and hiring staff with appropriate technical expertise.¹³⁷

3.5.3 Consideration

AI and machine learning have the potential to provide banks with effective tools to improve risk management and compliance with regard to AML. However, if these new technologies are not introduced with care and diligence, they could adversely affect AML systems by introducing greater burdens and risks. Some of the challenges presented by AI are similar to those posed by other technology-based solutions aimed at identifying and preventing money laundering. Machine learning, however, offers a relatively new and unique method of classifying information based on a feedback loop that enables the technology to ‘learn’ through determinations of probability.¹³⁸ Banks can thus analyse and classify information through learned anomaly detection algorithms, a technique that is more effective than traditionally programmed rule-based systems.¹³⁹ At the same time, the utilisation of AI can exacerbate the complexity and severity of the challenges inherent in AML compliance, particularly in relation to interpretation and explanation.¹⁴⁰ As discussed above, machine learning algorithms usually do not provide a rationale or reasoning for the outcomes they produce, making it difficult for compliance experts to validate the results and deliver clear reports to regulators.¹⁴¹ This is particularly concerning for banks, where trust, transparency, and verifiability are of great importance to ensure satisfaction and regulatory confidence.¹⁴² Nonetheless, in the current regulatory climate, it seems almost inevitable that banks will continue to leverage AI for AML compliance.

3.6 Conclusion

The traditional framework for AML compliance is largely premised on old banking models that do not adequately keep pace with the modern evolution of financial crime. Traditional rule-based monitoring systems are clearly inadequate to detect the increasingly sophisticated methods and technologically advanced strategies employed by criminals. Banks are burdened with false positives while most money laundering transactions remain unidentified, posing a significant threat to the integrity of banks and the financial system itself. Banks that do not meet their compliance obligations expose themselves to significant pecuniary losses and reputational damage.¹⁴³

The FATF has highlighted the potential of innovative technologies such as AI and machine learning to make AML measures faster, cheaper, and more effective than current monitoring processes. While rule-based algorithms remain relevant, harnessing AI and machine learning holds great promise for increasing the accuracy of risk identification and heightening its efficiency due to the large analytical capacity of these processes. While these initiatives may be costly and risky to implement, they offer an excellent return on investment for banks that seek to strengthen their internal AML regime. The implementation of AI is increasingly recognised as the next phase in the fight against financial crime.

Due to the various regulatory and operational challenges that are likely to arise, banks should approach the adoption and implementation of AI with cautious optimism. They should ensure that sophisticated AI and machine learning models can be adequately understood and explained. To achieve optimal outcomes, these technologies should operate in conjunction with human analysis, particularly in areas of high risk. However, banks should be aware that the emphasis on collaboration between analysts, investigators, and compliance officers with regard to AI technology may introduce its own legal and ethical complications relating to privacy, liability, and various unintended consequences, such as customer discrimination.

In the increasingly complex environment of financial crime and AML regulation, banks should thoroughly consider the advantages and challenges presented by AI and machine learning as they move towards the transformation of risk assessment by leveraging AI to mitigate money laundering risks.

4 AI Opacity in the Financial Industry and How to Break It

Zofia Bednarz and Linda Przhedetsky ^*

4.1 Introduction

Automated Banks – the financial entities using ADM and AI – feed off the culture of secrecy that is pervasive and entrenched in automated processes across sectors from ‘Big Tech’ to finance to government agencies, allowing them to avoid scrutiny, accountability, and liability.¹ As Pasquale points out, ‘finance industries profit by keeping us in the dark’.²

An integral part of the financial industry’s business model is the use of risk scoring to profile consumers of financial services, for example in the form of credit scoring, which is a notoriously opaque process.³ The use of non-transparent, almost ‘invisible’ surveillance processes and the harvesting of people’s data is not new: financial firms have always been concerned with collecting, aggregating, and combining data for the purposes of predicting the value of their customers through risk scoring.⁴ Automation⁵ introduces a new level of opacity in the financial industry, for example through the creation of AI models for which explanations are not provided – either deliberately, or due to technical explainability challenges.⁶

In this chapter we argue that the rise of AI and ADM tools contributes to opacity within the financial services sector, including through the intentional use of the legal system as a ‘shield’ to prevent scrutiny and blur accountability for harms suffered by consumers of financial services. A wealth of literature critiques the status quo, showing that consumers are disadvantaged by information asymmetries,⁷ complicated consent agreements,⁸ information overload,⁹ and other tactics that leave consumers clueless if, when, and how they have been subject to automated systems. If consumers seek to access a product or service, it is often a requirement that they be analysed and assessed using an automated tool, for example, one that determines a credit score.¹⁰ The potential harms are interlinked and range from financial exclusion to digital manipulation to targeting of vulnerable consumers and privacy invasions.¹¹ In our analysis we are mostly concerned with discrimination as an example of such harm,¹² as it provides a useful illustration of problems enabled by opacity, such as significant difficulty in determining if unfair discrimination has occurred at all, understanding the reasons for the decision affecting the person or group, and accessing redress.

The rules we examine will differ among jurisdictions, and our aim is not to provide a comprehensive comparative analysis of all laws that provide potential protections against scrutiny and increase the opacity of ADM-related processes of Automated Banks. We are interested in exploring certain overarching tendencies, using examples from various legal systems, and showing how financial firms may take advantage of the complex legal and regulatory frameworks applicable to their operations in relation to the use of AI and ADM tools.

As the use of AI and ADM continues to grow in financial services markets, consumers are faced with the additional challenge of knowing about, and considering how their ever-expanding digital footprint may be used by financial institutions. The more data exists about a person, the better their credit score (of course within certain limits, such as paying off debts on time).¹³ The exact same mechanism may underpin ‘open banking’ schemes: consumers who do not have sufficient data – often vulnerable people, such as domestic violence victims, new immigrants, or Indigenous people – cannot share their data with financial entities, may be excluded from accessing some products or offered higher prices, even if their actual risk is low.¹⁴

In Australia, consumers have claimed that they have been denied loans due to their use of takeaway food services and digital media subscriptions.¹⁵ Credit rating agencies such as Experian explicitly state that they access data sources that reflect consumers’ use of new financial products, including ‘Buy Now Pay Later’ schemes.¹⁶ As more advanced data collection, analysis, and manipulation technologies continue to be developed, there is potential for new categories of data to emerge. Already, companies can draw surprising inferences from big data. For example, studies have shown that seemingly trivial Facebook data can, with reasonable accuracy, predict a range of attributes that have not been disclosed by users: in one study, liking the ‘Hello Kitty’ page correlated strongly with a user having ‘[d]emocratic political views and to be of African-American origin, predominantly Christian, and slightly below average age’.¹⁷

Unless deliberate efforts are made, both in the selection of data sets and the design and auditing of AMD tools, inferences and proxy data will continue to produce correlations that may result in discriminatory treatment.¹⁸

This chapter proceeds as follows. We begin Section 4.2 with discussion of rules that allow corporate secrecy around AI models and their data sources to exist, focusing on three examples of such rules. We discuss the opacity of credit scoring processes and the limited explanations that consumers can expect in relation to a financial decision made about them (Section 4.2.1), trade secrecy laws (Section 4.2.2), and data protection rules which do not protect de-identified or anonymised information (Section 4.2.3). In Section 4.3 we analyse frameworks that incentivise the use of ADM tools by the financial industry, thus providing another ‘protective layer’ for Automated Banks, again discussing two examples: financial product governance regimes (Section 4.3.1) and ‘open banking’ rules (Section 4.3.2). The focus of Section 4.4 is on potential solutions. We argue it is not possible for corporate secrecy and consumer rights to coexist, and provide an overview of potential regulatory interventions, focusing on preventing Automated Banks from using harmful AI systems (Section 4.4.1), aiding consumers understand when ADM is used (Section 4.4.2), and facilitating regulator monitoring and enforcement (Section 4.4.3). The chapter concludes with Section 4.5.

4.2 Rules That Allow Corporate Secrecy to Exist

4.2.1 Opacity of Credit Scoring and the (Lack of) Explanation of Financial Decisions

Despite their widespread use in the financial industry, credit scores are difficult for consumers to understand or interpret. A person’s credit risk has traditionally been calculated based on ‘three C’s’: collateral, capacity, and character.¹⁹ Due to the rise of AI and ADM tools in the financial industry, the ‘three C’s’ are increasingly being supplemented and replaced by diverse categories of data.²⁰ An interesting example can be found through FICO scores, which are arguably the first large-scale process in which automated computer models replaced human decision-making.²¹ FICO, one of the best-known credit scoring companies,²² explains that their scores are calculated according to five categories: ‘payment history (35%), amounts owed (30%), length of credit history (15%), new credit (10%), and credit mix (10%)’.²³ These percentage scores are determined by the company to give consumers an understanding of how different pieces of information are weighted in the calculation of a score, and the ratios identified within FICO scores will not necessarily reflect the weightings used by other scoring companies. Further, while FICO provides a degree of transparency, the ways in which a category such as ‘payment history’ is calculated remains opaque: consumers are not privy to what is considered a ‘good’ or a ‘bad’ behaviour, as represented by data points in their transaction records.²⁴

Globally, many credit scoring systems (both public and private) produce three-digit numbers within a specified range to determine a consumer’s creditworthiness. For example, privately operated Equifax and Trans Union Empirica score consumers in Canada between 300 and 900,²⁵ whereas credit bureaus in Brazil score consumers between 1 and 1,000.²⁶ In an Australian context, scores range between 0 and 1,000, or 1,200, depending on the credit reporting agency.²⁷ By contrast, other jurisdictions use letter-based ratings, such as Singapore’s HH to AA scale which corresponds with a score range of 1,000–2,000,²⁸ or blacklists, such as Sweden’s payment default records.²⁹

Credit scoring, it turns out, is surprisingly accurate in predicting financial breakdowns or future loan delinquency,³⁰ but the way different data points are combined by models is not something even the model designer can understand using just intuition.³¹ Automated scoring processes become even more complex as credit scoring companies increasingly rely on alternative data sources to assess consumers’ creditworthiness, including ‘predictions about a consumer’s friends, neighbors, and people with similar interests, income levels, and backgrounds’.³² And a person’s credit score is just one of the elements lenders, Automated Banks, feed into their models to determine a consumer’s risk score. It has been reported that college grades, and the time of day an individual applies for a loan have been used to determine a person’s access to credit.³³ These types of data constitute ‘extrinsic data’ sources, which consumers are unknowingly sharing.³⁴

The use of alternative data sources is purported as a way of expanding consumers’ access to credit in instances where there is a lack of quality data (such as previous loan repayment history) to support the underwriting of consumers’ loan.³⁵ Applicants are often faced with a ‘Catch-22 dilemma: to qualify for a loan, one must have a credit history, but to have a credit history one must have had loans’.³⁶ This shows how ADM tools offer more than just new means to analyse greater than ever quantities of data: they also offer a convenient excuse for Automated Banks to effectively use more data.

Of course, increasing reliance on automated risk scoring is not the origin of unlawful discrimination in financial contexts. However, it is certainly not eliminating discriminatory practices either: greater availability of more granular data, even when facially neutral, leads to reinforcing of existing inequalities.³⁷ Automated Banks have been also shown to use alternative data to target more vulnerable consumers, who they were not able to reach or identify when only using traditional data on existing customers.³⁸ The quality change that AI tools promise to bring is to ‘make the data talk’: all data is credit data, if we have the right automated tools to analyse them.³⁹

Collection, aggregation, and use of such high volumes of data, including ‘extrinsic data’, also make it more difficult, if not impossible, for consumers to challenge financial decisions affecting them. While laws relating to consumer lending (or consumer financial products in general) in most jurisdictions provide that some form of explanation of a financial decision needs to be made available to consumers,⁴⁰ these rules will rarely be useful in the context of ADM and AI tools used in processes such as risk scoring.

This is because AI tools operate on big data. Too many features of a person are potentially taken into account for any feedback to be meaningful. The fact that risk scores and lending decisions are personalised make it even more complicated for consumers to compare their offer with anyone else’s. This can be illustrated by the case of Apple credit card,⁴¹ which has shown the complexity of investigation necessary for people to be able to access potential redress: when applying for personalised financial products, consumers cannot immediately know what features are being taken into account by financial firms assessing their risk, and subsequent investigation by regulators or courts may be required.⁴² The lack of a right to meaningful explanation of credit scores and lending decisions based on the scores makes consumers facing Automated Banks and the automated credit scoring system quite literally powerless.⁴³

4.2.2 Trade Secrets and ADM Tools in Credit Scoring

The opacity of credit scoring, or risk scoring more generally, and other automated assessment of clients that Automated Banks engage in, is enabled by ADM tools which ‘are highly valuable, closely guarded intellectual property’.⁴⁴ Complementing the limited duty to provide explanation of financial decisions to consumers, trade secrets laws allow for even more effective shielding of the ADM tools from scrutiny, including regulators’ and researchers’ scrutiny.

While trade secrets rules differ between jurisdictions, the origin and general principles that underpin these rules are common across all the legal systems: trade secrets evolved as a mechanism to protect diverse pieces of commercial information, such as formulas, devices, or patterns from competitors.⁴⁵ These rules fill the gap where classic intellectual property law, such as copyright and patent law, fails – and it notably fails in relation to AI systems, since algorithms are specifically excluded from its protection.⁴⁶ Recent legal developments, for example the European Union Trade Secrets Directive,⁴⁷ or the US Supreme Court case of Alice Corp. v CLS Bank,⁴⁸ mean that to protect their proprietary technologies, companies are now turning to trade secrets.⁴⁹ In practice, this greatly reduces the transparency of the ADM tools used: if these cannot be protected through patent rights, they need to be kept secret.⁵⁰

The application of trade secrets rules leads to a situation in which financial entities, for example lenders or insurers, who apply third party automated tools to assess creditworthiness of their prospective clients might not be able to access the models and data they use. Using third party tools is a common practice, and the proprietary nature of the tools and data used to develop and train the models will mean financial entities using these tools may be forced to rely on the supplier’s specifications in relation to their fairness as they may not be able to access the code themselves.⁵¹

Secrecy of ADM tools of course has implications for end users, who will be prevented from challenging credit models, and is also a barrier for enforcement and research.⁵² Trade secret protections apply not only to risk scoring models, but often extend also to data sets and inferences generated from information provided by individuals.⁵³ Commercial entities openly admit they ‘invest significant amounts of time, money and resources’ to draw inferences about individuals ‘using […] proprietary data analysis tools’, a process ‘only made possible because of the [companies’] technical capabilities and value add’.⁵⁴ This, they argue, makes the data sets containing inferred information a company’s intellectual property.⁵⁵

The application of trade secrets rules to credit scoring in a way that affects the transparency of the financial system is not exactly new: ‘[t]he trade secrecy surrounding credit scoring risk models, and the misuse of the models coupled with the lack of governmental control concerning their use, contributed to a financial industry wide recession (2007–2008)’.⁵⁶

In addition to trade secrets laws, a sui generis protection of source code of algorithms is being introduced in international trade law through free trade agreements,⁵⁷ which limit governments from mandating access to the source code. The members of the World Trade Organization (WTO) are currently negotiating a new E-commerce trade agreement, which may potentially include a prohibition on government-mandated access to software source code.⁵⁸ WTO members, including Canada, the EU, Japan, South Korea, Singapore, Ukraine, and the United States support such a prohibition,⁵⁹ which in practice will mean a limited ability for states to adopt laws that would require independent audits of AI and ADM systems.⁶⁰ It is argued that adoption of the WTO trade agreement could thwart the adoption of the EU’s AI Act,⁶¹ demonstrating how free trade agreements can impose another layer of rules enhancing the opacity of AI and ADM tools.

4.2.3 ‘Depersonalising’ Information to Avoid Data and Privacy Protection Laws: Anonymisation, De-identification, and Inferences

Automated Banks’ opacity is enabled by the express exclusion of ‘anonymised’ or ‘de-identified’ data from the scope of data and privacy protection laws such as the GDPR.⁶² In its Recital 26, the GDPR defines anonymised information as not relating to ‘an identified or identifiable natural person’ or as ‘data rendered anonymous in such a manner that the data subject is not or no longer identifiable’. This allows firms to engage in various data practices, which purport to use anonymised data.⁶³ They argue they do not collect or process ‘personal information’, thus avoiding the application of the rules, and regulatory enforcement.⁶⁴ Also, consumers to whom privacy policies are addressed believe that practices focusing on information that does not directly identify them have no impact on their privacy.⁶⁵ This in turn may mean privacy policies are misrepresenting data practices to consumers, which could potentially invalidate their consent.⁶⁶

There is an inherent inconsistency between privacy and data protection rules and the uses and benefits that ADM tools using big data analytics promise. Principles of purpose limitation and data minimisation⁶⁷ require entities to delimit, quite strictly and in advance, how the data collected are going to be used, and prevent them from collecting and processing more data than necessary for that specific purpose. However, this is not how big data analytics, which fuels ADM and AI models, works.⁶⁸ Big data means that ‘all data is credit data’, incentivising the Automated Banks to collect as much data as possible, for any possible future purpose, potentially not known yet.⁶⁹ The exclusion of anonymised or de-identified data from the scope of the protection frameworks opens doors for firms to take advantage of enhanced analytics powered by new technologies. The contentious question is at which point information becomes, or ceases to be, personal information. If firms purchase, collect, and aggregate streams of data, producing inferences allowing them to describe someone in great detail, including their age, preferences, dislikes, size of clothes they wear and health issues they suffer from, their household size and income level,⁷⁰ but do not link this profile to the person’s name, email, physical address, or IP address – would it be personal information? Such a profile, it could be argued, represents a theoretical, ‘model’ person or consumer, built for commercial purposes through aggregation of demographic and other information available.⁷¹

De-identified data may still allow a financial firm to achieve more detailed segmentation and profiling of their clients. There are risks of harms in terms of ‘loss of privacy, equality, fairness and due process’ even when anonymised data is used.⁷² Consumers are left unprotected against profiling harms due to such ‘narrow interpretation of the right to privacy as the right to anonymity’.⁷³

There is also discussion as to the status of inferences under data and privacy protection laws. Credit scoring processes are often based on inferences, where a model predicts someone’s features (and ultimately their riskiness or value as a client) on the basis of other characteristics that they share with others deemed risky by the model.⁷⁴ AI models may thus penalise individuals for ‘shopping at low-end stores’, membership in particular communities or families, and affiliations with certain political, religious, and other groups.⁷⁵ While AI-powered predictions about people’s characteristics are often claimed to be more accurate than those made by humans,⁷⁶ they may also be inaccurate.⁷⁷ The question is if such inferences are considered personal information protected by privacy and data laws.

Entities using consumers’ data, such as technology companies, are resisting against expressly including inferred information in the scope of data and privacy protections. For example, Facebook openly admitted that ‘[t]o protect the investment made in generating inferred information and to protect the inferred information from inappropriate interference, inferred information should not be subject to all of the same aspects of the [Australian Privacy Act] as personal information’.⁷⁸ The ‘inappropriate interference’ they mention refers to extending data correction and erasure rights to inferred information.

Second, there is an inherent clash between the operation of privacy and data protection rules and the inference processes AI tools are capable of carrying out. Any information, including sensitive information, may be effectively used by an ADM system, even though it only materialises as an internal encoding of the model and is not recorded in a human understandable way. The lack of explicit inclusion of inferred information, and its use, within the privacy and data protection frameworks provides another layer of opacity shielding financial firms (as well as other entities) from scrutiny of their ADM tools.

When information is ‘depersonalised’ in some way: de-identified on purpose through the elimination of strictly personal identifiers,⁷⁹ through use of anonymous ‘demographic’ data, through ‘pseudonymisation’ practices, or because it is inferred from data held (either personal or already de-identified), the result is the same – privacy and data protection rules do not apply. The firms take advantage of that exclusion, sometimes balancing on the thin line between legal and illegal data processing, making their data practices non-transparent to avoid scrutiny by consumers and regulators.

As a US judge in a recent ruling put it: ‘[i]t is well established that there is an undeniable link between race and poverty, and any policy that discriminates based on credit worthiness correspondingly results in a disparate impact on communities of color’.⁸⁰ The data used in large-scale AI and ADM models is often de-identified or anonymised, but it inherently mirrors historical inequalities and biases, thus allowing the Automated Banks to claim impartiality and avoid responsibility for the unfairness of data used.

The reason why privacy and data protection rules lack clear consideration of certain data practices and processes enabled by AI may be due to these tools and processes being relatively new and poorly understood phenomena.⁸¹ This status quo is however very convenient for the companies, who will often raise the argument that ‘innovation’ will suffer if more stringent regulation is introduced.⁸²

4.3 Rules That Incentivise the Use of ADM Tools by Financial Entities

In addition to offering direct pathways allowing Automated Banks to evade scrutiny of their AI and ADM models, legal systems and markets in the developed world have also evolved to incentivise the use of automated technology by financial entities. In fact, the use of ADM and AI tools is encouraged, or sometimes even mandated,⁸³ by legal and regulatory frameworks. After all, the fact that they are told to either use the technology, or to achieve outcomes that can effectively only be reached with the application of the tools in question, provides a basis for a very convenient excuse. Though this is mainly an unintended effect of the rules, it should not be ignored.

In this section, we discuss two examples of rules that increase the secrecy of AI or ADM tools used in the context of risk scoring: financial products governance rules and ‘open banking’ regimes.

4.3.1 Financial Products Governance Rules

Financial firms have always been concerned with collecting and using data about their consumers, to differentiate between more and less valuable customers. For example, insurance firms, even before AI profiling tools were invented (or at least before they were applied at a greater scale) were known to engage in practices referred to as ‘cherry-picking’ and ‘lemon-dropping’, setting up firms’ offices at higher floors in buildings with no lifts, so that it would be harder for disabled (potential) clients to reach them.⁸⁴ There is a risk that the widespread data profiling and use of AI tools may exacerbate issues relating to consumers’ access to financial products and services. AI tools may introduce new or replicate historical biases present in data,⁸⁵ doing so more efficiently, in a way that is more difficult to discover, and at a greater scale than was possible previously.⁸⁶

An additional disadvantage resulting from opaque risk scoring systems is that consumers may miss out on the opportunity to improve their score (for example, through the provision of counterfactual explanations, or the use of techniques including ‘nearby possible worlds’).⁸⁷ In instances where potential customers who would have no trouble paying back loans are given low risk scores, two key issues arise: first, the bank misses out on valuable customers, and second, there is a risk that these customers’ rejections, if used as input data to train the selection algorithm, will reinforce existing biases.⁸⁸

Guaranteeing suitability of financial services is a notoriously complicated task for policymakers and regulators. With disclosure duties alone proving largely unsuccessful in addressing the issue of consumers being offered financial products that are unfit for purpose, policymakers in a number of jurisdictions, such as the EU and its Member States, the United Kingdom, Hong Kong, Australia, and Singapore, have started turning to product governance regimes.⁸⁹ An important component of these financial product governance regimes is an obligation placed on financial firms, which issue and distribute financial products, to ensure their products are fitness-for-purpose and to adopt a consumer-centric approach in design and distribution of the products. In particular, a number of jurisdictions require financial firms to delimit the target market for their financial products directed at retail customers, and ensure the distribution of the products within this target market. Such target market is a group of consumers of a certain financial product who are defined by some general characteristics.⁹⁰

Guides issued by regulators, such as the European Securities and Markets Authority⁹¹ and the Australian Securities and Investment Commission,⁹² indicate which consumers’ characteristics are to be taken into account by financial firms. The consumers for whom the product is intended are to be identified according to their ‘likely objectives, financial situation, and needs’,⁹³ or five ‘categories’: the type of client, their knowledge and experience, financial situation, risk tolerance, and objective and needs.⁹⁴ For issuers or manufacturers of financial products these considerations are mostly theoretical: as they might not have direct contact with clients, they need to prepare a potential target market, aiming at theoretical consumers and their likely needs and characteristics.⁹⁵ Both issuers and distributors need to take reasonable steps to ensure that products are distributed within the target market, which then translates to the identification of real consumers with specific needs and characteristics that should be compatible with the potential target markets identified. Distributors have to hold sufficient information about their end clients to be able to assess if they can be included in the target market,⁹⁶ including:

– indicators about the likely circumstances of the consumer or a class of consumers (e.g. concession card status, income, employment status);

– reasonable inferences about the likely circumstances of the consumer or a class of consumers (e.g. for insurance, information inferred from the postcode of the consumer’s residential address); or

– data that the distributor may already hold about the consumer or similar consumers, or results derived from analyses of that data (e.g. analysis undertaken by the distributor of common characteristics of consumers who have purchased a product).⁹⁷

Financial products governance frameworks invite financial firms to collect data on consumers’ vulnerabilities. For example in Australia, financial firms need to consider vulnerabilities consumers may have, such as those resulting from ‘personal or social characteristics that can affect a person’s ability to manage financial interactions’,⁹⁸ as well as those brought about by ‘specific life events or temporary difficulties’,⁹⁹ in addition to vulnerabilities stemming from the product design or market actions.

The rationale of product governance rules is to protect financial consumers, including vulnerable consumers,¹⁰⁰ yet the same vulnerable consumers may be disproportionately affected by data profiling, thus inhibiting their access to financial products. Financial law is actively asking firms to collect even more data about their current, prospective, and past customers, as well as the general public. It provides more than a convenient excuse to carry out digital profiling and collect data for even more precise risk scoring – it actually mandates this.

4.3.2 How ‘Open Banking’ Increases Opacity

Use of AI and ADM tools, together with ever-increasing data collection feeding the data hungry models,¹⁰¹ is promoted as beneficial to consumers and markets, and endorsed by companies and governments. Data collection is thus held out as a necessary component of fostering AI innovation. Companies boast how AI insights allow them to offer personalised services, ‘tailored’ to individual consumer’s needs. McKinsey consulting firm hails ‘harnessing the power of external data’ noting how ‘few organizations take full advantage of data generated outside their walls. A well-structured plan for using external data can provide a competitive edge’.¹⁰²

Policymakers use the same rhetoric of promoting ‘innovation’ and encourage data collection through schemes such as open banking.¹⁰³ The aim of open banking is to give consumers the ability to direct companies that hold financial data about themselves to make it available to financial (or other) companies of the consumer’s choice. Thus, it makes it possible for organisations to get access to consumers’ information they could never get from a consumer directly, such as for example their transaction data for the past ten years.

Jurisdictions such as the EU, United Kingdom, Australia, and Hong Kong have recently adopted regulation promoting open banking, or ‘open finance’ more generally.¹⁰⁴ The frameworks are praised by the industry as ‘encourag[ing] the development of innovative products and services that help consumers better engage with their finances, make empowered decisions and access tailored products and services’.¹⁰⁵

While open banking is making it possible for financial firms to develop new products for consumers, the jury is still out as to the scheme’s universally positive implications for consumers and markets.¹⁰⁶ One thing that is clear, however, is that because of its very nature, open banking contributes to information and power asymmetry between consumers and Automated Banks.

Traditionally, in order to receive a financial product, such as a loan or an insurance product, consumers would have to actively provide relevant data, answering questions or prompts, in relation to their income, spending, age, history of loan repayments, and so on. Open banking – or open finance more broadly – means that consumers can access financial products without answering any questions. But these questions provided a level of transparency to consumers: they knew what they were being asked, and were likely to understand why they were being asked such questions. But when an individual shares their ‘bulk’ data, such as their banking transaction history, through the open banking scheme, do they really know what a financial firm is looking for and how it is being used? At the same time, in such a setting, consumers are deprived of control over which data to share (for example, they cannot just hide transaction data on payments they made to merchants such as liquor stores or pharmacies). The transparency for financial firms when data is shared is therefore significantly higher than in ‘traditional’ settings – but for consumers the process becomes more opaque.¹⁰⁷

4.4 Can Corporate Secrecy Coexist with Consumer Rights? Possible Regulatory Solutions

ADM tools contribute to maintaining corporate secrecy of Automated Banks, and as we argue in this chapter, legal systems perpetuate, encourage, and feed the opacity further. The opacity then increases the risk of consumer harm, such as discrimination, which is more difficult to observe, and more challenging to prove.

In this section we provide a brief outline of potential interventions that may protect against AI-facilitated harms, particularly if applied synchronously. This discussion does not aim to be exhaustive, but rather aims to show something can be done to combat the opacity and resulting harms.

Interventions described in academic and grey literature can be divided into three broad categories: (1) regulations that prevent businesses from using harmful AI systems in financial markets, (2) regulations that aid consumers to understand when ADM systems are used in financial markets, and (3) regulations that facilitate regulator monitoring and enforcement against AI-driven harms in financial markets. Approaches to design (including Transparency by Design¹⁰⁸) are not included in this list, and while they may contribute to improved consumer outcomes, they are beyond the scope of this chapter.

The somewhat provocative title of this section asks if corporate secrecy is the real source of the AI-related harms in the described context. The interventions outlined below focus on preventing harms, but can the harms really be prevented if the opacity of corporate practices and processes is not addressed first? Corporate secrecy is the major challenge to accountability and scrutiny, and consumer rights, including right to non-discrimination, cannot be guaranteed in an environment as opaque as it currently is. We submit that the regulatory interventions urgently needed are the ones that prevent secrecy first and foremost. AI and ADM tools will continue to evolve, and technology as such is not a good regulatory target¹⁰⁹ – the focus must be on harm prevention. Harms can only be prevented if the practices of financial firms, such as credit scoring discussed in this chapter, are transparent and easily monitored both by regulators and consumers.

4.4.1 Preventing Automated Banks from Designing Harmful AI Systems

International and national bodies in multiple jurisdictions have recently adopted, or are currently debating, various measures with an overarching aim of protecting consumers from harm. For example, the US Federal Trade Commission has provided guidance to businesses using AI, explaining that discriminatory outcomes resulting from the use of AI would contravene federal law.¹¹⁰ The most comprehensive approach to limiting the use of particular AI tools can be found in the EU’s proposed Artificial Intelligence Act. Its Recital 37 specifically recommends that ‘AI systems used to evaluate the credit score or creditworthiness of natural persons should be classified as high-risk AI systems’. This proposal is a step towards overcoming some opaque practices, through the provision of ‘clear and adequate information to the user’ along with other protections that enable authorities to scrutinise elements of ADM tools in high-risk contexts.¹¹¹ Early criticisms of the proposed Act note that while a regulatory approach informed by the context in which ADM is used has some merit, it does not cover potentially harmful practices such as emotion recognition and remote biometric identification,¹¹² which could be used across a range of contexts, generating data sets that may later be used in other markets such as financial services.

An alternative approach to regulating AI systems before they are used in markets is to limit the sources of information that can be used by ADM tools, or restrict the ways in which information can be processed. In addition to privacy protections, some jurisdictions have placed limitations on the kinds of information that can be used to calculate a credit score. For example, in Denmark, the financial services sector can use consumers’ social media data for marketing purposes but is explicitly prohibited from using this information to determine creditworthiness.¹¹³ Similarly, the EU is considering a Directive preventing the use of personal social media and health data (including cancer data) in the determination of creditworthiness.¹¹⁴ Such prohibitions are, however, a rather tricky solution: it may be difficult for the regulation to keep up with a growing list of data that should be excluded from analysis.¹¹⁵ One way of overcoming this challenge would be to avoid focusing on restricted data sources, and instead create a list of acceptable data sources, which is a solution applied for example in some types of health insurance.¹¹⁶

Imposing limits on how long scores can be kept and/or relied on by Automated Banks is another important consideration. In Australia, credit providers are bound by limits that stipulate the length of time that different pieces of information are held on a consumer’s file: credit providers may only keep financial hardship information for twelve months from the date the monthly payment was made under a financial hardship arrangement, whereas court judgements may be kept on record for five years after the date of the decision.¹¹⁷ In Denmark, where the credit reporting system operates as a ‘blacklist’ of people deemed more likely to default, a negative record (for instance, an unpaid debt) is deleted after five years, regardless of whether or not the debt has been paid.¹¹⁸ A challenge with these approaches is that the amount of time particular categories of data may be kept may not account for proxy data, purchased data sets, and/or proprietary scoring and profiling systems that group consumers according to complex predictions that are impossible to decode.

4.4.2 Aiding Consumers Understand When ADM Systems Are Used in Financial Services

Despite the development of many principles-based regulatory initiatives by governments, corporates, and think tanks,¹¹⁹ few jurisdictions have legislated protections that require consumers to be notified if and when they have been assessed by an automated system.¹²⁰ In instances where consumers are notified, they may be unable to receive an understandable explanation of the decision-making process, or to seek redress through timely and accessible avenues.

Consumers face a number of challenges in navigating financial markets, such as understanding credit card repayment requirements¹²¹ and failing to accurately assess their credit.¹²² For individuals, it is crucial to understand how they are being scored, as this will make it possible for them to be able to identify inaccuracies,¹²³ and question decisions made about them. Credit scoring is notoriously opaque and difficult to understand, so consumers are likely to benefit from requirements for agencies to simplify and harmonise how scores are presented.¹²⁴An example of a single scoring system can be found in Sri Lanka, where credit ratings, or ‘CRIB Scores’ are provided by the Central Information Bureau of Sri Lanka, a public-private partnership between the nation’s Central Bank and a number of financial institutions that hold equity in the Bureau. The Bureau issues CRIB Score reports to consumers in a consistent manner, utilising an algorithm to produce a three-digit number ranging from 250 to 900.¹²⁵ In Sri Lanka’s case, consumers are provided with a singular rating from a central agency, and although this rating is subject to change over time, there is no possibility of consumers receiving two different credit scores from separate providers.

Providing consumers with the opportunity to access their credit scores is another (and in many ways complementary) regulatory intervention. A number of jurisdictions provide consumers with the option to check their credit report and/or credit score online. For example, consumers in Canada¹²⁶ and Australia¹²⁷ are able to access free copies of their credit reports by requesting this information directly from major credit bureaus. In Australia, consumers are able to receive a free copy of their credit report once every three months.¹²⁸

However, such approaches have important limitations. Credit ratings are just one of many automated processes within the financial services industry. Automated Banks, with access to enough data, can create their own tools going outside the well-established credit rating systems. Also, it is consumers who are forced to carry the burden of correcting inaccurate information which is used to make consequential decisions about them, while often being required to pay for the opportunity to do so.¹²⁹

In addition, explainability challenges are faced in every sector that uses AI, and there is considerable investigation ahead to determine the most effective ways of explaining automated decisions in financial markets. It has been suggested that a good explanation is provided when the receiver ‘can no longer keep asking why’.¹³⁰ The recent EU Digital Services Act¹³¹ emphasises such approach by noting that recipients of online advertisements should have access to ‘meaningful explanations of the logic used’ for ‘determining that specific advertisement is to be displayed to them’.¹³²

Consumer experience of an AI system will depend on a number of parameters, including format of explanations (visual, rule-based, or highlighted key features), their complexity and specificity, application context, and variations suiting users’ cognitive styles (for example, providing some users with more complex information, and others with less).¹³³ The development of consumer-facing explainable AI tools is an emerging area of research and practice.¹³⁴

A requirement of providing meaningful feedback to consumers, for example, through counterfactual demonstrations,¹³⁵ would make it possible for individuals to understand what factors they might need to change to receive a different decision. It would also be an incentive for Automated Banks to be more transparent.

4.4.3 Facilitating Regulator Monitoring and Enforcement of ADM Harms in Financial Services

The third category of potential measures relies on empowering regulators, thus shifting the burden away from consumers. For example, regulators need to be able to ‘look under the hood’ of any ADM tools, including these of proprietary character.¹³⁶ This could be in a form of using explainable AI tools, access to raw code, or ability to use dummy data to test the model. A certification scheme, such as quality standards, is another option, the problem however is the risk of ‘set and forget approach’. Another approach to providing regulators insight into industry practices is the establishment of regulatory sandboxes, which nevertheless have limitations.¹³⁷

Financial institutions could also be required to prove a causal link between the data that they use to generate consumer scores, and likely risk. Such approach would likely reduce the use of certain categories of data, where correlations between data points would not be supported by a valid causal relationship. For example, Android phone users are reportedly safer drivers than iPhone users,¹³⁸ but such rule would prevent insurers from taking this into account when offering a quote on car insurance (while we do not suggest they are currently doing so, in many legal systems they could). In practice, some regulators are looking at this solution. For example, while not going as far as requiring direct causal link, the New York State financial regulator requires a ‘valid explanation or rationale’ for underwriting of life insurance, where external data or external predictive models are used.¹³⁹ However, such approach could result in encouraging financial services providers to collect more data, just to be able to prove the causal link,¹⁴⁰ which may again further disadvantage consumers and introduce more, not less, opacity.

4.5 Conclusions

Far from being unique to credit scoring, the secrecy of ADM tools is a problem affecting multiple sectors and industries.¹⁴¹ Human decisions are also unexplainable and opaque, and ADM tools are often made out to be a potential, fairer and more transparent, alternative. But the problem is secrecy increases, not decreases, with automation.¹⁴²

There are many reasons for this, including purely technological barriers to explainability. But also, it is obviously cheaper and easier not to design and use transparent systems. As we argue in this chapter, opacity is a choice made by organisations, often on purpose, as it allows them to evade scrutiny and hide their practices from the public and regulators. Opacity of ADM and AI tools used is a logical consequence of the secrecy of corporate practices.

Despite many harms caused by opacity, the legal systems and market practice have evolved to enable or even promote that secrecy surrounding AI and ADM tools, as we have discussed using examples of rules applying to Automated Banks. However, the opacity and harms could be prevented with some of the potential solutions which we have discussed in this chapter. The question is whether there is sufficient motivation to achieve positive social impact with automated tools, without just focusing on optimisation and profits.