Part VI - Responsible Corporate Governance of AI Systems

19 From Corporate Governance to Algorithm Governance Artificial Intelligence as a Challenge for Corporations and Their Executives

Jan Lieder

I. Introduction

Every generation has its topic: The topic of our generation is digitalization. At present, we are all witnessing the so-called industrial revolution 4.0.¹ This revolution is characterized by the use of a whole range of new digital technologies that can be combined in a variety of ways. Keywords are self-learning algorithms, Artificial Intelligence (AI), autonomous systems, Big Data, biometrics, cloud computing, Internet of Things, mobile internet, robotics, and social media.²

The use of digital technologies challenges the law and those applying it. The range of questions and problems is tremendously broad.³ Widely discussed examples are self-driving cars,⁴ the use of digital technologies in corporate finance, credit financing and credit protection,⁵ the digital estate,⁶ or online dispute resolution.⁷ In fact, digital technologies challenge the entire national legal system including public and criminal law as well as EU and international law. Some even say we may face ‘the beginning of the end for the law’.⁸ In fact, this is not the end, but rather the time for a digital initiative. This chapter focuses on the changes that AI brings about in corporate law and corporate governance, especially in terms of the challenges for corporations and their executives.

From a conceptual perspective, AI applications will have a major impact on corporate law in general and corporate governance in particular. In practice, AI poses a tremendous challenge for corporations and their executives. As algorithms have already entered the boardroom, lawmakers must consider legally recognizing e-persons as directors and managers. The applicable law must deal with effects of AI on corporate duties of boards and their liabilities. The interdependencies of AI, delegation of leadership tasks, and the business judgement rule as a safe harbor for executives are of particular importance. A further issue to be addressed is how AI will change the decision-making process in corporations as a whole. This topic is closely connected with the board’s duties in Big Data and Data Governance as well as the qualifications and responsibilities of directors and managers.

By referring to AI, I mean information technology systems that reproduce or approximate various cognitive abilities of humans.⁹ In the same breath, we need to distinguish between strong AI and weak AI. Currently, strong AI does not exist.¹⁰ There is no system really imitating a human being, such as a so-called superintelligence. Only weak AI is applied today. These are single technologies for smart human–machine interactions, such as machine learning or deep learning. Weak AI focuses on the solution of specific application problems based on the methods from math and computer science, whereby the systems are capable of self-optimization.¹¹

By referring to corporate governance, I mean a system by which companies are directed and controlled.¹² In continental European jurisdictions, such as Germany, a dual board structure is the prevailing system with a management board running the day-to-day business of the firm and a supervisory board monitoring the business decisions of the management board. In Anglo-American jurisdictions, such as the United States (US) and the United Kingdom (UK), the two functions of management and supervision are combined within one unitary board – the board of directors.¹³

II. Algorithms As Directors

The first question is, “Could and should algorithms act as directors?” In 2014, newspapers reported that a venture capital firm had just appointed an algorithm to its board of directors. The Hong Kong based VC firm Deep Knowledge Ventures was supposed to have appointed an algorithm called Vital (an abbreviation for Validating Investment Tool for Advancing Life Sciences) to serve as a director with full voting rights and full decision-making power over corporate measures.¹⁴ In fact, Vital only had an observer and adviser status with regard to the board members, which are all natural persons.¹⁵

Under German law according to sections 76(3) and 100(1)(1) AktG,¹⁶ the members of the management board and the supervisory board must be natural persons with full legal capacity. Not even corporations are allowed to serve as board members. That means, in order to appoint algorithms as directors, the law must be changed.¹⁷ Actually, the lawmaker could legally recognize e-persons as directors. However, the lawmaker should not do so, because there is a reason for the exclusion of legal persons and algorithms under German law. Both lack personal liability and personal accountability for the management and the supervision of the company.¹⁸

Nevertheless, the European Parliament enacted a resolution with recommendations to the Commission on Civil Law Rules on Robotics, and suggested therein

creating a specific legal status for robots in the long run, so that at least the most sophisticated autonomous robots could be established as having the status of electronic persons responsible for making good any damage they may cause, and possibly applying electronic personality to cases where robots make autonomous decisions or otherwise interact with third parties independently.¹⁹

The most fundamental requirement for legally recognizing an e-person would be its own liability – either based on an ownership fund or based on a mandatory liability insurance. In case corporations are appointing AI entities as directors (or apply it otherwise), they should be strictly liable for damages caused by AI applications in order to mitigate the particular challenges and potential risks of AI.²⁰ This is because strict liability would not only delegate the risk assessment and thus control the level of care and activity, but would also create an incentive for further developing this technology.²¹ At the same time, creditors of the company should be protected by a compulsory liability insurance, whereas piercing the corporate veil, that is, a personal liability of the shareholders, must remain a rare exception.²² However, at an international level, regulatory competition makes it difficult to guarantee comparable standards. Harmonization can only be expected (if ever) in supranational legal systems, such as the European Union.²³ In this context, it is noteworthy that the EU Commission’s White Paper on AI presented in 2020 does not address the questions of the legal status of algorithms at all.²⁴

However, even if we were to establish such a liability safeguard, there is no self-interested action of an algorithm as long as there is no strong AI. True, circumstances may change in the future due to technological progress. However, there is a long and winding road to the notorious superintelligence.²⁵ Conversely, weak AI only carries out actions in the third-party interest of people or organizations, and is currently not in a position to make its own value decisions and judgemental considerations.²⁶ In the end, current algorithms are nothing more than digital slaves, albeit slaves with superhuman abilities. In addition, the currently applicable incentive system of corporate law and governance would have to be adapted to AI directors, because – unlike human directors – duties of loyalty can hardly be applied to them, but rather they decide according to algorithmic models.²⁷ At present, only humans have original creative power, only they are capable of making decisions and acting in the true sense of the word.²⁸

III. Management Board

Given the current limitations of AI, we will continue to have to get by with human directors for the next few decades. Although algorithms do not currently appear suitable for making independent corporate decisions, AI can nonetheless support human directors in their management and monitoring tasks. AI is already used in practice to analyze and forecast the financial development of a company, but also to identify the need for optimization in an entrepreneurial value chain.²⁹ In addition, AI applications are used in the run-up to mergers and acquisitions (M&A) transactions,³⁰ namely as part of due diligence, in order to simplify particularly labor-intensive processes when checking documents. Algorithms are also able to recognize unusual contract clauses and to summarize essential parameters of contracts, even to create contract templates themselves.³¹ Further examples for the use of AI applications are cybersecurity³² and compliance management systems.³³

1. Legal Framework

With regard to the German corporate governance system, the management board is responsible for running the company.³⁴ Consequently, the management board also decides on the overall corporate strategy, the degree of digitalization and the use of AI applications.³⁵ The supervisory board monitors the business decisions of the management board, decides on the approval of particularly important measures,³⁶ as well as on the appointment and removal of the management board members;³⁷ whereas the shareholders meeting does not determine a company’s digitalization structures.³⁸

2. AI Related Duties

In principle, the use of AI neither constitutes a violation of corporate law or the articles of association,³⁹ nor is it an expression of bad corporate governance. Even if the use of AI is associated with risks, it is difficult to advise companies – as the safest option – to forego it completely.⁴⁰ Instead, the use of AI places special demands on the management board members.

a. General Responsibilities

Managers must have a fundamental understanding of the relevant AI applications, of their potentials, suitability, and risks. However, the board members do not need to have in-depth knowledge about the detailed functioning of a certain AI application. In particular, the knowledge of an IT expert cannot be demanded, nor a detailed examination of the material correctness of the decision.⁴¹ Rather, they need to have an understanding of the scope and limits of an application and possible results and outcomes of the application in order to perform plausibility checks to prevent incorrect decisions quickly and effectively.⁴² The management board has to ensure, through test runs, the functionality of the application with regard to the concrete fulfilment of tasks in the specific company environment.⁴³ If, according to the specific nature of the AI application, there is the possibility of an adjustment to the concrete circumstances of the company, for example, with regard to the firm’s risk profile or statutory provisions, then the management board is obliged to carry out such an adjustment.⁴⁴ During the use of the AI, the board of directors must continuously evaluate and monitor the working methods, information procurement, and information evaluation as well as the results achieved.

The management board must implement a system that eliminates, as far as possible, the risks and false results that arise from the use of AI. This system must assure that anyone who uses AI knows the respective scope of possible results of an application so that it can be determined whether a concrete result is still within the possible range of results. However, that can hardly be determined abstractly, but requires a close look at the concrete AI application. Furthermore, the market standard is to be included in the analysis. If all companies in a certain industry use certain AI applications that are considered safe and effective, then an application by other companies will rarely prove to breach a management board’s duty of care.

Under these conditions, the management board is allowed to delegate decisions and tasks to an AI application.⁴⁵ This is not contradicted by the fact that algorithms lack legal capacity, because in this context the board’s own duties are decisive.⁴⁶ In any event, a blanket self-commitment to the results of an AI application is incompatible with the management responsibility and personal accountability of the board members.⁴⁷ At all times, the applied AI must be manageable and controllable in order to ensure that no human loss of control occurs and the decision-making process is comprehensible. The person responsible for applying AI in a certain corporate setting must always be able to operate the off-switch. In normative terms, this requirement is derived from section 91(2) AktG, which obliges the management board to take suitable measures to identify, at an early stage, developments that could jeopardize the continued existence of the company.⁴⁸ In addition, the application must be protected against external attacks, and emergency precautions must be implemented in the event of a technical malfunction.⁴⁹

b. Delegation of Responsibility

The board may delegate the responsibility for applying AI to subordinate employees, but it is required to carefully select, instruct, and supervise the delegate.⁵⁰ Under the prevailing view, core tasks, however, cannot be delegated, as board members are not allowed to evade their leadership responsibility.⁵¹ Such non-delegable management tasks of the management board include basic measures with regard to the strategic direction, business policy and the organization of the company.⁵² The decision as to whether and to what extent AI should be used in the company is also a management measure that cannot be delegated under the prevailing view.⁵³ Only the preparation of decisions by auxiliary persons is permissible, as long as the board of directors makes the decision personally and of its own responsibility. In this respect, the board is responsible for the selection of AI use and the application of AI in general. The board has to provide the necessary information, must exclude conflicts of interest and has to perform plausibility checks of the results obtained. Furthermore, the managers must conduct an ongoing monitoring and ensure that the assigned tasks are properly performed.

c. Data Governance

AI relies on extensive data sets (Big Data). In this respect, the management board is responsible for a wide scope and high quality of the available data, for the suitability and training of AI applications, and for the coordination of the model predictions with the objectives of the respective company.⁵⁴ In addition, the board of directors must observe data protection law limits⁵⁵ and must pursue a non-discriminatory procedure.⁵⁶ If AI use is not in line with these regulations or other mandatory provisions, the management board violates the duty of legality.⁵⁷ In this case, the management board does not benefit from the liability privilege of the business judgement rule.⁵⁸

Apart from that, the management board has an entrepreneurial discretion with regard to the proper organization of the company’s internal knowledge organization.⁵⁹ The starting point is the management board’s duty to ensure a legal, statutory, and appropriate organizational structure.⁶⁰ The specific scope and content of the obligation to organize knowledge depends largely on the type, size, and industry of the company and its resources.⁶¹ However, if, according to these principles, there is a breach of the obligation to store, forward, and actually query information, then the company will be considered to have acted with knowledge or negligent ignorance under German law.⁶²

d. Management Liability

If managers violate these obligations (and do not benefit from the liability privilege of the business judgement rule)⁶³, they can be held liable for damages to the company.⁶⁴ This applies in particular in the event of an inadmissible or inadequate delegation.⁶⁵ In order to mitigate the liability risk for management board members, they have to ensure that the whole framework of AI usage in terms of specific applications, competences, and responsibilities as well as the AI-related flow of information within the company is well designed and documented in detail. Conversely, board members are not liable for individual algorithmic errors as long as (1) the algorithm works reliably, (2) the algorithm does not make unlawful decisions, (3) there are no conflicts of interest, and (4) the AI’s functioning is fundamentally overseen and properly documented.⁶⁶

Comprehensive documentation of the circumstances that prompted the management board to use a certain AI and the specific circumstances of its application reduces the risk of being sued for damages by the company. This ensures, in particular, that the members of the management board can handle the burden of proof incumbent on them according to section 93(2)(2) AktG. They will achieve this better, the more detailed the decision-making process regarding the use of AI can be understood from the written documents.⁶⁷ This kind of documentation by the management board is to be distinguished from general documentation requirements discussed at the European and national level for the development of AI models and for access authorization to this documentation, the details of which are beyond the scope of this chapter.⁶⁸

e. Composition of the Management Board

In order to cope with the challenges that the use of AI applications causes, the structure and composition of the management and the board has already changed significantly. That manifests itself in the establishment of new management positions, such as a Chief Information Officer (CIO)⁶⁹ or a Chief Digital Officer (CDO).⁷⁰ Almost half of the 40 largest German companies have such a position at board level.⁷¹

In addition, soft factors are becoming increasingly important in corporate management. Just think of the damage to the company’s reputation, which is one of the tangible economic factors of a company today.⁷² Under the term Corporate Digital Responsibility (CDR), specific responsibilities are developing for the use of AI and other digital innovations.⁷³ For example, Deutsche Telekom AG has enacted nine guidelines for responsible AI in a corporate setting. SAP SE established an advisory board for responsible AI consisting of experts from academia, politics, and the industry. These developments, of course, have an important influence on the overall knowledge attribution within the company and a corporate group. AI and Big Data make information available faster and facilitate the decision-making process at board level. Therefore, the management board must examine whether the absence of any AI application in the information gathering and decision-making process is in the best interest of a company. However, a duty to use AI applications only exists in exceptional cases and depends on the market standard in the respective industry. The greater the amount of data to be managed and the more complex and calculation-extensive the decisions in question, the more likely it is that the management board will be obliged to use AI.⁷⁴

3. Business Judgement Rule

This point is closely connected with the application of the business judgement rule as a safe harbour for AI use. Under the general concept of the business judgement rule that is well-known in many jurisdictions⁷⁵, as it is in Germany according to section 93(1)(2) AktG, a director cannot be held liable for an entrepreneurial decision if there is no conflict of interest and she had good reason to assume she was acting based on adequate information and for the benefit of the company.

a. Adequate Information

The requirement of adequate information depends significantly on the ability to gather and analyse information. Taking into account all the circumstances of the specific individual case, the board of directors has a considerable amount of leeway to judge which information is to be obtained from an economic point of view in the time available and to be included in the decision-making process. Neither a comprehensive nor the best possible, but only an appropriate information basis is necessary.⁷⁶ In addition, the appropriateness is to be assessed from the subjective perspective of the board members (‘could reasonably assume’), so that a court is effectively prevented during the subsequent review from substituting its own understanding of appropriateness for the subjective assessment of the decision-maker.⁷⁷ In the context of litigation, a plausibility check based on justifiability is decisive.⁷⁸

In general, the type, size, purpose, and organization of the company as well as the availability of a functional AI and the data required for operation are relevant for answering the question of the extent to which AI must be used in the context of the decision-making preparation based on information. The cost of the AI system and the proportionality of the information procurement must also be taken into account.⁷⁹ If there is a great amount of data to be managed and a complex and calculation-intensive decision to be made, AI and Big Data applications are of major importance and the members of the management board will hardly be able to justify not using AI.⁸⁰ Conversely, the use of AI to obtain information is definitely not objectionable.⁸¹

b. Benefit of the Company

Furthermore, the board of directors must reasonably assume to act in the best interest of the company when using AI. This criterion is to be assessed from an ex ante perspective, not ex post.⁸² According to the mixed-subjective standard, it depends largely on the concrete perception of the acting board members at the time of the entrepreneurial decision.⁸³ In principle, the board of directors is free to organize the operation of the company according to its own ideas, as long as it stays within the limits of the best interest of the corporation⁸⁴ that are informed solely by the existence and the long-term and sustainable profitability of the company.⁸⁵ Only when the board members act in a grossly negligent manner or take irresponsible risks do they act outside the company’s best interest.⁸⁶ Taking all these aspects into account, the criterion of acceptability proves to be a suitable benchmark.⁸⁷

In the specific decision-making process, all advantages and disadvantages of using or delegating the decision to use AI applications must be included and carefully weighed against one another for the benefit of the company. In this context, however, it cannot simply be seen as unacceptable and contrary to the welfare of the company that the decisions made by or with the support of AI can no longer be understood from a purely human perspective.⁸⁸ On the one hand, human decisions that require a certain originality and creativity cannot always be traced down to the last detail. On the other hand, one of the major potentials of AI is to harness particularly creative and original ideas in the area of corporate management. AI can, therefore, be used as long as its use is not associated with unacceptable risks. The business judgement rule allows the management board to consciously take at least justifiable risks in the best interest of the company.

However, the management board may also conclude that applying AI is just too much of a risk for the existence or the profitability of the firm and therefore may refrain from it without taking a liability risk under section 93(1)(2) AktG.⁸⁹ The prerequisite for this is that the board performs a conscious act of decision-making.⁹⁰ Otherwise, acting in good faith for the benefit of the company is ruled out a priori. This decision can also consist of a conscious toleration or omission.⁹¹ The same applies to intuitive action,⁹² even if in this case the other requirements of section 93(1)(2) AktG must be subjected to a particularly thorough examination.⁹³ Furthermore, in addition to the action taken, there must have been another alternative,⁹⁴ even if only to omit the action taken. Even if the decision makers submit themselves to an actual or supposed necessity,⁹⁵ they could at least hypothetically have omitted the action. Apart from that, the decision does not need to manifest itself in a formal act of forming a will; in particular, a resolution by the collective body is not a prerequisite. Conversely, with a view to a later (judicial) dispute, it makes sense to sufficiently document the decision.⁹⁶

c. Freedom from Conflicts of Interest

The executive board must make the decision for or against the use of AI free of extraneous influences and special interests.⁹⁷ The business judgement rule does not apply if the board members are not solely guided by the points mentioned above, but rather pursue other, namely self-interested goals. If the use of AI is not based on inappropriate interests and the board of directors has not influenced the parameters specified for the AI in a self-interested manner, the use of AI applications can contribute to a reduction of transaction costs from an economic point of view and mitigate the principle-agent-conflict, as the interest of the firm will be aligned with decisions made by AI.⁹⁸ That is, AI can make the decision-making process (more) objective.⁹⁹ However, in order to achieve an actually objective result, the quality of the data used is decisive. If the data set itself is characterized by discriminatory or incorrect information, the result will also suffer from those weaknesses (‘garbage in – garbage out’). Moreover, if the management board is in charge of developing AI applications inside the firm, it may have an interest in choosing experts and technology designs that favor its own benefit rather than the best interest of the company. This development could aggravate the principle-agent-conflict within the large public firm.¹⁰⁰

IV. Supervisory Board

For this reason, it will also be of fundamental importance in the future to have an institutional monitoring body in the form of the supervisory board, which enforces the interests of the company as an internal corporate governance system. With regard to the monitoring function, there is a distinction to be made as to whether the supervisory board makes use of AI itself while monitoring and advising the management of the company, or whether the supervisory board is monitoring and advising with regard to the use of AI by the management board.

1. Use of AI by the Supervisory Board Itself

As the members of the management board and of the supervisory board have to comply with the same basic standards of care and responsibility under sections 116(1) and 93(1)(1) AktG, the management board’s AI related duties¹⁰¹ essentially apply to the supervisory board accordingly. If the supervisory board is making an entrepreneurial decision, it can also rely on the business judgement rule.¹⁰² This is true, for example, for the granting of approval for transactions requiring approval under section 111(4)(2) AktG, with regard to M&A transactions.¹⁰³ Furthermore, the supervisory board may use AI based personality and fitness checks when it appoints and dismisses management board members.¹⁰⁴ AI applications can help the supervisory board to structure the remuneration of the management board appropriately. They can also be useful for the supervisory board when auditing the accounting and in the compliance area, because they are able to analyze large amounts of data and uncover inconsistencies.¹⁰⁵

2. Monitoring of the Use of AI by the Management Board

When it comes to the monitoring and advice on the use of AI by the management board, the supervisory board has to fulfil its general monitoring obligation under section 111(1) AktG. The starting point is the reporting from the management board under section 90 AktG.¹⁰⁶ Namely strategic decisions on the leading guidelines of AI use is part of the intended business policy or at least another fundamental matter regarding the future conduct of the company’s business according to section 90(1)(1) AktG. Furthermore, the usage of certain AI applications may be qualified as transactions that may have a material affect upon the profitability or liquidity of the company under section 90(1)(4) AktG. In this regard, the management board does not need to derive and trace the decision-making process of the AI in detail. Rather, it is sufficient for the management board to report to the supervisory board about the result found and how it specifically used the AI, monitored its functions, and checked the plausibility of the result.¹⁰⁷ In addition, pursuant to section 90(3) AktG, the supervisory board may require at any time a report from the management board on the affairs of the company, on the company’s legal and business relationships with affiliated enterprises. This report may also deal with the AI related developments on the management board level and in other entities in a corporate group.

Finally, the supervisory board may inspect and examine the books and records of the company according to section 111(2)(1) AktG. It is undisputed that this also includes electronic recordings,¹⁰⁸ which the supervisory board can examine using AI in the form of a big data analysis.¹⁰⁹ Conversely, the supervisory board does not need to conduct its own inquiries using its information authority without sufficient cause or in the event of regular and orderly business development.¹¹⁰ Contrary to what the literature suggests,¹¹¹ this applies even in the event that the supervisory body has unhindered access to the company’s internal management information system.¹¹² The opposing view not only disregards the principle of a trusting cooperation between the management board and the supervisory board, but also surpasses the demands on the supervisory board members in terms of time.¹¹³

With a view to the monitoring standard, the supervisory board has to assess the management board’s overall strategy as regards AI applications and especially systemic risks that result from the usage of AI in the company. This also comprises the monitoring of the AI-based management and organizational structure of the company.¹¹⁴ If it recognizes violations of AI use by the management board, the supervisory board has to intervene using the general means of action. This may start with giving advice to the management board on how to optimize the AI strategy. Furthermore, the supervisory board may establish an approval right with regard to the overall AI-based management structure. In addition, the supervisory board may draw personnel conclusions and install an AI expert on the management board level such as a CIO or CDO.¹¹⁵

V. Conclusion

AI is not the end of corporate governance as some authors predicted.¹¹⁶ Rather, AI has the potential to change the overall corporate governance system significantly. As this chapter has shown, AI has the potential to improve corporate governance structures, especially when it comes to handling big data sets. At the same time, it poses challenges to the corporate management system, which must be met by carefully adapting the governance framework.¹¹⁷ However, currently, there is no need for a strict AI regulation with a specific focus on corporations.¹¹⁸ Rather, we see a creeping change from corporate governance to algorithm governance that has the potential to enhance, but also the risks to destabilize the current system. What we really need is the disclosure of information about a company’s practices with regard to AI application, organization, and oversight as well as potentials and risks.¹¹⁹ This kind of transparency would help to raise awareness and to enhance the overall algorithm governance system. For that purpose, the already mandatory corporate governance report that many jurisdictions require, such as the US,¹²⁰ the UK¹²¹ and Germany,¹²² should be supplemented with additional explanations on AI.¹²³

In this report, the management board and the supervisory board should report on their overall strategy with regard to the use, organization, and monitoring of AI applications. This specifically relates to the responsibilities, competencies, and protective measures they established to prevent damage to the corporation. In addition, the boards should also be obliged to report on the ethical guidelines for a trustworthy use of AI.¹²⁴ In this regard, they may rely on the proposals drawn up on an international level. Of particular importance in this respect are the principles of the European Commission in its communication on ‘Building Trust in Human-Centric Artificial Intelligence’,¹²⁵ as well as the ‘Principles on Artificial Intelligence’ published by the OECD.¹²⁶ These principles require users to comply with organizational precautions in order to prevent incorrect AI decisions, provide a minimum of technical proficiency, and ensure the preservation of human final decision-making authority. In addition, there is a safeguarding of individual rights, such as privacy, diversity, non-discrimination, fairness, and an orientation of AI to the common good, including sustainability, ecological responsibility, and overall societal and social impact. Even if these principles are not legally binding, a reporting obligation requires the management board and supervisory board to deal with the corresponding questions and to explain how they relate to them. It will make a difference and may lead to improvements if companies and their executives are aware of the importance of these principles in dealing with responsible AI.

20 Autonomization and Antitrust On the Construal of the Cartel Prohibition in the Light of Algorithmic Collusion

Stefan Thomas

I. Introduction

The use of algorithms is associated with a risk of collusion. This bears on the construal of the cartel prohibition, on which the present chapter focuses. The hypothesis is that algorithms may achieve a collusive equilibrium without any involvement of natural persons. Against this backdrop, it is questionable whether and to what extent such an outcome can be qualified as a concerted practice in terms of the law.

The analysis will be structured as follows: first, it will be assessed in what way algorithms can influence competition on markets (Section II). Subsequently, the article will deal with the traditional criteria of distinction between explicit and tacit collusion, which might reveal a potential gap in the existing legal framework with respect to algorithmic collusion (Section III). Finally, it must be analyzed whether the cartel prohibition can be construed in a way that captures the phenomenon appropriately (Section IV). The chapter will close with a summary (Section V).

II. Algorithmic Collusion As a Phenomenon on Markets

It is widely accepted that the use of algorithms can precipitate collusive outcomes, at least in theory. There is no lack of attempts to systematize the different ways algorithms can be involved here. Since the first groundbreaking publications by Ariel Ezrachi, Maurice Stucke, and Salil Mehra, as well as other authors in the following, the matter has come into the focus of antitrust scholarship and practice.¹ Agencies have started to look into it.² First cases have emerged about restraints implemented on platforms involving computer technology. With respect to the United States (US) Topkins³ must be mentioned, which involved alleged horizontal price fixing on a digital platform based on algorithms. For the European Union (EU), the Eturas⁴ case comes to mind. The operator of a travel booking platform had informed the travel agencies using that platform that it intended to cap rebates granted to end-consumers. The European Court of Justice (ECJ) held that this amounted to a horizontally concerted practice among the travel agencies to the extent that these had not objected to this proposal. The Luxembourg competition authority in 2018 found the taxi booking app Webtaxi to be exempt from the cartel prohibition although the system involved an algorithmic horizontal alignment of prices. The agency found offsetting efficiencies to the benefit of consumers.⁵ These cases have in common that the natural persons representing the companies involved were aware of the restrictions, or that they at least ought to have known about them. Algorithms were an element of implementing the restriction, yet the ultimate decision about the competitive restraint was taken by human beings. Under these conditions the cases did not pose severe difficulties in establishing explicit collusion. There is not a fundamental difference to cases in which parties communicate, for example, by way of hub-and-spoke cartelization through traditional means and forms of communication.⁶

A greater legal challenge is caused by the risk of autonomous algorithmic collusion. Computers with machine learning capabilities can possibly achieve or sustain a collusive equilibrium without any involvement of human knowledge or intent. The underlying scholarly discussion usually orbits around q-learning mechanisms.⁷ The hypothesis is that algorithms with machine learning capabilities can act as computer agents exploring the success of their own actions, from which a collusive strategy can emerge as the optimum. In this event, it is conceivable that even the programmer of the algorithm was not aware of this potential outcome.⁸ The market effect, therefore, can be a collusive equilibrium, albeit absent any human involvement.

Lawyers, economists, and computer scientists are still at odds over the likeliness and actual occurrence of autonomous algorithmic collusion. Some consider it a realistic scenario that tends to be underestimated.⁹ The German Bundeskartellamt and the French Autorité de la Concurrence have refrained from a definitive conclusion so far.¹⁰ Ulrich Schwalbe, in his seminal article, points out that the game theoretical dilemma that has to be solved by autonomous computer agents to achieve a stable collusive equilibrium is huge and cannot be easily overcome in practice.¹¹ A more recent study by Emilio Calvano and others¹², however, concludes that q-learning algorithms, in fact, can autonomously collude. The EU Commission, in its proposal for a ‘New Competition Tool’, mentions the risk that digital platforms can create ecosystems in which collusion arises, which can be read as a recognition of the phenomenon as a matter of concern.¹³ Against this backdrop, it is expedient to elaborate further on the application of the cartel prohibition in such cases.

III. On the Scope of the Cartel Prohibition and Its Traditional Construal

The conceptual problem behind the traditional construal of the cartel provision is the difference in the structure of the law on the one hand and the economic determinants for collusive equilibria on the other.¹⁴ Anticompetitive collusive equilibria are characterized by the fact that the participants consider it individually rational to pursue such a strategy. That is the case for types of collusion that are usually referred to as explicit and which are illegal in the same way as it holds true for so-called tacit collusion, which is seen as not to fall within the scope of the prohibition. Agreements in breach of the cartel prohibition are null and void so that they cannot be enforced. Therefore, any cartel is only stable so long as the firms participating in it consider it rational to remain involved. All types of collusive equilibria can, therefore, be considered as non-cooperative games in terms of game theory.¹⁵ Yet still, the law does not prohibit the achievement or sustaining of a collusive equilibrium as such. Instead, the provision is confined to certain types of measures, which are described as agreement, concerted practice, or decision. Mere tacit collusion is supposed to be distinct from these aforementioned types of anticompetitive conduct. While explicit collusion (i.e. achieved by agreement), concerted practice, or decision, is prohibited, tacit collusion is found to be legitimate.

As becomes obvious, the traditional construal of the law rests on a description of the means and forms by which firms interact when defining collusion. In the context at hand, the category of concerted practices has the greatest relevance. It is conceived of as something whereby a ‘practical cooperation’ is substituted for the risks of competition. Such practical cooperation, in turn, is supposed to be different from merely observing a rival’s conduct and reacting to it.¹⁶ Similar approaches of distinction apply under section 1 of the US Sherman Act.¹⁷ There, so-called conscious parallelism is deemed not to fall within the scope of the prohibition.¹⁸ For the finding of a cartel, so-called plus factors¹⁹, or ‘facilitating practices’/‘facilitating devices’ need to be established.²⁰

It is argued that, whereas firms when tacitly colluding merely observe each other and react independently, the mechanism allegedly differs if they opt for a practical cooperation. A private exchange of pricing information can serve as an example for the latter. The jurisprudence of the courts requires that such practical cooperation must be substituted ‘knowingly’ for the risks of competition for it to amount to a concerted practice.²¹ The concept, therefore, hinges on the inner sphere of the firms involved.

Against the afore, it is questionable whether autonomous algorithmic collusion is prohibited under the traditional enforcement paradigms. If the firms lack of knowledge or intent with respect to the fact that their computer agents pursue a collusive strategy, it cannot be said that these firms ‘knowingly’ substitute a practical cooperation for competition. Several authors, therefore, point to the risk that the cartel prohibition might stop short of preventing such outcomes. Ezrachi and Stucke argue that collusion achieved by machine learning systems can fall outside the scope of the cartel prohibition for ‘lack of evidence of an anticompetitive agreement or intent.’²² In a similar vein, Calvano and others conclude from their study²³:

From the standpoint of competition policy, these findings should probably ring an alarm bell. Today, the prevalent approach to tacit collusion is relatively lenient, in part because tacit collusion among human decision-makers is regarded as extremely difficult to achieve. While we have no direct comparative evidence for algorithms relative to humans, our results suggest that algorithmic collusion might not be that improbable. If this is so, then the advent of algorithmic pricing could well heighten the risk that tolerant antitrust policy will produce too many false negatives.

Some authors, therefore, highlight that the enforcement paradigms might warrant amendments to close such regulatory gaps.²⁴

IV. Approaches for Closing Legal Gaps

1. On the Idea of Personifying Algorithms

It is questionable whether the potential enforcement gap in antitrust²⁵ can be overcome by defining algorithms as ‘undertakings’. The notion of an undertaking in EU antitrust, indeed, is a very broad concept which has many facets and functions. As a working definition that applies to the most common types, an undertaking can be described as a combination of assets and people that act on a market governed by a management body or further representatives irrespective of the legal personality or corporate structure. For an undertaking to act, or to have knowledge or intent, it is the action, the knowledge, or the intent of the human beings representing it which is attributed to it. If a company manager knowingly enters into an exchange of sensitive pricing information with the manager of a rival, it can, therefore, be said that these ‘undertakings’ substituted a practical cooperation for the risks of competition.

Yet what substantive meaning would terminology such as a ‘practical cooperation’ or ‘knowingly’ have, if they were applied to an algorithm? The problem is that the legal terminology is coined on human interaction and cognition, so that it will be vastly deprived of its meaning if transferred to a computer system. How shall an action of an algorithm be identified as ‘knowingly’, as opposed to another action that is supposed to happen un-knowingly? In what way is it meaningful to consider the actions of an algorithm as a ‘practical cooperation’, as opposed to a mere intelligent adaption to information obtained by this algorithm on the market? To rely on such human concepts of cognition with respect to the regulation of algorithms will likely end up in semantic exercises with limited substance.

2. On the Idea of a Prohibition of Tacit Collusion

Another way of dealing with the problem could be in equating tacit collusion and explicit collusion.²⁶ This would mean that under antitrust law any collusive strategy would qualify as an illicit cooperation so that any further distinctions based on the inner sphere of the persons involved would become obsolete. Such view has been suggested in the past independently of the issue of algorithmic collusion. Notable proponents were Richard Posner in his earlier writings²⁷ (he has since changed his view²⁸), Richard Markovits²⁹, and more recently Louis Kaplow.³⁰

One might feel inclined to hold that such a view does not reconcile with the structure of the law. Yet it is questionable whether this counterargument would be very strong. The notion of concerted practices can possibly be construed in a way as to extend to cases in which the collusive outcome is based on a mechanism of observation and retaliation, which is characteristic for tacit collusion as it is for explicit collusion. As outlined earlier, both categories share the feature that they can be described as a non-cooperative game in terms of game theory. It is, therefore, rather a semantic issue whether some ways of engaging in such a non-cooperative strategy can be tagged as a ‘practical cooperation’ or not, while the underlying economic principles remain the same. Firms observe and react in ways that are deemed individually optimal irrespective of which words are used to describe this phenomenon. Especially in the grey area between typical cases of explicit cooperation on the one hand and tacit oligopoly conduct on the other, it becomes apparent how brittle the traditional concept of distinction is.³¹ Consider that even in cases that would usually be qualified as tacit collusion, firms cooperate in that they observe each other and react to the information they have obtained from observing each other. One might ask the question: In what way is this not a ‘practical cooperation’? Also, a collusive equilibrium can bear negatively on consumer welfare³² irrespective of the means and forms used by firms to sustain it. This seems to add to the argument that the distinction between tacit and explicit collusion is of limited expedience.

Yet still the idea of equating tacit collusion with explicit collusion faces severe objections, which might explain why it has not gained more recognition among scholars and enforcers.³³

Any law must provide the addressee the opportunity to abide by it through choosing a compliant course of action. Otherwise, the law would be perplexing. Recall, now, that collusion in the realm of antitrust is a non-cooperative game.³⁴ This means that the strategy is individually rational for each participant. Sanctioning collusion without more, therefore, would amount to prohibiting the pursuit of an individually rational strategy. Kaplow reacts to this objection by pointing out that it is a common feature of the legal order to prohibit types of conduct that are individually rational³⁵, such as the stealing of an apple.³⁶ By imposing a sanction, the law ensures that it becomes rational to refrain from that course of action, he advances.

This argument, however, is not able to overcome the conceptual problems that would arise if the law prohibited the collusive outcome as such. While it is perfectly clear what a person must do to ‘not steal an apple’, it is much less obvious what course of action a firm would have to take in order to ‘not pursue a collusive strategy’. The negatory of stealing an apple is unambiguous. The course of action in order to avoid the sanction is to not steal the apple. With a prohibition of a collusive market outcome, however, it would be much more difficult to describe, in an unambiguous way, what the compliant course of action would be. One of the reasons for this is that, from an ex ante perspective, it is unclear what outcome a collusive strategy among firms might produce. If that is not known, however, it is equally unclear what price a firm must set in order to not charge at a collusive level. If the firms do not know whether a collusive strategy would yield a market price of € 5 or rather of merely € 4, they cannot know ex ante whether individually charging a price of € 4 would be ‘non-collusive’ or not.

On a more philosophical level, another objection buoys. Effectively, the law would require the addressee to pursue any market strategy so long as it is not rational, assumed that the rational strategy would be collusion. It is questionable, however, whether an addressee of the law can be required to intentionally act irrationally. Any legal prohibition must conceive of the addressee as an intelligent entity, for if the addressee were not intelligent, it could not abide by the law in the first place. From a philosophical perspective, it is unclear, however, whether a person can ‘rationally act irrationally’. Can a firm be obliged to randomize prices in order to protect itself from being accused of acting rationally-collusive?

Yet even ignoring this fundamental problem and moreover assuming it were possible to define a hypothetical collusive price level ex ante, it would remain unclear whether a non-collusive strategy could be defined with a sufficient degree of precision so that firms could abide by the law. Would it suffice to undercut the hypothetical collusive price by, for example, 2%? Or would the law require every firm to price at marginal cost, for under perfect competition this would be the hypothetical competitive price, even though perfect competition usually does not exist? Merely imposing a sanction on the pursuit of a collusive strategy does not solve any of these conceptual problems.³⁷

3. Harmful Informational Signals As Point of Reference for Cartel Conduct

a. Conceptualization

Another conceptual venture to solve the issue could be to maintain a distinction between illicit cartel conduct and legitimate coordination, yet to substitute a new criterion for the traditional paradigm, viz. for the practical cooperation adage. Such alternative criterion of distinction could be checking the informational signals released by colluding firms for their propensity to create a net consumer harm. Accordingly, an illicit concerted practice would be found if firms, or the algorithms relied on by firms, released informational signals which reduced consumer rent if compared to a counterfactual in which such informational signals were absent. The counterfactual, therefore, would not be a hypothetical market without collusion. It would be the same market absent a particular informational signal.³⁸ To clarify the specifics of this approach, and to contrast it with the view expressed by Kaplow and others, the following explanations shall be made.

In contrast to a situation where any collusive equilibrium is prohibited, so that it is unclear which course of action to take in order to abide by the law, confining the prohibition to harmful signals leaves the addressee a binary choice that is conceptually clear: an informational signal that is ultimately harmful to consumers can either be released or not. There is no element of imposed irrationality in such a prohibition. To refrain from the release of a harmful informational signal can also mean that the addressee must choose to release a different signal in order to avoid a harmful effect. To exemplify the idea, reference can be made to the EU Commission’s remedy decision in Container Shipping, where public price announcements were not abandoned completely but limited in scope in order to avoid the creation of a consumer harm.³⁹

A relevant signal in this sense can be any release of market-related information independent of whether it takes place publicly or in private, whether consumers are involved or not. Even price lists, price announcements, or other types of public display of prices or other parameters can qualify as a signal that ultimately leads to a collusive equilibrium. In contrast to the aforementioned opinion of Kaplow, however, this would not suffice for the conclusion that a restriction of competition in terms of the cartel prohibition is established. Rather, the release of such informational signals would fall outside the scope of Article 101(1) TFEU to the extent that it produces offsetting consumer benefits.

This reflects the fact that there is a plethora of cases where public market information leads to collusive equilibria, although the benefit to consumers being derived from this information outweighs the gross harm of the collusive outcome if compared to a situation in which such informational signals were not made. The reason why, in most cases, public price lists and similar forms of information are not prohibited despite their risk to precipitate collusive equilibria lies in the fact that they usually bring about benefits to consumers that offset the potential harm.⁴⁰ If firms refrain from the release of such pricing signals, the distribution of goods might be significantly impeded. Consumers can face difficulties in planning their purchases ahead or in pursuing multisource strategies. It can be said, therefore, that sometimes ‘vertical transparency’ creates a greater benefit to consumers than what is taken away by the horizontal collusion that concomitantly results from it. It is necessary, therefore, to balance both effects in order to get a full picture of the economic impact of an informational signal. If such an analysis of the economic effects is integrated into the assessment of a concerted practice, this creates a system in which harm and benefit of a measure can be distinguished without the need to make recourse to notions such as the ‘practical cooperation’ adage or the intention of the firms involved.

This concept would allow an entity to deal with the phenomenon of autonomous algorithmic collusion under the cartel prohibition. Achieving or sustaining such a market outcome would be prohibited if and to the extent that the harm to consumers were greater than the benefits associated with the release of the underlying informational signals.⁴¹ If algorithms achieved a collusive equilibrium by communicating with each other and without this interaction providing any useful information to consumers, this could, therefore, constitute a concerted practice in terms of the law. If, on the other hand, a digital platform aggregated and provided information to consumers in a way that benefits them, and if the efficiency potential of this platform could not materialize without this release of information, Article 101(1) TFEU would not be triggered even though the conduct might, ultimately, give rise to a collusive equilibrium, if and to the extent that the benefit of the former offsets the harm of the latter. The counterfactual, therefore, would not be a digital platform without collusion. It would be a situation in which the informational signals were not released. If the platform, in such a counterfactual scenario, were not operated or operated with a lesser efficiency, consumers could be deprived of the benefits resulting from it. This could mean a smaller range of suppliers being visible to them, less information being available to help consumers plan their purchases, etc.

This demonstrates that a collusive outcome can be an inevitable consequence of an algorithm-based system that produces benefits, although horizontal restraints, even on prices, are involved. The decision of the Luxembourg competition agency has made this clear with respect to the taxi app Webtaxi.⁴² The agency found consumer benefits in the fact that the platform improved on the supplies with transportation services, even though a horizontal collusion on prices was an inevitable side effect. While the decision accounted for these efficiencies within the scope of an exemption from the cartel prohibition, the concept presented here would integrate this analysis already into the assessment of a concerted practice. As previously outlined, this is a consequence resulting from the substitution of an effects analysis for the less useful ‘practical cooperation adage’ in order to discriminate between legitimate and illicit collusion in algorithm cases.

b. Possible Objections

Such a concept of harmful informational signals, of course, provokes objections. They shall be dealt with in the remainder of this chapter. One might want to invoke that this construal of the law does not reconcile with the structure of the provision as shaped by the jurisprudence. Admittedly, the courts have not yet recognized an interpretation as suggested here. Rather, the Court of Justice, currently, relies on the notion of ‘practical cooperation’ in order to distinguish between concerted practices and tacit collusion. The established set of criteria does not contain a place for an economic effects assessment. On the other hand, the Court of Justice has not yet had an opportunity to hone the law with respect to the phenomenon of autonomous algorithmic collusion. It is, therefore, conceivable that the courts, upon preparatory enforcement steps done by the agencies, ultimately consider the option to readjust some of the enforcement paradigms in order to close potential regulatory gaps.

Also, it should be noted that even under the current decisional practice an effects analysis can be part of the assessment of Article 101(1) TFEU. As to the distinction between restrictions by effect and those by object, the potential of the measure to produce restrictive effects bears significance.⁴³ In that regard, the Court of Justice has made clear that, among other things, it must be analyzed whether and in what way consumers might suffer from the measure at stake.⁴⁴ This line of reasoning already mirrors an effects analysis based on the consumer welfare paradigm, which also is the backbone of the present proposal. In a similar way, the Commission practice demonstrates that the notion of a competitive restraint involves an analysis of the effects on consumer rent. Even though the Court of Justice has ruled that, for a restriction of competition to arise, it is unnecessary to demonstrate concrete consumer harm⁴⁵, the Commission takes this into account if it helps to discriminate between legitimate and illicit conduct. The Commission, for example, argues that even horizontal price fixing, depending on the structure and function of the cooperation, might warrant a close examination within the ‘by effect category’, which demonstrates that it is possible to conceive of cases where such conduct falls outside of the scope of Article 101(1) TFEU without any further examination of the legal exemption rule in Article 101(3) TFEU.⁴⁶

On a practical level, one might want to invoke that the assessment of whether an informational signal precipitates a net consumer harm or not is too complicated to rely on as an enforcement paradigm. Yet it must be noted that even the currently existing decisional practice is not void of elements of an effects analysis, as demonstrated previously. Beyond that, firms and enforcers face exactly these difficulties already within the realm of Article 101(3) TFEU, so that it could not be said that such difficulties are idiosyncratic to the proposal made here.

Beyond that, one might raise the question in what way firms can become responsible for the conduct of an algorithm, if the strategy pursued by the latter is unknown to the former. In legal terms, however, it is possible to hold someone responsible for the organization of an enterprise. Firms, therefore, could be considered obliged to terminate the use of an algorithm or to alter its paradigms, if and to the extent that it produces more harm than benefit to consumers. Firms could, therefore, be ordered, by way of an administrative decision, to make such adjustments to their business strategy either by tweaking the algorithm or by stopping its use altogether. Antitrust economists already expound ways to design platforms in a way that counters collusive risks emerging on it from machine learning systems. Justin Johnson, Andrew Rhodes, and Matthijs Wildenbeest published a study in 2020 on how a choice algorithm on a sales platform can impact the likeliness of collusion among independently acting q-learning algorithms.⁴⁷ It goes without saying that the imposition of a fine, or a liability for damages would, in any event, require negligence or intent on the firm’s part. That, in turn, would require the agency to establish that some degree of knowledge or intent with respect to the achievement of a collusive equilibrium can be established among the actual firms that relied on the algorithm. Such would be absent if the equilibrium were achieved or sustained independently by a machine learning process not guided or anticipated by the firms that rely on the outcome.⁴⁸ Yet still, in such an event a mere administrative order, absent any sanction or damages award, could be issued.

Finally, one might want to invoke that it would be disproportionate to make such far-reaching amendments to the construal of the cartel prohibition for the sole purpose of closing a regulatory lacuna with respect to algorithms. It is not the intention behind this proposal, however, to render the established enforcement paradigms obsolete in their entirety. Rather, the suggestion made here should be conceived of as a mere addition to the established principles that would still bear relevance in the majority of non-algorithmic collusion cases. A private exchange of pricing information between the managers of two rivals, for example, could still be considered a practical cooperation without further effects analysis required, for its potential to precipitate a consumer harm, as reflected in the Commission’s Horizontal Guidelines.⁴⁹ The informational signal approach suggested here, on the other hand, could become relevant if, in an algorithm case, the traditional criteria did not allow the application of the law in a meaningful way. The present suggestion, therefore, is meant as a humble contribution to existing paradigms, not as a postulate of a total substitution for them.

V. Conclusion

This article is a conceptual sketch of a way to deal with the intricacies coming along with autonomous algorithmic collusion. Such risk is being discussed, especially, with respect to q-learning algorithms. Even though practical cases have not yet emerged, there is sufficient reason to address potential issues as a precautionary measure at this point in the scholarly debate. It was the intention to demonstrate that the traditional construal of the law, which relies on a description of human behavior, appears inapt for effectively tackling machine-induced equilibria. Applying the established criteria in such cases would very likely lead to a harping on words without substance. There is simply no point in venturing to assess whether algorithms ‘practically cooperated’ as opposed to ‘merely observed each other’. The present proposal, therefore, is intended to serve as a conceptualization of the notion of concerted practices for cases that would otherwise elude the cartel prohibition. To conclude, the entire problem of autonomous algorithmic collusion is an example of the necessity for interdisciplinary research between lawyers, economists, and computer scientists. At the same time, the problem highlights how enforcement paradigms, that hinge on descriptions of the inner sphere and conduct of human beings, may collapse when applied to the effects precipitated by independent computer agents. The subject matter of this chapter is, therefore, an example for the greater challenges that the entire legal order faces in light of the progress of machine learning.

21 Artificial Intelligence in Financial Services New Risks and the Need for More Regulation?

Matthias Paul

*

I. Introduction

The financial services industry has been at the forefront of digitization and big data usage for decades. For the most part, data processing has been automized by information management systems. Not surprisingly, Artificial Intelligence (AI) applications, capturing the more intelligent ways of handling financial activities and information, have increasingly found their way into the financial services industry over the last years; from algorithmic trading, smart automized credit decisions, intelligent credit card fraud detection processes, personalized banking applications and even into areas like so-called robo-advisory services and quantitative investment and asset management more recently.¹

The financial industry has also been one of the most regulated industries in the world. In particular, since the collapse of Lehmann Brothers in 2008, leading into one of the most severe financial crises in history, regulation efforts of all kinds of finance-related activities and financial organizations as a whole by the different regulators around the world have significantly increased. In general, most regulations relating to the financial industry, in particular those put into place after the financial crisis in 2008, have focused on areas like safeguarding the financial institutions themselves, safeguarding the customers of financial institutions, and making sure the institutions comply with general laws overall and on a global scale, given the truly global nature of the financial industry.

More recently, authors have argued that with the emergence of AI-based applications in the financial industry, new kinds of risks have emerged that require additional regulations.² They have pointed for instance to increased data processing risk, cybersecurity risks, additional challenges to financial stability, and even to general ethical risks stemming from AI in financial services. Some regulators like the Monetary Authority of Singapore (MAS) have proposed an AI governance framework for financial institutions.³ The EU also explored this topic and published a report on big data risks for the financial sector, including AI, stressing appropriate control and monitoring mechanisms.⁴ Scholars have developed this topic further by adopting so-called personal responsibility frameworks to regulate any new emerging AI-based applications in the financial industry.⁵ In its recent draft regulation, the EU has presented a general risk-based regulatory approach of AI which regulates and even prohibits certain so-called high risk AI system; and some of them can supposedly also be found in the financial industry.⁶

This chapter will explore this entire topic of AI in the financial industry (which will also be referred to as robo-finance) further. One focus of the article will be on whether AI in the financial industry gives rise to new kinds of risks or merely increases existing risks already present in the industry. Further, the article will review one prominent general regulatory approach many scholars and regulators have put forward to limit or mitigate these alleged new risks, namely the so-called (personal) responsibility frameworks. In the final section of this chapter, a different proposal will be presented on how and to what extent best to regulate robo-finance, which will take up key elements and concepts from the recent Draft EU AIA.⁷ To lay the groundwork for the discussion of these topics, the nature of AI, in particular as a general-purpose technology, will be explored first. In addition, an overview of the current state of AI applications in financial services will be given, and the different regulatory layers or focus areas for regulations that are present in the financial industry today will be presented. Based on these introductory discussions, the main topics of the chapter can then be spelled out.

II. AI As a New General Purpose Technology

Electricity is a technology or technology domain which came into life more than 150 years ago, and it still drives a lot of change today. It comprises different concepts like electrical current, electrical charge, electric field, electromagnetics etc. which have led to many different application areas in their own right; from the light ball to electrical telegraphs or to electric engines, to mention only a few. It is fair to say that electricity as a technology field or domain has revolutionized the world in many ways, and it still does. And it has changed and transformed whole industries as it transforms the automotive industry with the transition from combustion engines to electric cars.

Given its wide range of underlying concepts with multiple specific application areas of their own right, several authors have referred to electricity as a general-purpose technology (GPT).⁸ What is characteristic of GPTs is that there exists a wide range of different use cases in different industries, thus GPTs are not use-case-specific or industry-specific technologies but have applications across industries and across many types of use cases. Other examples of GPTs which scholars have identified are the wheel, printing, the steam engine, and the combustion engine, to mention a few.⁹ As such, GPTs are seen as technologies that can have a wide-ranging impact on an entire economy and, therefore, have the potential to drastically alter societies through their impact on economic and social structures.¹⁰

Several authors have claimed or argued in recent years that AI can or should also be considered a GPT, ‘the most important one of our era’ in fact.¹¹ Or as Andrew Ng says: ‘AI will transform multiple industries’.¹² AI’s impact on societies as a whole is seen as significant, for instance, changing the way we work, the way we interact with each other and with artificial devices, how we drive, how wars might be conducted, etc. Further, like in the case of electricity, there are many different concepts underlying AI today, from classical logic or rule-based AI to machine learning and deep learning based AI, as employed so successfully today in many areas. Some hybrid applications combine both concepts.¹³ These concepts have allowed for many new types of AI applications, similar to the case of electricity, where different concepts have been merged together as well.

In fact, because the use cases for AI technologies are so enormous today, companies like Facebook have created their internal AI labs or what they have called their ‘AI workshop’ where many different applications of AI technologies, in particular machine learning applications, get explored and developed.¹⁴ The underlying assumption of such companies is that AI can be applied to so many different areas and tasks that they need to find good ways to leverage their technological expertise in all such different areas.

Clearly, AI is still in its early stages of technological development, with fewer implementations in widespread operation than in the case of electricity. But there have been language and speech processing applications, visual recognition applications like face recognition in smartphones, photo optimization algorithms in digital cameras, many kinds of big data analytics applications, etc. AI technologies have also changed the interface between humans and machines, some turn machines into helpful assistants, others allow for intelligent ways of automating processes and so on. The applications of AI are already widespread today, and we seem to be just at the beginning of a long journey of bringing more applications to life.¹⁵

In the following, we will look at the financial industry as one major application area for AI as a general-purpose technology. The financial industry is interesting in so far as it is heavily regulated on the one hand, but also highly digitalized and technologically advanced on the other hand, with many kinds of AI use cases operational already today.

III. Robo-Finance: From Automation to the Wide-Spread Use of AI Applications

The financial industry has been one of the most data-intensive and digitized industries for decades. In 1973, SWIFT was founded and launched, the so-called Society for Worldwide Interbank Financial Telecommunication, bringing together 239 banks from 15 countries worldwide with the aim of handling the communication of cross-border payments. The main components of the original system included a computer-based messaging platform and a standard message system.¹⁶ This system disrupted the manual processes of the past, and today more than 11,000 financial institutions from more than 200 countries are connected through SWIFT’s financial global technology infrastructure. Nasdaq, to give another example, the world’s first electronic stock exchange, began its operations even earlier, in 1971, leading the way to fully digitized exchanges for the trading of any kinds of financial securities, which are the standard and norm today. And real-time financial market data and news, probably the first big data sets used in history, were made available in the early 1980s by companies such as Thomson Reuters and Bloomberg through their market data feeds and terminal services.¹⁷

In the years to follow, the financial industry has been at the forefront of leveraging information (management) systems to manage and process the vast amounts of data and information available.¹⁸ In fact, today, many financial institutions resemble technology companies more than traditional banking houses, and it is no surprise that companies like Paypal or, more recently, many new fintech players were able to further transform this traditional industry by leveraging new technologies like the Internet or mobile services, platforms, and infrastructures.¹⁹

This development of digitizing financial information and financial transactions has made the automation of data handling and processing, not just a possibility but rather a necessity to maintain and defend one’s competitiveness and to deal with and manage the various kinds of risks inherent in the financial industry. The execution of payments within the international banking system or the execution of buying or selling orders on the exchanges can be fully automized today based on simple parameters (such as dates and amounts, or stop-loss orders to manage risk, etc.). Clearly, these ways of automizing financial transactions and processes are in no way intelligent, nevertheless they have helped the investment banks and other actors in the financial industry tremendously to increase process speed, accuracy, and also improve risk management.²⁰ Therefore, it is no surprise that many actors in the industry have constantly searched and tried to develop more sophisticated processes, which has opened the doors for AI applications in the financial industry.

Today, there is a wide range of AI applications present in the financial industry of which the following are just key application areas with multiple kinds of use cases:²¹

1. (1) Customer Related Processes:

   1. a. new ways of segmenting customers based on the use of so-called cluster algorithms or analyses,²²

   2. b. personalized banking services and offers based, for instance, on profiling algorithms,²³

   3. c. robo-advisory services replacing human financial advisory with machines,²⁴

   4. d. intelligent chatbots advising or providing information to clients in different areas of their financial decision making.²⁵

2. (2) Operations and Risk Management:

   1. a. underwriting automation in credit decisions and algorithmic credit scoring,²⁶

   2. b. automized stress testing.

3. (3) Trading and Investment Management:

   1. a. algorithmic trading – from simple rule-based AI to more sophisticated machine learning based algorithms,²⁷

   2. b. automatic portfolio rebalancing in asset management adjusting the portfolio to the predefined asset allocation scheme based on simple rule-based algorithms,

   3. c. big data and machine learning–based (assisted or fully automized) asset management.²⁸

4. (4) Payment Processes:

   • fraud detection algorithms in credit card payments using big data analytics and learning algorithms.²⁹

5. (5) Data Security and Cybersecurity:³⁰

   1. a. data security – algorithms protecting the data from inside a financial institution,

   2. b. cybersecurity – algorithms protecting the data from outside attacks.³¹

6. (6) General Regulatory Services and Compliance Requirements:³²

   1. a. Anti Money Laundering (AML) automation and protection algorithms helping to identify politically exposed people (so-called Peps) or criminals involved in certain financial transactions,

   2. b. detection of compliance breaches in case of insider trading etc.

As shown here, AI is already employed today in many areas of the financial industry, and new applications are emerging every day. The question is whether additional or increased risks stem from these applications, which might require additional regulations, as argued by some authors.³³ This line of argument will be reviewed in more detail in the following sections. But first, it is important to understand from a high-level perspective the main areas and layers of regulations in the financial industry today.

IV. A Short Overview of Regulation in the Financial Services Industry

The financial services industry is probably one of the most regulated industries. Regulation of trading practices for instance dates back to the seventeenth century when in 1610 in Holland, some first forms of short selling became prohibited.³⁴ At the same time, the first central banks were created, such as the Swedish Riksbank in 1668, to regulate payment transactions on a national level and establish national currencies by issuing banking notes. Some of the early regulation was ‘private self-regulation’, in other words, bottom up norm creation,³⁵ as in the case of regulatory practices around many of the emerging exchanges, but some regulation was already at these early times government- or state-driven (top down) as in the case of the establishment of central banks and their key role in establishing standardized payments practices based on backed up currencies.³⁶

Today the financial industry is heavily regulated by national or supranational bodies, for instance, by the ESMA³⁷ in the EU or by the SEC³⁸ in the US in regard to activities on the different financial markets. Some of the regulations are financial-industry-specific, others are general regulations that severely impact the financial industry. Overall, the different types or layers of regulations in the financial industry can be classified by their underlying aims, namely: (i) regulations meant to safeguard overall financial stability, (ii) regulations for the protection of consumers of financial services, and (iii) regulations that are meant to make sure financial services can operate in a challenging and diverse international environment with sometimes conflicting rules and principles.³⁹

The following overview tries to capture the main regulation areas or layers and their specific purpose or aim as they are present in the financial industry today. Some of the layers directly link up to the categories just mentioned, some are cutting across the different categories, and some are also mirroring the classification of the previous Section of AI-impacted application domains in the financial industry:

1. (1) Equity and liquidity requirements for banks and financial institutions to adhere to minimum capital ratios and liquid asset holdings to prevent financial stress, improve risk management, and promote transparency. Examples are the Basel I, Basel II, Basel III regulations which are global voluntary regulatory frameworks adhered to by most financial institutions today;⁴⁰

2. (2) Infrastructure regulations, many still in the proposal stage, to improve financial services firms’ operational resilience (in case of major disasters, for instance), and their responses to cyberattacks;⁴¹

3. (3) Pre- and post-trading regulations to strengthen investor protection and improve the functioning of financial markets, making them more efficient, resilient, and transparent like banning certain trading practices or making kickbacks by product issuers transparent. The MiFID I and II regulations in the EU are examples of such kinds of regulations;⁴²

4. (4) Payment services regulations, like the PSD II directive (2015) in the EU, with the aim of creating more integrated payments markets, making payments safer and more secure and also protecting consumers, for instance, from the financial damage resulting from fraudulent credit card payments;

5. (5) Various kinds of compliance regulations, for instance anti-money-laundering or terrorist financing regulations etc., to ensure that financial institutions obey the treaties and laws and do not enter any illegal transactions or practices, also regarding cross-border transactions;⁴³

6. (6) General data privacy protections like the GDPR⁴⁴ in the EU, which is highly relevant as financial transactions involve much sensitive personal data.

As we can see, there are no specific AI regulations of financial services – although many of the regulations will also impact AI-based financial services. In fact, there are even very few regulations regarding the underlying technologies in financial services, but most of them focus on the use cases or financial activities, processes, and on the outcomes themselves. Yet, recently some scholars and some regulators have argued that there might be new risks stemming from AI applications and technologies in financial services which require additional regulation. In the following, we will look at some of the alleged risks as pointed out by scholars in the field and explore to what extent they might be covered by the above regulations already or whether there is a need for new regulatory frameworks.

V. New Risk Categories in Robo-Finance Stemming from AI?

Dirk Zetzsche and others, in their recent paper, have identified the following four risk categories or risk areas allegedly related to AI applications in the financial industry:

1. (1) Data risks

2. (2) Cybersecurity risks

3. (3) Financial stability risks

4. (4) Ethical risks.⁴⁵

Although I agree with the authors that all these kinds of risks are related to AI applications in financial services, it appears that these risks already existed before the emergence of robo-finance, given the advanced stage of the industry in terms of digitization and data dependency and usage. In fact, some of these risks might even be reduced or vanish when AI comes into place. Let us look at the different risk areas one by one.

Firstly, starting with the data risks of AI applications, Zetzsche and others bring up the following more specific arguments: (i) Because the data quality might be poor, there can be deficiencies stemming from AI applications. As a matter of fact, data quality has often been poor in many parts of the financial industry, for instance, outages at the data centers of the exchanges or of the market data providers leading to the misstating of prices of securities, which can have negative effects on investors’ decisions and the markets overall. AI could actually be used to deal with the data issues in terms of detecting and even resolving them.⁴⁶ (ii) Besides, they argue that data used for AI analyses might suffer from biases, for instance, relating to what they call ‘oversight’ in a financial organization. Again, biases have already been influencing decision making in the financial industry even before the emergence of AI applications, maybe not in the form of what they call data-biases, but biases residing more generally in human decision, for instance in the making of credit decisions or consumer lending.⁴⁷ AI application might free us from certain biases by providing a more neutral stance if programmed accordingly or at least be sensitive to such kinds of biases. (iii) And it is claimed that AI interdependency can lead to what they call ‘herding’, for instance all systems selling securities triggered by certain market events, which can lead to what has been referred to as ‘flash crashes’.⁴⁸ Again ‘herding’ behavior has existed in the financial markets for a long time, and whether the emergence of AI in electronic trading systems has been the cause of what has been called ‘flash crashes’ seems rather questionable. Simple rule-based algorithms, which today by industry experts would rather not be classified as AI systems can give rise to such behavior in contrast to more sophisticated systems trained on historical data relating to such events.

Secondly, let us look at cybersecurity risks. Obviously, they have also existed before the arrival of AI, with most attacks initiated and conducted by human individuals directly or by simple processes, methods, or algorithms. Examples are emails carrying malware that, after it has installed itself on someone’s computer, can silently send all sorts of confidential data from the computer or computer network to the attacker; a similar case of phishing attacks through links to websites – for instance, of online banks that mimic the log-in pages one is familiar with; or finally, simply the reuse of a user’s credentials which the attackers have somehow got hold of – for instance, by one of the already mentioned measures or by simply spying on people in combination with our carelessness in setting passwords. That ‘algorithms can be manipulated in an effort to transfer wealth’ has nothing to do with the presence of AI systems because this could be done already before such systems were in place and it currently happens every day in many different ways within traditional information system environments.⁴⁹ It rather seems plausible that AI might provide some help in identifying and preventing cybersecurity attacks.⁵⁰ Many services are offered today in this regard, and this seems to be one of the areas where the financial industry could benefit from employing AI-based solutions and thereby reduce potentially harmful cybersecurity risks.

Thirdly, when Zetzsche and others talk about financial stability risks, it is fairly unclear what they have in mind since they mention almost all areas of AI applications in financial services – as laid out above – from consumer facing and supporting applications, to trading and portfolio management systems, to general regulatory and compliance systems. Overall, their main concern here seems to be the emergence of ‘additional third-party dependencies’ to AI technology providers up to the ‘level of oligopoly or monopoly’. Since many of these third-party technology providers are unregulated today, as they point out, and it might even be hard to regulate them as ‘AI-related expertise beyond those developing the AI is limited’, there appears to be a major risk. As they say, ‘these third-party dependencies […] could have systemic effects’.⁵¹

What can be said against this last part of their arguments is that many of the actors at the forefront of using AI in financial services today develop their applications inhouse, like the dominant hedge funds or algo trading shops set up by IT specialists.⁵² AI-based technology has become such a core asset these days and a competitive factor of financial services that many financial institutions resemble IT companies more and more today to keep all that knowledge inhouse or with high IT expertise inside the organizations to manage their IT service providers or outsourcing partners – far from being entirely dependent or in the hands of monopolistic or oligopolistic structured IT providers.⁵³ Hence, their worry about systemic effects stemming from such dependencies seems to be overstated, at least in certain critical banking areas. Moreover, in many instances there are quite a few technology providers that offer similar services to the financial industries, for instance market data providers which increasingly have started to use AI technologies to organize and manage the quality of their market data feeds.⁵⁴ Financial institutions, at least in critical areas like trading, often make use of different providers at the same time, which also helps them to reduce their third-party-dependencies. Furthermore, with higher education AI or machine learning programs popping up at many educational institutions around the world, new graduates are also increasingly being educated and trained in these key areas. Thus, knowledge is building up quickly and will also be more widely available, reducing the fear of there being a kind of ‘mystery science’ only a few people have access to and can take advantage of.

Finally, let us focus on what Zetzsche and others refer to as new ‘ethical risks’ stemming from AI applications in financial services. The starting point of their argument is that algorithms do not feel anything, nor do they have values which the authors equate with a lack of ethical foundation in AI-decision making. For instance, they point out that such ‘unethical’ AI systems might nudge people to purchase unsuitable financial products, which might further be facilitated by the fact humans would easily develop a higher level of trust in the AI-based systems because with them, human–machine communication can nowadays be quite sophisticated. What this ultimately can and will lead to is reputational risk for the financial institution employing such systems, for example, when people are driven to make the wrong financial decisions and this becomes public or will be reported in the media or brought up to the courts.

There are quite a few problems with this line of reasoning as there are many financial institutions that do not have much direct interaction with human consumers, like mutual funds, hedge funds, credit card companies, etc. Besides, it is also conceivable that AI systems can have an ethical foundation, for instance thinking of utilitarian approaches which are less focused on being able to feel anything or have values. Such aligned AI systems might still be able to calculate the best outcome for society as a whole. But the main counter argument seems to be that the financial industry has not been a role model for ethical behavior to start with. Quite to the contrary, over many decades, financial institutions have been prone to all kinds of ethical misconduct. Just to give a few examples: (i) consumers have been pushed by financial advisors, humans with feelings and values, employed by financial institutions, to buy financial products which were often not suitable or beneficial to them, yet by selling them, the advisors were able to boost their commission payments, and the financial institutions could thereby boost their profits;⁵⁵ (ii) insider trading has happened frequently,⁵⁶ (iii) market manipulation has occurred, for instance in the case of the Libor scandal, and many other examples in different areas of the financial industry.⁵⁷ Thus, it is far from clear why AI-based systems and processes would make the industry less ethical than it has been in the past. In fact, the case could be made that AI-based systems and processes might allow society to create and control financial institutions and make them less driven by greed but more by higher motives to bring benefits to consumers and install fairness within the systems.

But this line of reasoning might sound overly naïve, given how many actors in the financial industry have successfully used technology over the last decades to their advantage, and to the disadvantage of other actors. One example has been the area of high frequency trading and the so-called dark pools where ‘fast moving robot trading machines were front-running long term investors on exchanges’.⁵⁸ Darks pools are markets established by the financial actors themselves for trading securities outside of the exchanges, usually virtually unregulated. The benefits for the market actors were faster processing of orders with less or even no fees from the exchanges. But the real benefits for the involved high frequency trading firms were obviously financial: with their algorithms and high frequency trading infrastructures, they were able to read the directions markets were going, and being able to buy securities before the real investors could do it and then selling the securities back to them at a higher price only milliseconds after the initial orders by the investors had been made. This practice allowed them to make huge profits stealing away money from the long-term investors like pension funds, etc.

This is a very sophisticated version of an old, mostly considered illegal practice of so-called front-running – in other words, someone trading a stock or any other financial asset based on insider knowledge of a future transaction that is about to affect its price. One important point is that this practice has been around before the emergence of AI and the high frequency data processing infrastructures. But it needs to be acknowledged that the new technologies have allowed for a more sophisticated and harder-to-control form of front-running. Yet the problem here is not that the employed AI algorithms are unethical. The problem is that the actors have used the technologies in an unethical way which obviously needs to be prevented for the benefits of the wider investor community and for society as a whole. Again, this is not a new risk, but it shows that AI and technology can be an accelerator of existing risks inherent in the financial industry.

In sum, then, the arguments by Zetzsche and others that there are many new risks stemming from AI-based applications in financial services are not fully convincing. To the contrary, it is feasible that the employment of AI applications in the financial industry might provide a route of managing existing and inherent risks in a better way, or even being able to reduce or eliminate some of these risks.⁵⁹ But clearly, there are also cases like front-running based on algorithmic high frequency trading, where it seems obvious that through the employment of AI, existing inherent risks in the financial industry have increased and can cause additional damage. Therefore, it is also important to look at ways how such damages resulting from the use of the new technologies can be avoided. In this regard, in the following section one prominent regulatory approach, the so-called responsibility frameworks, will be discussed.

VI. Responsibility Frameworks As a Solution for Managing AI Risks in financial services?

In regulating the financial industry, many regulators have moved to so-called responsibility frameworks in recent years, like the EU’s EBA/ESMA guidelines or the FCA in the UK.⁶⁰ The proposed measures focus on personal managerial responsibility, for example, the personal responsibility of directors, senior management, and individual line managers. Initially, such frameworks were meant to be applied to mitigate the risks of financial services in general, but recently authors have argued that they can also be applied to the emerging AI-based processes in the financial industry.⁶¹

The responsibility-driven regulations by the EU, published by EBA and ESMA, focus mainly on the management bodies of financial institutions, in particular on their role in conducting their overall operational duties, but with a particular focus on risk management conduct. They are meant to ensure that a sound risk culture has been implemented in their respective organizations consistent with the individual risk profile and the overall business model of the institution. The UK’s senior management regulatory framework for financial institutions has evolved from the overall EU framework but it has strengthened the establishment of clear conduct rules for senior managers. These rules specify in more detail the steps necessary to ensure that the business of the financial institution is controlled effectively and is in compliance with existing regulatory frameworks. Also, requirements are made on the delegation of responsibilities and on the disclosure of relevant information for the regulators. Other States like the US or Singapore have issued similar guidelines.⁶²

Although these responsibility frameworks have been very general in nature, meant to capture all kinds of aspects of risk management in financial institutions, Zetzsche has argued that they will give us the right framework to address and manage any new risks stemming from AI applications in financial services. They write: ‘personal responsibility frameworks provide the basis for an appropriate system to address issues arising from AI in financial services’.⁶³ They have suggested the following three distinct instruments or measures for regulating activities related to the development and use of AI applications in financial services:

1. 1. AI Review Committees: the installation of AI review committees is meant to address what they call the information asymmetry as to the function and limits of an AI system, namely the problem that third party vendors or inhouse AI developers understand the algorithms far better than the financial institutions that acquire and use them, and the supervisors of the institutions. These committees are meant to augment decision making and should not ‘detract from the ultimate responsibility vested in management […] regarding AI governance.⁶⁴

2. 2. AI Due Diligence: mandatory AI due diligence should be put in place, which should be done prior to any AI employment and should include what they call “a full stock of all the characteristics of the AI […] in particular the mapping of the data set used by AI”, including an analysis of data gaps and data quality.⁶⁵

3. 3. AI Explainability: the explainability requirement is proposed to be necessary as a minimum standard ‘demanding that the function, limits and risks of AI can be explained to someone at a level of granularity that enables remanufacturing of the code’. And this someone ‘should be a member of the executive board responsible for the AI’.⁶⁶

Before we review this proposal, it is fair to mention that the authors themselves note a few limitations, of which I want to focus on the main one, namely the inability of their responsibility framework to control what they call ‘autonomous AI’. What they mean by this are cases in which developers lose control over self-learning AI, not understanding anymore what the algorithms are doing.⁶⁷ What they propose is the concept of being able to always switch off the AI (as a kind of human oversight) while the provided services would still be functioning. This seems, prima facie, to be a reasonable request, looking for instance at the example of self-learning AI application in payment fraud detection based on the analysis of large transaction data sets. The system might modify its outlier detection algorithm in a way which might force the financial institution to switch it off, maybe because fraudsters have fed the system with data to facilitate fraudulent transactions. In this setting, switching the AI system off would make sense, but the delivery of the basic payment services should not be impacted by this – for instance, in the case of a credit card company. Yet, there will be applications where such a switch-off mechanism might be more difficult to realize without causing any further damage, as in the case of trading financial securities where orders or transactions ‘might get lost’ by switching off applications.⁶⁸

Overall, I agree with the authors that their approach is important and should be part of any software-based technology development in financial services. In fact, many elements have been in place in the industry for years already, for instance in terms of regular due diligence audits of the financial services’ technology providers.⁶⁹ Hence, the financial industry is already prepared and experienced in conducting due diligence audits on a regular basis, and they do this frequently before the release of new technology and software systems or installations, irrespective of whether these systems would include AI technology or not.

Yet, on the other hand, the authors propose some specific requirements for their AI due diligence and also in regard to their explainability requirement for AI. As will be argued, these requirements are not entirely clear and potentially will also be hard to fulfill given the nature of many AI applications.

Firstly, they argue that any AI due diligence should comprise taking full stock of all the characteristics of the AI, ‘in particular the mapping of the data set used by AI, including an analysis of data gaps and data quality. It is not clear what is meant by ‘a full stock of all the characteristics of the AI’. Besides talking about the different functionalities of the AI system, they also seem to focus on the underlying data set used by it. The problem here is that data sets might be potentially infinite and/or not be fully determined at the outset. In the case of its employment, a self-learning AI might discover new data (sets), and as it is often the case with big data, there can be quality issues and gaps. What does this mean for the AI system: should it not be launched under such circumstances? Or is it just enough to be aware of such limitations?

Secondly, their explainability requirement seems even harder to deal with in the case of AI applications. Even in regard to existing non-AI applications, it is questionable whether this requirement can be met given the complexity of many software solutions in the financial sector with millions of lines of code and often old legacy systems.⁷⁰ In the case of AI-based application, the situation is even more complex because learning AI systems are less static but more dynamic in nature, which could mean that the system might even rewrite its code in the course of its operations. Making explainability a minimum standard in the sense defined above could be the end of many applications in financial services, not only AI-driven ones. But what might be argued for is that a reduced version of this principle can be applied, not to require the remanufacturing of the actual code but, at a minimum, the possibility for the functions, limits, and risks of the AI systems to be understood on a higher functional level.

Thus, to conclude this section, the so-called responsibility frameworks can provide a basis for limiting the risks of AI applications in financial services, given that new risks really emerge. In a way, they have been present in the financial industry before the rise of AI applications, such as regular due diligence audits of technical systems and key software applications. But they also have their limitations, in particular, when certain requirements are taken in a very strict sense, as it was the case above in regard to the discussed explainability requirement. Yet reducing such requirements to a lower level raises the question to what extent the risk stemming from any kinds of new potential high risk AI applications in financial services can be contained. A slightly different approach will be presented in the final section, which builds on the approach put forward recently by the EU.

VII. Standardization, High Risk AI Applications, and the New EU AI Regulation

As noted before in this chapter, AI, being a general-purpose technology, will impact not just one industry but will also have many different kinds of use cases in different industries. There are already a lot of AI applications in the financial industry today, as pointed out earlier, and more are being added constantly by different financial institutions, their IT service providers, and innovative fintech companies. Many of the solutions might be simple or fairly basic, like the use of face recognition as an identification method giving users access to their financial accounts or applications. The others will be more complex, like developing so-called robo advisors, with AI systems engaging with users in natural languages trying to understand their financial needs and giving them suitable financial advice.

What will be important going forward is that regarding some kinds of key AI applications – such as identification processes or human–machine interaction – there can and will be standards across industries which companies need to comply with, like there are standards and norms for the use of electricity irrespective of their specific domain of use.⁷¹ Looking at the example of AI systems intended to interact with humans, providers of such systems might require that they be transparent to the user, that they are not communicating with humans but with a machine. Such notification obligations can then become part of the standard for such human–machine interaction enabling AI systems.⁷²

Besides such general standard AI applications used across industries, there might also be ones very specific to certain industries like the financial industry which need to be dealt with outside of the model of standardization. In particular, when these specific applications give rise to higher or new risks, additional specific regulations might need to be put into place. For instance, there have been attempts to contain the risks of algorithmic trading applications in the financial industry, which can cause (and probably have already caused to some extent) significant financial damage in the form of leading markets to crash, thereby diminishing or blowing away investors’ money in the course of seconds.⁷³ Such flash crashes have been at the center of some debates over the last years, and regulators such as the EU have tried to contain this risk by putting additional obligations into place through the MiFID II framework as discussed earlier.⁷⁴

In its recent Draft EU AIA the EU has also distinguished between ‘high risk’ and non-high-risk (standard) AI applications.⁷⁵ The new proposed regulation starts with the assumption that AI applications are ultimately and potentially just tools to increase human well-being. Thus, the technology development of AI should not be hindered by any unnecessary constraints, but the rules should be balanced and proportionate. The regulation is centered on a ‘risk based regulatory approach, whereby legal intervention should be tailored to those concrete situations where there is a justified cause of concern’.⁷⁶ A key distinction is made between the so-called high risk AI systems for which special requirements and obligations then apply and other AI systems with much more limited requirements and obligations. The classification of AI systems as high-risk is thereby mainly based on their intended purpose and their harmful impact on health and safety and human rights. High risk systems are more or less identified in a two-step process, namely whether they can cause certain harms to protected goods or rights and by the severity of the harm caused and the probability of occurrence.

Given this approach, it seems obvious that there cannot be one final list of high-risk AI applications because the technology is still emerging and new applications are being launched every day. The EU acknowledges this as it lists in its Draft EU AIA only a limited number of high-risk AI applications (Annex III). Further, it allows the EU Commission to amend this list over time based on criteria spelled out in Article 7.

Interestingly, many of the high-risk applications listed by the Draft EU AIA are not specific to one industry but are general AI applications that can be present in many industries. Examples are applications that embody what is called ‘manipulative AI practices’ and a second group with ‘indiscriminate surveillance’ practices. But there are also many very specific high risk AI applications listed in the draft. When it comes to high-risk AI applications in financial services, the EU draft of the regulation lists, prima facie, only one class, namely AI systems that evaluate the creditworthiness of persons (Annex III No 5 lit. b).⁷⁷ This class of applications is included in the high-risk list because of (i) possible discrimination of persons of certain ethnic or racial origin based on the potential perpetuation of historical patterns by the AI algorithms, and (ii) the potential severity of such acts of discrimination, as in the way such discriminating credit decisions can significantly affect the course of life of people.⁷⁸

The second kind of AI application that can be associated with the financial services industry, listed in the Draft EU AIA, is the one written about above, namely AI systems intended to interact with natural persons or generate content consumed by such person. Such systems do not necessarily classify as high-risk systems, for instance they might just help someone to enter information or explain a product, but they pose the specific risk of impersonation and deception, and therefore they are subject to specific transparency obligations according to the Draft EU AIA that means that the natural persons have to be informed that they are interacting with an AI system (Article 52).

Overall, the risk-based regulatory approach which underpins the Draft EU AIA makes much more sense than any kind of generalized approach of regulating AI applications as a whole as embodied in the responsibility frameworks discussed above. As a general-purpose technology, there will be so many kinds of applications that not one standard set of rules can be applied across the board. A rigorous case by case approach is required, which also allows for amendments and revisions, as embodied in the outline of the EU regulation.

VIII. Conclusion

What has been shown in this chapter is first, that it is questionable whether there are many new additional risks stemming from AI applications in financial services today. The risks that have emerged recently, like data risks, cybersecurity risks, financial stability risks, and ethical risks have been inherent in the financial industry as a highly digitized and also complex global industry for decades. The author has taken the more positive view that by using AI these risks will not necessarily increase, but on the contrary, AI might help to mitigate and reduce them. Second, the responsibility frameworks as developed over the last few years, which are meant to deal with and limit the risks of AI in the financial industry overall, do not provide a suitable framework beyond what has been put in place already to manage the risks with more standard IT and software systems and applications in the financial industry. Furthermore, overseeing all AI applications in financial services will quickly become as complex as overseeing all types of applications in the area of electricity, to mention another general-purpose technology. What has been argued in this chapter is that for some kinds of key applications – like identification processes or human–machine interaction – there should be standards defined across industries with which companies need to comply. But for other very specific, potentially new high-risk financial AI applications, in case they emerge, there might be the need for additional very specific regulation, as in the case of certain algorithmic high frequency trading applications. But this will be less a regulation of the technology but more of the practices and intended uses of the technology, which has also been the core thinking underlying the recent Draft EU AIA of AI applications. In fact, this new EU regulation, like the GDPR a few years ago in regard to data privacy protection, in many ways points to the right direction of how to deal with AI and potential risks arising from it.