Operational Risk Working Party - Validating Operational Risk Models

Abstract

Operational Risk is one of the most difficult risks to model. It is a large and diverse category covering anything from cyber losses to mis-selling fines; and from processing errors to HR issues. Data is usually lacking, particularly for low frequency, high impact losses, and consequently there can be a heavy reliance on expert judgement. This paper seeks to help actuaries and other risk professionals tasked with the challenge of validating models of operational risks. It covers the loss distribution and scenario-based approaches most commonly used to model operational risks, as well as Bayesian Networks. It aims to give a comprehensive yet practical guide to how one may validate each of these and provide assurance that the model is appropriate for a firm’s operational risk profile.

1. Introduction

The Operational Risk Working Party aims to assist actuaries and others in the modelling and management of operational risk. One challenge faced may be to validate models of operational risk as either internal or external reviewers. This paper sets out points which actuaries and other risk professional may wish to consider in validating such models, both in general and with respect to the following types of operational risk model:

• Loss distributions approaches (LDA)

• Scenario-based approaches (SBA)

• Causal factor-based approaches such as Bayesian Networks (BNs).

It also considers the validation of methods for aggregating different types of operational loss allowing for diversification, and the allocation of aggregate figures by risk type and legal entity.

The paper focuses on financial services companies and in particular insurers, banks and asset managers, but it is hoped this also has wider relevance to other firms.

2. General

In validating operational risk models, the validator will need to first understand the design and structure of the model, its operation in practice, and how it fits in with other risk models. The following points should be considered:

1. (a) Operational Risk taxonomy/definition: there should be a clear articulation of what is classed as operational risk – and what is not classed as operational risk – and how the model addresses the former. Without a clear taxonomy, operational risks may not be captured by the model and/or the model may make allowance for non-operational risks already modelled.

2. (b) Operational Risk profile and changes to this: validators should seek to gain a reasonable understanding of the organisation’s current operational risk profile from regular risk reporting, including any changes to this. This will help the validator to understand what the key operational risk types are, and where the validation should focus particular attention, as well as recent or upcoming changes to risk profile (e.g. new distribution channels) which may not be picked up by modelling.

3. (c) Model choice: what is the rationale behind the operational risk model chosen (e.g. LDA)?

   1. i. This should include an articulation of what the purpose of the model is: for financial services firms, the model will typically be developed with a view to assessing capital requirements for operational risks, but models could also be developed for wider management purposes, e.g., to model the potential impact of cyber-attacks.

   2. ii. The rationale should be clear as to why a particular approach was chosen over other approaches, having regard to model purpose.

   3. iii. Parsimony: validators should always consider whether a model is overly complex for the uses intended, but particularly for operational risks where issues with data often mean that model complexity is spurious.

4. (d) Governance: validators should consider the governance around operational and other models. Amongst other things, this should include:

   1. i. Model risk policy and standards setting out how models should be developed, validated, approved, and reviewed, as well as the roles of different stakeholders

   2. ii. Assessments of model limitations and risks, with logs kept of limitations, key expert judgements required, and planned model developments to address limitations

   3. iii. A regular model oversight committee to ensure models are regularly reviewed and revised where appropriate, and that model risk policy and standards are being adhered to.

   4. A proper model governance framework does not necessarily mean an operational risk model is fit-for-purpose, but the absence of such a framework makes it more likely the model will be flawed. The Institute and Faculty of Actuaries (IFoA) Model Risk Working Party has produced several papers on model risk, how this can be mitigated, and model governance ¹ . The IFoA has also produced a paper (Ashcroft et al., 2015) on expert judgement and the framework for using such judgements.

5. (e) Model use and integration with wider risk and capital management:

   1. i. For UK and EU insurers looking to use an operational model as part of a wider internal model, validators will need to consider how the operational model satisfies the use test requirement in Article 120 of the Solvency II Directive ² .

   2. ii. More generally, validators should seek evidence for how the model is used in practice, and whether it is being used for its intended purposes. If a model is not being used for its intended purposes and/or is not embedded in the risk and capital management system of a firm, then a question arises as to how serious the modelling effort is – if the firm isn’t prepared to use model results in practice, it calls into question whether model results are suitable for, say, determining regulatory capital requirements.

   3. iii. Alternatively, if the model is being used for purposes beyond its original scope there is a risk it may be misused, raising questions as to model governance.

   4. iv. Note that when it comes to the use of operational risk models, the journey may be as important as the end result: more may be gained from the use of loss data or scenario analysis in business as usual (BAU) risk management than the capital figures arising from these inputs to models.

   5. v. Model use is not a one-way street: there should be evidence of feedback from BAU risk and capital management as to how well the model is performing for its intended purpose, and this feedback should be fed into a programme of continuous model improvement.

6. (f) Documentation: validators should consider whether documentation is of a standard such that they or another independent knowledgeable third party would be able to understand the design, structure and operation of the model and how it interacts with other models. To the extent a validator is not able to fully rely on documentation but must query points further with developers, this would indicate that model documentation is unsatisfactory and as such does not comply with the requirements of Article 125 of the Solvency II Directive ³ .

7. More generally, poor documentation is often the sign of poor model development practices; this increases key person risk in cases where knowledge is missing in documentation; and makes it more likely the model will be misunderstood and misused.

8. (g) Culture: validators should seek to understand the culture of the organisation and how this influences modelling, loss reporting and risk management. For instance, CFOs and CEOs may push for lower capital requirements so they can improve dividends, and this could introduce a downward bias to subjective assumptions and hence model results.

9. Another aspect of culture is the extent to which staff own up to mistakes and weaknesses. A fearful corporate culture may inhibit the reporting of operational losses or weaknesses in controls, resulting in model results which understate the true operational risk profile.

10. It is acknowledged that culture can be difficult to define, let alone measure, but validators should try to assess the firm’s culture and the extent to which this might affect operational risk management and modelling.

11. (h) Benchmarking: finally, ORIC International ⁴ , ORX ⁵ , KPMG ⁶ , EY and others produce regular modelling surveys including operational risks which can be useful to benchmark methodology and aggregate results.

3. Loss Distribution Approach (LDA) Models

LDA models are based on fitting statistical distributions to internal and external loss data. Validation needs to consider first the loss data used, and then the results from distributions fitted.

3.1. Loss Data

The Operational Risk Working Party has produced a paper (Kelliher et al., 2016) on inputs to operational risk models, which addresses loss data and which should be considered in validating loss data, but in essence loss data should be assessed for:

• Accuracy: are loss figures correct? What controls are in place to ensure accuracy of loss figures?

• Completeness: how far back do they go? How significant are events not in data (ENID)? Do they miss notable periods of stress? Do they fail to capture key elements of loss?

  • ○ Are there any shifts in loss experience which may call into question the relevance of prior losses? ⁷

  • ○ Are there any outliers in data which should be excluded? Or conversely any data points excluded as outliers which should be included in loss data?

• Appropriateness: are historical losses relevant to the firm’s current risk profile (noting this includes legacy exposure)?

Key points which validators may wish to consider are detailed in the subsections that not follow.

3.1.1. Data governance and quality assurance

1. (a) Is there a documented process in place for collecting operational loss data? Lack of such a process makes it more likely that loss data will be inconsistent, incomplete and/or corrupted.

2. (b) Do second line risk management and/or internal audit review loss data? This would give some comfort as to accuracy, completeness, and appropriateness of data.

3. (c) Does the senior manager provide attestation to data on losses arising from their area? Again, this would provide some comfort on the accuracy, completeness, and appropriateness of data, but if senior managers don’t sign-off, there is a risk their staff may not pay due attention to recording losses accurately.

3.1.2. Losses captured

1. (a) Does loss data include “near misses” or events which give rise to gains (e.g. dealing errors where the market moves in a firm’s favour)? Excluding these could ignore significant information relating to risk events.

2. (b) Are these still relevant? For example, legacy mis-selling losses may no longer be relevant if a firm is no longer involved in providing advice.

3. (c) Even if relevant, there may be a case to adjust loss amounts for inflation. There may also be a case to scale losses with an exposure measure such as the number of customers.

4. (d) Does loss data capture impacts on the present value of future profits (“PVFP”) such as caps imposed on future charges? These would be relevant for life insurers as they would affect Solvency II Own Funds and so should be included in loss figures.

5. (e) Assessments of economic capital typically exclude new business and, if so, loss data should exclude new business impacts of operational risk events.

6. (f) Similarly, some impacts such as lapses may be covered elsewhere in the economic assessment so should similarly be excluded.

7. (g) Loss data might include costs such as management time, which may be part of BAU costs as opposed to the marginal costs arising from the operational loss event. Such BAU costs should be excluded for the purposes of economic capital assessments.

8. (h) Validators should consider whether operational loss data includes boundary events which may be captured under other risk models, e.g., where an operational risk event gives rise to a credit loss, as this may be captured under credit risk, or where it affects insurance claims, as it may be implicitly allowed for as part of insurance risk. That said, where operational losses are assumed to be covered under another category, validators should confirm this assumption.

9. (i) For insurers, some operational risk losses might be captured as part of expense analysis and so might be implicitly allowed for as part of base maintenance expense assumption and expense risk capital. If this is the case, then these losses should be excluded from LDA data to prevent double counting.

10. However, the validator should verify this by reviewing the expense analysis. Sometimes, operational losses which are assumed to be implicit in expense assumptions (e.g. minor customer complaints compensation payments) may in fact be deliberately excluded from the expense analysis. They will thus be excluded from maintenance expense assumptions based on this analysis, and from expense risk capital based on stresses to these expense assumptions.

11. (j) Recurring losses: to the extent losses may be recurring, there may be a need to consider whether these should be included in base maintenance expense assumptions and budgets as opposed to being covered as part of operational risk capital, in which case such losses should be excluded from LDA data.

12. (k) Recoveries: best practice would be to model losses on a gross basis in the first instance before insurance and other recoveries, as modelling net losses may implicitly extrapolate recoveries beyond sum insured limits; and insurance arrangements may change.

13. (l) A variation of insurance recoveries would be where losses are recovered from third party suppliers or passed on to customers. For the former, there may be limits to the indemnity provided by the supplier, or the ability to charge back losses may be disputed; while for the latter there would be regulatory rules around treating customers fairly and other constraints. For these reasons, loss data should ideally be modelled on a gross basis in the first instance to avoid recoveries being implicitly extrapolated beyond recovery limits in the model.

3.1.3. External loss data

Where a firm uses external operational loss data, validators should consider the following:

1. (a) How are losses scaled for the firm’s own size? What might be the impact of alternative methods of scaling on LDA results?

2. (b) Linked to this, are any adjustments made to external losses to reflect risk profile? For instance, a mortgage lender may not have the same exposure to dealing errors as an investment bank.

3. (c) Are losses relevant to a firm’s operations? For instance, unit-linked pricing errors would not be relevant to an annuity writer.

4. (d) Could similar losses arise given the firm’s control environment? For instance, a cyber loss arising at a peer due to the use of unsupported software may not be relevant to a firm if it keeps software up to date ⁸ .

Ideally there would be second line risk management and/or internal audit review and challenge of external loss data points included.

3.1.4. Risk coverage/ENID

There is a risk that historical data may not capture the tail of operational risk loss events for a category and/or material operational risks. Thus, the validator should consider:

1. (a) What is the distribution of losses by risk type? Are there any categories in the firm’s operational risk taxonomy with no data? Ideally there would be sufficient data to calibrate models for each Basel Level 2 risk category or the equivalent under the firm’s own risk taxonomy ⁹ .

2. (b) What is the length and size of the dataset? How many data points does it contain? How far does it go back? Could it omit any notable losses before data collections started, e.g., mis-selling losses? Or are there too few points to properly understand and model the tail?

3. (c) Are there any large losses deliberately omitted from data? If so, what is the rationale for this? If these could plausibly recur then they should be included in scope.

4. (d) If external data is not being included in the LDA, validators should consider external loss events and whether there are any types of loss types which are not captured by internal loss data.

5. (e) To the extent that losses below a certain threshold may be excluded, there is a need to understand the scale of these, lest in aggregate they are significant.

6. This could perhaps be assessed by comparing recorded operational loss totals with any accounting figures for total operational loss, or perhaps unexplained variances from budget.

7. To the extent that there are significant differences, consideration should be given to an add-on to operational risk capital to allow for these. Note, however, that these low-level losses may be included in expense analyses and so may be implicitly allowed for already in expense assumptions.

3.2. Distribution Fitting and Results

Having validated data inputs, the next stage will be to validate the approach to fitting distributions and the results from distributions fitted.

3.2.1. Distribution fitting methodology

1. (a) Frequency/Severity: a key model choice is whether to fit distributions to operational loss frequency and severity separately or just fit a single distribution to combined loss data. Most banks and insurers adopt the former approach ¹⁰ , so the choice of the latter is something that will need to be explained.

2. For separate frequency/severity models, the same severity distribution is usually assumed for all loss events, but the validator should challenge the reasonableness of this. It may be that the second and subsequent events are more severe than the first. For example, if the firm was less resilient as a result of a loss event, or subsequent losses could be less severe (where the first event reduced exposure to subsequent losses). Where there is a significant probability of 2 or more loss events arising, the validator should question the assumption of the same severity distribution for all events.

3. (b) Choice of distributions to be fitted: the validator should review the rationale for distributions chosen to be fitted to loss data and whether there are plausible alternative distributions ¹¹ which could be also used but which were not considered. Ideally, the sensitivity of results to these plausible alternatives should be quantified.

4. Often a compound severity distribution may be fitted, e.g., a Lognormal distribution may be fitted to the body of losses, with a Pareto distribution fitted to the tail. The validator should review the loss threshold at which the tail distribution is fitted, including the rationale for the loss threshold chosen and the sensitivity of results to alternative thresholds.

5. (c) Approach to parameter estimation: method of maximum likelihood versus method of moments – the latter may give rise to biased estimates but may be simpler to implement.

6. (d) Granularity: there will be a trade-off between (a) number of risks modelled and the homogeneity of each risk category; and (b) volume of data in each cell. Too many cells may result in insufficient data to credibly fit distributions, but validators should be aware that too few cells could mean a distribution is fitted for heterogeneous risks, akin to fitting a single distribution to motor and property claims in general insurance. Another issue is how granular data is, e.g., loss data may not be broken beyond the equivalent of the Basel Level 1 categories.

7. (e) Treatment of extreme losses: are these excluded from fit? If so, there needs to be a valid rationale exclusion.

8. Alternatively, if extreme loss events are included in loss data, assumptions may be made as to the percentiles these constitute in fitted distributions. The validator should identify any such assumptions; ensure there is a rationale for the choice of percentile; and assess the sensitivity of results to alternative assumptions.

9. (f) Maximum loss caps: caps on losses modelled need to be justified in terms of boundary constraints for losses such as portfolio size, other measures of total exposure (e.g. total number of staff per location) and legal boundaries such as time-bars.

3.2.2. Model results

In reviewing results, the validator should consider:

1. (a) Goodness of fit: a wide range of goodness of fit statistics should be produced as part of the fitting process including amongst other things:

   1. i. Kolmogorov–Smirnov (K-S) test scores

   2. ii. Akaike information criterion (AIC) and/or Bayesian information criterion (BIC) scores

   3. iii. Q-Q plots.

2. These should support the choice of distributions fitted, though validators should also be wary of over-fitting complex distributions to immature loss datasets.

3. Testing goodness of fit might also include tests for unimodality which assess whether distributions with a single peak value – like most distributions typically fitted – are appropriate for the data, or whether a bimodal distribution would be better. Tests for modality include Hartigans’ dip test (Hartigan & Hartigan, 1985).

4. (b) Sensitivity analysis: amongst other things this should consider the sensitivity of modelled results to:

   1. i. Different distributions

   2. ii. Different assumptions, e.g., percentile corresponding to historical extreme loss events or threshold for fitting separate tail severity distributions

   3. iii. Adjusted data sets, e.g., excluding extreme loss events (if included in data), the impact of excluding new data, or different scaling approaches to external data.

   4. Validators should challenge models which are very sensitive to small changes in data, while the sensitivity results to different distributions and assumptions are something that should be clearly signposted in model documentation and in communication of results to model users.

5. (c) Simulation error: where separate frequency and severity distributions are fitted, the combined distribution of losses will usually be derived through simulation, but validators should seek to establish the extent of simulation error around results. In particular, the number of simulations chosen should be supported by convergence testing.

6. (d) Kurtosis: immature datasets with few extreme loss events may give rise to thin-tailed severity distributions which may not be fit-for-purpose where there is the possibility of extreme losses under a category.

7. (e) Modelled recoveries: the validator should ensure these are in line with maximum sums insured and indemnity limits and/or consistent with what is possible to charge back to customers under consumer protection legislation.

   1. i. Validation should assess the possibility that losses could arise because of uninsured perils, or that there could be a dispute over coverage. It should also consider the possibility of disputes over liability with third party suppliers which could limit recoveries under indemnities.

   2. ii. For losses charged back to policyholders, there is a need to consider limits such as regulatory rules on treating customers fairly or, for a with profits or participatory fund, its Principles and Practices of Financial Management (PPFM) or equivalent. For with profits, any modelled chargeback should be put to the With Profit Committee for their views as to whether this would be acceptable.

4. Scenario-Based Approach (SBA) models

LDA models are common in banks in part because the Basel II Advanced Measurement Approach (AMA) was built on such an approach, but also because banks often have large amounts of operational loss data. However, insurers typically have less loss data to work with, so the predominant approach is to model operational risk using a scenario-based approach (SBA) where models are based on loss scenarios derived by expert judgement, or a hybrid approach combining elements of LDA and SBA ¹² . From ORIC’s 2020 Capital Benchmarking Survey of insurers and asset managers, 70% of respondents used SBA while the remainder used a hybrid approach ¹³ .

4.1. Scenario analysis results

Scenario-based approaches are driven by the judgements of subject matter experts (SMEs) and are inherently subjective. However, validators should ensure that scenarios are assessed in a structured fashion, as free as possible from bias. The Operational Risk Working Party’s paper on inputs to operational risk models (Kelliher et al., 2016) sets out good practice for scenario analysis in this regard. Taking this into account, key points that should be considered in validating SBA approaches are detailed in sections 4.1.1–4.1.9.

4.1.1. Risk coverage

SBA will typically focus on a limited number of scenarios, and validators should consider if this set of scenarios adequately captures the range of operational risks to which a firm may be exposed:

1. (a) Granularity: often risks will be considered by the equivalent of Basel Level 2 risk type which should ensure at least every Level 2 category is covered, but if not there is a need to map scenarios allocated to Level 2 category to ensure no such risk type is excluded from scenario analysis.

2. (b) While it would be impractical to consider all lower level (Level 3) sub-risks, there should be evidence that these have been considered as part of the process for identifying representative scenarios for a particular risk category. This might for example include detailed breakdown of risks by sub-type as part of background material to be considered by those involved in scenario analysis, and minutes of scenario workshops considering these sub-types.

3. (c) The Working Party is of the view that scenario analysis carried out just for the seven Basel Level 1 high-level categories (or their equivalent) is unlikely to give a suitably broad coverage of risks.

4. (d) Scenario participants should also be asked to consider internal and external loss examples to ensure no notable operational risks are missed.

5. (e) Scenarios considered but discarded before arriving at final scenarios should be captured as part of scenario documentation to provide evidence they have been considered. Validators should consider if there is bias in scenarios chosen compared to those discarded, e.g., is there a focus on more common but lower impact scenarios? Or vice versa on less likely, but higher impact scenarios?

6. (f) Often scenario analysis will be based on previous years’ scenarios but with scenarios refreshed but these is a risk this might miss new risks emerging so there is a need to consider how well the scenario process covered changes in risk profile and notable new losses and emerging risks (e.g. increase in home-working with Covid-19).

4.1.2. Scenario analysis process

Given the subjective nature of SBA, it is important that the process for arriving at scenarios is robust, so validators should consider the following:

1. (a) How is scenario analysis performed? Is it through workshops or some other process for eliciting SME views? One such approach is the Delphi method ¹⁴ where SMEs may be asked first to provide their responses individually rather than in a workshop. This can reduce the impact of certain biases, such as group think.

2. (b) How are workshops moderated? There is a risk that these can be dominated by an individual and their concerns, to the exclusion of other valid scenarios.

3. (c) How well are workshops minuted? Can the validator get a sense for the points discussed? And for differences in opinion between SMEs?

4. (d) Linked to this, is there any evidence of bias in discussions? For example, is there a focus on recent events, and/or a lack of consideration for events which have not occurred yet.

5. (e) What follow-up is carried out? A common weakness is where loss estimates are arrived at during a workshop without follow-up investigation to firm up loss estimates.

4.1.3. SMEs involved

The quality of SBA depends on the quality of SMEs contributing to the development of scenarios, so validators should consider how these were chosen – do they represent a suitable breadth of expertise across the organisation? This should include consideration of their level of seniority, length of service and relevant qualifications.

4.1.4. Quality of background information provided to SMEs

The quality of scenario outputs will be linked to the quality of background information supplied to SMEs. Validators should review the material considered by SMEs before arriving at scenarios. Ideally this should include:

1. (a) Definition of operational risk to be considered: needed to ensure clarity in terms of the scope of the operational risk being assessed and ensure that all sub-risks are considered as above, but also that there is no duplication of assessments.

2. (b) Relevant historical losses: both internal and external, and ideally including “near misses” and gains.

3. (c) Details of risk control assessments and any current risk and control issues.

4. (d) Relevant exposure details, e.g., current sales by product and channel for mis-selling risk, or the number of employees for employee relations risk.

5. (e) Details of strategy and plans which may affect exposure, e.g., planned investment in controls and system upgrades, outsourcing initiatives, or planned new products (especially if these new products are complex or involve new distribution channels).

4.1.5. Scenario outputs – general

1. (a) Scenarios chosen and their associated frequency and loss estimates may reflect control failures. Scenario analysis outputs should be clear on what controls, if any, are assumed to fail. The validator should consider whether sufficient allowance is made for control failures in scenarios or if too much faith is being placed in controls with little consideration for the impact of failure.

2. (b) For a frequency/severity approach where conditional loss estimates are sought, i.e., loss assuming a loss event has occurred, a common issue is that SMEs confuse this with the unconditional probabilities. By way of example, we may have a scenario with a 1-in-5-year frequency and a 1-in-20 conditional loss estimate. The latter should correspond to a 1-in-100 loss, but too often SMEs seek to articulate the loss at a 1-in-20 level. Validators should be aware of the potential for confusion in this instance.

3. (c) Where we seek a typical loss and a more extreme loss estimate, it is common for the latter figure to be based on the typical loss event, albeit of greater severity, but it may be better to consider a different scenario for the latter, e.g., for a cyber risk scenario, the typical loss could be based on a data breach, but the extreme loss could encompass not just a data breach but also a ransomware attack.

4.1.6. Scenario outputs – frequency

Typically, a frequency/severity approach will be adopted with scenario analysis seeking to arrive at parameters for the frequency of loss events and several estimates for the loss given an event has occurred. For the former, it may help to ask SMEs to choose from a limited range of frequencies (e.g. once in every 2/5/10/20/40 years) to help ensure consistency between assessments and avoid spurious estimates.

The frequency parameter could be validated against historical loss event frequencies but for high impact, low probability risks, a firm’s own experience is typically not long enough for such risks to crystallise. Even where an event has arisen, it can be difficult to judge frequency based on a single data point.

Another issue to be aware of is whether the frequency parameter relates to the likelihood of a material loss event arising in a particular risk category, reflecting all sub-risks, or whether it just relates to the sub-risks covered by loss scenarios. If the latter, consideration needs to be given to how well other sub-risks are covered by modelling – it may be that the scenario addresses low frequency, high impact loss events but that high frequency, low impacts may need to be separately addressed (see section 4.2.1 below).

4.1.7. Scenario outputs – loss quantification

Typically, two or more loss estimates will be derived to inform the severity distribution. The quality of these loss estimates is often an issue, particularly where these are arrived at in a workshop and not as part of a follow-up analysis.

The validator should assess:

1. (a) Loss elements assessed: validators should check whether certain types of loss are missed, e.g., the impact on PVFP if a scenario impacts future charge or premium income, which should be included in loss estimates as this may affect Own Funds. Validators should also consider if there are loss elements such as new business impacts which may not be relevant to an economic capital assessment; or certain costs such as management time which may be part of BAU costs as opposed to the marginal costs due to the operational loss.

2. (b) Boundary losses: as for LDA, there is a need for validators to consider whether scenario losses relating to credit losses or insurance claims could already be covered as part of credit and insurance risk capital already.

3. (c) Quality of data inputs used in assessing loss estimates, e.g., validators should check the source and veracity of portfolio details used to assess exposure to cyber-attacks.

4. (d) Assumptions including research used to set these, e.g., for data theft, there are several studies published ¹⁵ which could be used to set loss assumptions and/or validate these.

5. (e) Tools: validators should consider whether there is proper quality assurance around spreadsheets used for loss quantification.

6. (f) Maximum loss caps: as for LDA, any caps to loss assumed should be justified in terms of boundary constraints for loss such as portfolio size, other measures of total exposure (e.g. total number of staff per location) and legal boundaries such as time-bars.

4.1.8. Scenario outputs – recoveries

As for LDA, best practice would be to model gross losses in the first place, but to the extent that recoveries are allowed these need to be consistent with sums insured and supplier limits of indemnity.

1. (a) Validation should also consider the possibility that the loss may arise because of an uninsured peril, or that there could be a dispute over coverage/supplier liability.

2. (b) For losses charged back to customers, there is a need to ensure this is consistent with legislation and ideally reviewed by any committee responsible for the fair treatment of customers, such as the With Profit Committee for UK with profit products.

4.1.9. Scenario analysis governance

Given the high level of subjectivity around scenario analysis, it is important that there is robust governance around the process. Validators should look for evidence of the following:

1. (a) Second line/Internal Audit independent review and challenge of scenarios.

2. (b) Senior management sign-off: ideally, individual managers would be responsible for the sign-off of scenarios relating to their area (e.g. HR manager signs off on employee relations scenarios). This should ensure senior management engagement with the scenario analysis process, and helps to drive up standards.

3. If senior managers are not required to sign-off results, validators should consider whether the scenario exercise has been given proper consideration or whether it has been delegated to junior staff with little “buy-in” from SMEs.

4. If senior managers do sign-off on results, validators should also be wary of challenges and changes made by senior managers. For instance, a senior manager may be concerned about their reputation if the potential for a large loss is identified in their area and may seek to artificially reduce figures to save face.

4.2. Reviewing SBA Model Results

Key aspects to consider in validating the model and results:

1. (a) Deriving distributions from loss estimates: typically, we might collect 2 or more loss estimates encompassing an expected/typical loss and one or more severe/extreme loss estimates. The former might be assumed to be the mean/median of the loss distribution, with severe loss estimates assumed to relate to higher percentiles, but results can be very sensitive to what percentiles are assumed.

2. Sensitivity analysis should be performed on the impact on model results if different percentiles were assumed.

3. (b) Kurtosis: depending on the loss estimate values, how close these are to each other, and the percentiles of distribution these are assumed to represent, the resulting loss distribution could be thin-tailed or extremely fat-tailed, so there is a need to assess the kurtosis of calibrated distributions.

4. (c) Distribution: sensitivity analysis should be performed to understand the impact of alternative distributions on results, e.g., for frequency, Negative Binomial instead of Poisson (the former allows greater variance in events); or for severity, Weibull instead of Lognormal. However, given the subjectivity of scenario analysis inputs, the use of complex distributions with 4 or more parameters is not likely to be appropriate.

5. (d) Stability: often, Monte Carlo simulation is used to model the combined frequency and severity distribution, but results can be very unstable at the tail due to simulation error, and it may require more than one million simulations to achieve stability for a tail loss estimate. The number of simulations used should be supported by convergence testing.

6. Validators should also look for sensitivity testing to small changes in scenario inputs, including any changes as a result of scenario refreshes, and challenge models where small variations in inputs lead to large changes in results (though from (b), this could arise due to the gap between loss estimates and the resulting kurtosis of the scenario distribution).

7. (e) Back-testing: having calibrated the model, it would then be useful to compare this against historical losses. If the model ascribes a low probability to a recent large loss, then it may be the case that the model is weak.

8. (f) Alternative scenarios: it would be useful to consider the impact of alternative scenarios discarded as part of the process, e.g., a lower impact but higher frequency scenario, though it may be the case that loss estimates for these may be less robust.

4.2.1. Recurring losses

Scenario analysis may capture infrequent, high impact losses but not high frequency, low impact losses. The validator should consider whether an additional allowance may be required for these on top of scenario analysis capital modelled ¹⁶ . This could be based on the historical average of such losses, perhaps capitalised to reflect their recurring nature. On the other hand, it may be that such losses are implicitly included in base expense assumptions and expense risk capital (see 3.1.2 (h) above).

4.3. Structured Scenarios

A development to be aware of in the field of SBA is the use of what may be termed structured scenarios, which seeks to adopt a more objective approach to scenario quantification. It does this by constructing formulae to calculate losses based on how the scenario affects underlying drivers, for example the number of customers impacted by a cyber-attack, or the value of physical assets which could be affected by a natural disaster scenario. This approach could be used to strengthen expert judgement and avoid some of the pitfalls associated with scenario analysis results outlined in section 4.1 above. Further detail on structured scenarios can be found in Kramer and Ramakrishna (2016).

4.4. Scenario Analysis Example

Validating scenarios and modelled results involves a good understanding of operational risks as well as knowledge of risk modelling. Appendix B gives an example of the issues the validator may need to consider just for a single risk category, in this case financial reporting.

5. Aggregation and Allocation

Under LDA and SBA, operational losses are generally modelled by risk type, so there is a need to aggregate these allowing for diversification between operational risk types and possibly with non-operational risks. The IFoA Operational Risk Working Party has produced a paper on dependencies (Kelliher et al., 2020) which gives a good overview of the issues surrounding operational risk aggregation but validators should consider the following:

5.1. Correlation Assumptions

1. (a) Empirical assumptions: need to consider whether these may be compromised by lack of data, while for low frequency risk types, correlations may be systematically under-estimated ¹⁷ .

2. (b) There is likely to be some reliance on expert judgement to set these, but there needs to be second line/internal audit independent review and challenge of these.

3. (c) Correlation assumptions should ideally be supported by causal analysis, with underlying drivers of each risk type identified and compared between risk types to identify common drivers.

4. (d) Macro scenarios could help identify common linkages. Examples of these could include (see Appendix A):

   1. i. Change of government leading to a different legal and regulatory environment.

   2. ii. Pandemics like Covid-19: amongst other things, these could disrupt service, increase backlogs, and potentially increase cyber risk exposure due to increased working from home.

   3. iii. Economic downturns could lead to an increase in fraud or expose existing loans and other fraud.

   4. iv. Market falls could lead to mis-selling and other conduct losses arising.

   5. v. Change programmes could go awry, leading to system outages, data breaches and reporting errors amongst other things.

   6. vi. If these have not been considered already, validators could use these scenarios to assess the reasonableness of correlation assumptions, probing SMEs on scenario impacts to understand common exposures and dependencies.

5. (e) Consideration should also be given to the impact of Reverse Stress Test, ORSA, regulatory and other scenarios carried out as part of wider risk management on operational losses and what this implied for correlation assumptions.

5.2. Aggregated Results

1. (a) Sensitivity analysis should show the impact on results of different correlation assumptions (e.g. +/−25%).

2. (b) Ideally sensitivity testing would also involve testing different methodologies, e.g., if using a Gaussian copula, the impact of using a t-copula; of if using a t-copula, the impact of different degrees of freedom parameters. That said, the choice of copula may be spurious given the subjectivity of operational risk correlation assumptions.

3. (c) Given correlation assumptions and undiversified requirements, it should be easy to aggregate the latter using a variance-covariance approach as a broad check on copula results, though for technical reasons this is likely to be higher than Gaussian copula requirements.

4. (d) Benchmarking studies often give details of diversification benefits which can be used to assess the level of own diversification benefits, but it should be noted that these will vary with the number of risks modelled by each firm, with those modelling more operational risks typically seeing greater diversification benefits.

5.3. Aggregation across Business Units and Legal Entities

Often operational risk is modelled at Business Unit (BU) and/or legal entity level, and there is then a need to aggregate the results across these to arrive at a Group figure. This could be a simple additive approach, or allowance could be made for diversification between BUs and/or legal entities. If diversification is allowed for, the validator should consider the reasonableness of correlation assumptions using causal analysis as above, noting amongst other things that:

1. (a) Weaknesses in Group-wide governance, risk management and compliance could lead to diverse operational losses arising across BUs and legal entities.

2. (b) There may also exposure in respect of Group-wide systems, e.g., a key Group system failing could affect multiple BUs and legal entities, or Group systems could be exploited as part of cyber-attacks resulting in transmission across IT networks.

3. (c) Reputation damage suffered by one part of the Group could trigger downgrades and mass lapses across the group. This in turn could result in operational losses across the Group, e.g., because of backlogs arising in lapse processing, or the need to invoke deferral clauses on unit-linked funds due to mass lapsing.

5.4. Allocation

Having aggregated operational risk across categories, BUs and legal entities, there may be a need to allocate diversified capital back to these levels. The validator should have regard to the following:

1. (a) Allocations should have regard to service level agreements between companies which may preclude the charging back of certain operational losses, e.g., employee relations losses.

2. (b) Operational risk capital allocated back to with-profit funds needs to be consistent with PPFM or other governance arrangements for the funds and what these say about charging losses to the fund, e.g., it would be inappropriate to allocate conduct risk capital to a with-profit funds if these cannot be allocated to the fund in practice.

6. Causal Factor-based Models

LDA and SBA operational risk models have been widely criticised for not being dynamically risk sensitive. This is driven by many reasons, but most notably that historical loss data makes no allowance for current exposures of the firm, scenario analysis is subjective, and it is difficult to incorporate a direct link between risk and controls in existing model designs. Furthermore, operational risk models are often only recalibrated on an annual basis. Thus, operational risk models may not have a direct link to the dynamic risk exposures of the firm.

As a result, an emerging field in operational risk modelling is the use of causal factor-based models such as Bayesian Networks. These approaches assume that operational risk exposure can be described by a function of a set of underlying causal factors or drivers. By identifying and quantifying these factors, a risk sensitive model can be developed, based on how these factors link together to drive the frequency and severity of loss events. To the extent that data feeds can be automated to calibrate data-driven exposures, this can lead the way to real-time assessment of operational risk.

6.1. Bayesian Network Models

Bayesian networks are built upon a framework of causal factors, pulling together both the known (factors) and unknown (probabilities) into a visual node map to describe each risk process. The factors will be a mixture of firm-specific and external risk drivers.

Causal factors are used to identify key exposure metrics and conditional probabilities are used between the factors to estimate the frequency and size of loss when a given exposure is impacted. Together these can estimate the value at risk for a given unit of exposure. Appendix C provides a simple example of how causal factor modelling is overlaid with conditional probabilities for loss events, control failure and impact to provide a holistic distribution of losses.

Bayesian Network models can aggregate several risk processes or scenarios, via the use of conditional probabilities and common risk drivers, meaning that the resulting loss outputs are sufficient to calculate capital requirements, negating the need for assessing subjective correlations between risk types.

One of the key advantages of Bayesian Network models is that they can easily be visualised with the use of the node or factor map. These visuals are intuitive and easy to understand for any stakeholder in the business, allowing a much wider breadth of stakeholder challenge. This in turn will increase understanding of each business process and associated risk and controls, providing a much closer link between risk measurement and risk management approaches.

Conditional probabilities can be quantified to take into account the impact of controls and mitigating actions, thus providing a platform for what-if analysis, helping to analyse the impact of specific controls, and identifying which controls have the most impact on resultant loss, which can be used in future planning and investment choices.

Bayesian Networks can be complex and difficult to set up initially, requiring specialised knowledge of key business processes and risk drivers. External consultancies and software packages can however be utilised to support with the initial set up, and the effort involved can be repaid by the greater insight into operational risks and how these interact.

Validation of model design will be easiest when starting with the simplest designs and introducing additional complexity in a step-by-step process. Care needs to be taken to ensure the model does not become overly complex too quickly, resulting in too many subjective estimates and resulting in spurious accuracy of model results. Each additional variable added should be tested to ensure that the value-add in accuracy of model output outweighs the additional complexity.

The model could be utilised across several areas in addition to capital, for example, operational resilience or recovery and resolution testing, as well as emerging areas such as climate and cyber risk quantification.

Validation points to consider are:

• What causal factors are driving losses? Are there any notable omissions? Appendix A provides a list of basic causal factors which could be considered.

• Risk coverage: while it would be unrealistic for a Bayesian Network model to cover every sub-risk (of which there could be 300+), the validator should assess its ability to model key risk types across each Basel Level 2 or equivalent high-level category.

• Conditional probability assessments are difficult and can be subjective. Where these are data-driven, data choice and reliability should be regularly validated. Where these are judgement driven, a log of judgements, SMEs contributing to the judgement and materiality should be kept.

• Discrete probability nodes may utilise a “high, medium, low” assessment, which when combined with other continuous probability nodes may give the user a false sense of accuracy to the output. Sensitivity testing of node materiality is important in communicating this.

• The model will be highly sensitive to the inter-dependence of nodes, particularly across different risk processes (akin to the high sensitivity of traditional models to dependency assumptions). These variables will likely be the most difficult to quantify given the limited data available and hence communication of this subjectivity will be very important.

• Back-testing the model against material historical losses, annual losses in stressed periods, and to existing scenario assessments, will help validate model performance and identify any gaps.

• Quantification of the impact from specific controls failing may be challenging. SME judgement could be sought alongside analysis of near-miss event data.

• Scenario analysis: ideally, firms would supplement Bayesian Networks with forward looking scenario analysis as a check on the former model’s ability to cater for scenario losses, but if not, a validator may wish to construct his/her own scenarios to test this, perhaps based on notable historic loss events.

• IT systems: some IT packages may struggle with multiple drivers of risk (e.g. where there are more than 2 drivers of a particular loss) and the computation this gives rise to. Validators need to understand any IT limitations of the Bayesian Network model, noting Article 245 (f) of Solvency II Delegated Regulations requires limitations of IT used in internal models to be documented.

7. Conclusion

Validating operational risk models can be a complex undertaking but it is hoped this paper provides a useful starting point. While considerations will vary by type of model, a few common themes should be borne in mind:

• Risk profile: validators need to have a good understanding of the operational risk profile being modelled and whether any changes in it are being picked up.

• Risk coverage: operational risk is a diverse category with potentially hundreds of sub-risk types. While it is probably not feasible to model every sub-risk type, validators should ensure the resulting model is broadly representative of the risks which may arise under a particular category.

• Recurring losses: modelling may naturally focus on high impact, low frequency operational losses, but validators should consider how low impact, high frequency losses are allowed for.

• Loss data: this may not be enough to calibrate loss distributions and/or capture tail exposures. Validators should also be aware that loss data may:

  • ○ not capture certain relevant impacts (e.g. impacts affecting PVFP)

  • ○ include impacts not relevant to economic capital (e.g. lost new business)

  • ○ include losses no longer relevant (e.g. legacy mis-selling)

  • ○ include boundary losses that may be covered by other risk models.

• External loss data: validators need to consider how this is scaled to a firm’s size and whether it is relevant to the firm’s business, control environment and risk profile.

• Subjectivity: expert judgement is likely to be required for most operational risk models and validators should ensure this is made as part of a structured process, with the appropriate level of expertise involved and robust review, challenge and sign-off of assumptions.

• Recoveries: exposures should first be modelled on a gross basis and validators should consider whether modelled recoveries are in line with insurance policy limits and coverage, outsourcing arrangements and/or regulatory and PPFM constraints for amounts charged back to policyholders.

• Aggregation and allocation: correlation assumptions are likely to be subjective and should be tested against underlying causal factors, while allocations to legal entities and with-profit funds should be consistent with service level agreements and PPFMs respectively.

• Documentation: in general, poor model documentation may highlight a lack of professionalism around how the model was developed, while it is critical that scenario documentation is sufficient to evidence the breadth of risks covered and the quality of discussions around the choice of scenario and loss quantification.

Finally, validators need to consider intangible factors such as the culture of the firm commissioning the operational risk model and whether this could lead to bias and/or the understatement of risks; and/or the quality of SMEs and extent of senior management involvement, which can point to a lack to commitment to proper model development.