Model Risk Management Under Deep Uncertainty
A Case for Climate Risk Models in Financial Institutions
With Ali Arab
A model is a simplified representation of reality, which at its best is as good as its underlying assumptions. Model risk is the risk of potential adverse consequences from decisions made based on incorrect or misused models. In our most recent memory, the global financial meltdown of 2008 revealed to us the devastating impacts of aggregated risk of decisions made based on incorrect or misused models by banks and other financial firms. In the wake of that crisis, the highest level of scrutiny on model risk management (MRM) practices has been implemented within the financial industry, and the Federal Reserve Bank mandated unprecedented standards to ensure that model risk remains under control. As a result, the financial industry has become “the gold standard” for cutting-edge solutions in model risk management. While the first generation of the model risk practice was initiated as a regulatory compliance framework for “too-big-to-fail” banks, it has evolved to a standard operation in the financial industry. Successful implementation of CCAR and CECL stress testing program grounded with a robust model risk management framework has yielded confidence from markets and stakeholders in the performance of the financial institutions. With the emergence of climate risk models as a result of ESG initiatives, anticipated regulatory mandates, and business imperatives in financial institutions, model risk management practices need to evolve to address the complexity of this emerging function.
Managing climate risk requires development of appropriate quantification methodologies, climate and economic scenarios, and data. Unique challenges driven by limited relevance from historical data and long-term nature of forecasts is pushing the modelling efforts into an unchartered territory. Given the distinctive nature of climate risk, we need to have a clear understanding of various modeling approaches, underlying assumptions, limitations, and climate related scenarios (to translate the climate risk into financial risk) in order to exercise an objective review and challenge as required by the model risk management framework. Financial institutions started to develop risk management frameworks to conduct climate risk analysis involving, risk identification, exposure measurement followed by mitigation strategies. Accordingly, they are investing in developing climate risk modeling capabilities to assess both physical and transition risks and translate them into credit, market operational and liquidity risks in accordance with firm’s model risk management framework. As part of this effort, stress testing methodologies, scenario design, benchmarking, and sensitivity analysis tools are becoming indispensable for capturing the uncertainty associated with the climate risk models. Further, model risk management function started to leverage the existing model validation framework to evaluate the conceptual soundness and perform fit-for-purpose assessments.
Climate risk analysis requires specification of scenarios for various climate-related variables that affect the underlying financial assets. At this time, financial institutions and regulators are considering scenarios provided by NGFS and Representative Concentration Pathways (RPCs) adopted by the Intergovernmental Panel on Climate Change (IPCC) as a starting point in climate risk analysis. NGFS scenarios provide a common framework for analyzing the impacts of climate change (physical risks) and policy responses (transition risks). NGFS has developed six scenarios under three categories (Orderly, Disorderly and Hot House World) and transition pathways have been developed using integrated assessment models (IAMs) including GCAM, MESSAGEix-GLOBIOM, and REMIND-MAgPIE that integrate energy, climate, water, agriculture, land use and macroeconomic interactions. These models allow for various assumptions including changes in the underlying technologies, contribution of renewable energy growth, evolution of emissions and temperature, among others. NGFS provides scenario narratives associated with six scenarios. For instance, IEA’s Net Zero 2050 scenario intends to limit the global temperatures to 1.5 through an ambitious climate policy implementation. Under this scenario, assumptions such as emission levels, carbon price, evolution of energy commodity market, and mobilization of capital to renewable energy sector, will capture the variations in transition pathways. Subsequently, these scenarios are further downscaled to assess their potential implications at country level. Financial institutions should develop capabilities in leveraging these scenarios, understand the assumptions involved in the IAMs, and build additional scenarios tailored to their portfolios to conduct climate impact quantification. These are just some of the examples to demonstrate the distinctive nature of climate risk models compared to well-established, classic models used in financial institutions.
There is a deep uncertainty associated with climate risk models, given their nature. As stated by DMDU Society, “deep uncertainty exists when parties to a decision do not know, or cannot agree on, the system model that relates action to consequences, the probability distributions to place over the inputs to these models, which consequences to consider and their relative importance.” Deep uncertainty usually involves decisions that should be made over time in dynamic interaction with the system, as is the case in climate risk models. Most of the models that are currently being used in financial institutions are classified under Level 2 or Level 3 of model uncertainty (where there is a manageable number of plausible futures with a few alternative stochastic system models), as shown in the following table. Climate risk models, however, fall under uncertainty Level 4, where there are many plausible and even “unknown” system models and actual outcomes. Therefore, this is an unchartered territory for model risk management practice that requires special attention and revisiting the model risk management framework to meet the complexity of this new business imperative. That said, it is important to leverage the current capabilities and existing industry standards in model risk domain to build a cutting-edge solution that addresses the model risk requirements for the emerging climate risk models in financial institutions.
We believe that the current model risk management frameworks, to a large extent, can address the model risk requirements as it relates to validation of methodological aspects of climate stress testing models and their outputs. However, given the deep uncertainty associated with the model inputs that are derived from climate scenarios, this area should be the focus of attention for redesign of the existing model risk practices. Accordingly, we propose 10 recommendations to be considered, as follows:
1) Model risk management should consider building a separate climate risk model validation function. This group should draw upon quantitative skills similar to other modeling areas in addition to tapping into expertise from areas including climate science, financial risk, engineering technologies, and energy markets.
2) Model risk management should draw a clear line on where their “model jurisdiction” starts and ends and update policies and procedures commensurate with the deep uncertainty associated with climate risk models. For instance, MRM should not be in the business of validating the climate scenarios as it relates to future climatological events resulting from climate change. However, MRM should play an active role in validating the scenarios (e.g., the impacts of California wildfires on residential mortgage portfolios) emanating from climate scenarios as they relate to idiosyncratic impacts of physical and transition risks on their financial institution.
3) Quantitative and qualitative assumptions pertaining to physical risk emanating from various environmental factors such as rising temperature, precipitation levels, heat wave impacts and their linkages to economic variables (e.g., productivity and GDP) to the financial institution’s performance, should be evaluated rigorously to capture the wide range of plausible model outcomes expected in Level 4 of model uncertainty.
4) Similarly, for transition risk, assumptions on the policy changes, technology evolution and socioeconomic changes should be tested through external discussions with lines of business, risk management, compliance and internal audit functions.
5) Inherent complexity and novelty of application of climate-related methodologies such as IAMs that combine multiple component interactions to portfolio exposures, will test the limits of conceptual soundness evaluation or fit-for-purpose assessments. To be able to do this effectively, model risk management function should incorporate interconnectivity impacts into the overall forecast results.
6) Model risk management should actively participate in discussions with regulators, think tanks, and other interest groups to ensure the quality and accuracy of climate data (e.g., scope 1, 2, 3 emissions) that are part of the quantification methodologies. They should not limit their assessment to identification of the gap and limitations but should help to drive the conversations with lines of business in implementation of appropriate approaches to address the data quality issues.
7) Model risk management should develop appropriate benchmarking methodologies to assess the proprietary climate models (e.g., portfolio alignment tools, complex IAMs, rating methodologies, and machine learning tools) from various vendors to capture the uncertainty associated with these types of models.
8) Model risk management should actively interact with internal and external stakeholders to identify weaknesses in climate risk models due to technical and business limitations. For instance, issues arising from nonlinear interactions across time horizon, data availability, portfolio maturation, and ambitious climate policies, will pose challenges in identification of limitation during the validation process.
9) Model risk management should develop a set of key performance metrics that can help to monitor the degree of uncertainty in model forecasts and linkage to scenarios as part of the outcome analysis in an ongoing basis. In the absence of historical data, backtesting of climate risk models is not feasible. Therefore, ongoing monitoring analysis of these models should include review of multitudes of scenarios and “unknown” model outcomes due to a Level 4 of model uncertainty.
10) Model risk management should work with the enterprise risk management (ERM) team to develop an integrated risk management framework for climate risk analysis, where credit, market, liquidity and operational risk functions are aligned together with climate goals of financial institutions. That will help evaluate the validity and degree of uncertainty of model outcomes at the firm level.
Model risk management evolved significantly from their nascent levels to a well-established practice during the last decade. However, climate risk management is still an evolving practice at its early stage in financial institutions. Now these functions are up to a new challenge to take on enhancing the overall risk management framework at enterprise level. That requires an active role to be played by model risk management via close collaboration with various stakeholders in the climate risk space.
