Navigating the Continuous Industry and Regulatory Expectations on Model Risk

Three industry leaders share their insights on the latest financial model trends and risk and control considerations for model risk and internal audit teams. 

The outcomes of the COVID-19 pandemic have changed the economic landscape dramatically. The global health emergency has also impacted many financial models designed and used in the pre-pandemic world.

The Risk Management Association has hosted a session on model validation and governance as part of RMA’s risk readiness webinar series where speakers share with the nearly 700 attendees new challenges that internal and external auditors currently face when managing model risk.

Three subject matter advisors – James Gannon (Ernst & Young), Chris Imperatrice (Ernst & Young), and Kevin Oden (Kevin D. Oden & Associates) share their insights on the latest market events, regulatory expectations, the impact from the pandemic, and what internal audit functions should consider when demonstrating the appropriate coverage of models.

We have conducted polling during the session, which reflects that there are imminent issues arising when auditing models and navigating the model risk management. In one example, about 33% of the poll participants, indicated the lack of appropriate quantitative skill sets as part of their internal audit team as a potential challenge when auditing financial models.

Additionally, 33% of the voting participants, have chosen “model governance” as the main area that their internal audit function focuses on when auditing models and model risk management. The webinar participants include individuals working as internal auditors in the banking and the financial services industry.

MODEL RISK AND THE ROLE OF INTERNAL AUDIT

Kevin Oden notes three components of a model: ‘choices and assumptions’, ‘data’ and ‘the uncertainty or the probability of the outcomes.’

Obviously, there is a risk of model failure. The model risk mainly arises from the design and the usage. First, a model may have fundamental errors if it does not produce accurate outcomes consistent with its design objective. Another flaw may occur when given assumptions and sample choices become irrelevant. A model may fail if it is used incorrectly, or a user misapplies the model outputs and reports.

Errors can occur at any point from design through the implementation of a model. According to Oden, many models in the post-pandemic world should be recalibrated, or they will simply break down.

The role of an internal audit team in assessing model risk has been growing. As the usage of complex models continues to grow across industries, auditors have been better equipping themselves and their organizations to effectively challenge these models.

The effective challenge, also known as a guiding principle of the model risk management in the industry, starts with a critical analysis. A group of objective and informed parties would typically conduct this analysis. It is to identify limitations and assumptions of a model and therefore to challenge developers on their methodologies and approaches.

WHERE IS THE LINE?

The speakers note that the 2011 Supervisory Guidance on Model Risk Management (SR 11-7), has brought significant changes to the model risk management industry. It is because the regulators have acknowledged a life cycle of a model, according to Jim Gannon, a managing director of consulting services at EY.

He notes that the Office of the Comptroller of the Currency guideline (OCC 2011) also requires banking organizations to have a firm-wide model inventory, including specifics such as model use, products, responsible parties, and planned activities. “Model Risk is federated across your audit plan,” Gannon adds.

However, the firm-wide approach may confuse many people when splitting duties among the three lines of defense in the context of model risk management: the modelers, the model risk management (MRM), and the internal audit team. It is because risks arising from financial models are not necessarily confined to a particular business unit.

As per the OCC guideline, model developers, users and MRM should document their work including ongoing monitoring, process verification, benchmarking, and outcome analysis. Also, the guideline suggests that internal auditors should verify relevant policies and processes, confirm the completeness of the model inventory and the model validation work, and review the reliability of data used by models, among others.

Answering an audience question of what controls internal auditors should expect in the first line of defense, Gannon says, “You want the second line of defense to put requirements in place. What you want to do for the first line of defense from the internal audit perspective [is that] you want to make sure the first line [modelers] as they are developing models and making adjustments to models are operating within that framework.”

THE AUDITS

Two illustrations provided by EY exhibit four pillars of the internal audit model risk management framework: functional model audits, baseline audits, model-specific audits and continuous monitoring.

According to Chris Imperatrice, a senior manager of the enterprise risk consulting group at EY, an internal audit team typically focuses on specifics of the operation for model risk management coverage. Internal auditors look at data integrity and controls, methodologies and analytics, model implementation, business processes and model usage, governance, and model performance reporting.

The speakers say, especially due to the pandemic, internal auditors and model risk management team leaders may ask the following questions before the model risk audits:

• Has your model risk increased in this environment?
• Are there specific businesses, usages and models impacted?
• Is your existing model risk management framework functioning as expected in times of stress?
• Are there any new product and business related models which need to be reassessed?
• Have business continuity plans been fully activated and are they sustainable for an extended period of remote operation, including offshore locations?

Additional considerations include machine learning and artificial intelligence-driven models. A webinar participant has asked the speakers about what changes the internal audit function should look to see from the first or the second line of defense to capture the increased risk with the artificial intelligence and machine learning-driven models given the fast-growing implementation of these models across the financial services industry.

According to the answers by the speakers, the machine learning-driven model audit comprises of ensuring both the first and the second lines of defense have sufficient talent to understand the models they are utilizing, having policies and procedures to ensure the interpretability of the processes, as well as, continuously monitoring the governance and the model performance.

The speakers note that there are potential issues with model-related software upgrades as well. This occurs when auditors may not be able to review what is behind the changes. “The risk of embedded problems, unwanted upgrades and the deprecation are things that have to be managed in the process,” notes Oden.

Key challenges in model risk management remain as model risks are still rising amid the public health crisis. As the presenters note, banking institutions should assess and reassess the pandemic fallouts on models and the future direction of the model risk management as we gain more clarity on COVID-19 and its impact.

WATCH NOW

Register to watch this session, Model Validation and Governance, and many more from the Internal Audit webinar series from RMA.

---

As the Managing Director of the RMA Model Validation Consortium, Kevin is passionate about providing high-quality model validation services at a competitive price point for RMA member banks. Kevin holds a Ph.D. in math from UCLA and was a leader in risk management and model validation for Wells Fargo Bank.

 

---

As Product Marketing Specialist, Adalla is responsible for driving both the strategic and tactical aspects of RMA’s product sales. Prior to RMA, Adalla worked as a reporter for global media services and publication groups such as PEI Media and the Financial Times Group. She started her career at Campbell Lutyens, a global private capital advisory group. As an avid learner and a curious adventurer, Adalla speaks Korean, English, and Spanish, and has traveled to 19 countries. She graduated from Incheon National University with a Bachelor of International Trade in Northeast Asia Economic Studies.