Who will mind the models?

Governance

Model governance is foundational to effective model risk management practices. Good governance structures can facilitate compliance with current laws and regulations through accountability, documentation standards, and oversight. Specifically, the modeling process should have model owners, institutional accountability (including for vendor models), an up-to-date model inventory, stakeholder involvement, appropriate management approval, and contingency plans.

Federal regulators have broadly interpreted what is considered a model for purposes of SR 11-7. Based on the regulators’ definition, any “quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates” is a model.

The definition goes further to encompass many internal processes that some organizations would refer to as a “tool” rather than a model. In SR 11-7 regulators further clarify that, “the definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgement, provided the output is quantitative in nature.”

For example, some institutions would consider a Customer Risk Rating methodology for BSA/AML purposes a “tool” because “traditional” modeling techniques (e.g., regression) are not being used to interpret the data. However, under SR 11-7 guidance, this type of analytical tool should be considered a “model.” The Credit Risk Rating methodology is considered a “model” because the analysis has elements that are subject to model risk: 1) an information input component (qualities of the customer); 2) a processing component (translating those properties into a score); and 3) a reporting component (categorizing low, medium, and high-risk customers). There is model risk any time varying data points are used to create a quantitative measure (a risk score) to make a qualitative judgement (riskiness of a customer).

After determining which methods, systems, and approaches are eligible for MRM, the organization must systemize the management and oversight of these functions through appropriate documentation and accountability.

1 Policies and Procedures

Given this level of regulatory attention, it’s important to formalize policies and procedures that govern the modeling process. The goal is to generate models which consistently produce reliable results. When designing policy and procedure coverage, financial services institutions should ensure these topics are covered:

• Roles and Responsibilities/Approvals: Indicate “Who approves a model for use?” and “Who approves changes to the model?”
• Model Inventory: Maintain a centralized model inventory that includes all models used, the generic risk ratings, and other relevant information (e.g., last validation date, next scheduled validation date, etc.).
• Documentation: Establish guidelines for model development documentation, what elements the documentation should contain, and the person or team responsible for approving the documentation for use.
• Risk appetite: Specify the acceptable level of risk. Typically, this will also outline the model risk rating process.
• Model validation procedures: Determine how often models of each risk rating are validated, which department is responsible for working with independent validators, etc.
• Contingencies: Identify who escalates the situation and decides what to do if a model is not working properly, what action is taken if policies and procedures are not followed (e.g., if a model is not validated on time, who is notified to ensure immediate action is taken to rectify), and who becomes the model owner if key personnel exit the organization?

These principles are not the only areas where financial institutions need to create policies and procedures. Other elements may be jurisdiction dependent, operationally specific, or reflective of the specific type of institution.

2 Documentation

Institutions also frequently struggle with model documentation. Especially in the case of a vendor-supplied model, banks often rely on boiler-plate model manuals or user procedures from the vendor that provide insufficient detail on the model’s methodology, limitations, and controls. The goal of model documentation is to communicate the function and methodology of a model to both users and non-users alike. These best practices will help achieve that goal:

• Model Owner: Who has ultimate accountability for model use and performance?
• Model Purpose: What is the model used for and, equally, what is it not used for? For example, a Fair Lending Model should be used to determine disparate impacts in mortgage lending but should categorically not be used for underwriting purposes.
• Assumptions and Limitations: The assumptions that inform a model are too frequently left undocumented, even though they usefully contextualize inputs and outputs. For example, an underlying assumption of a portfolio projection model might be that the model only holds if no economic shocks occur.
• Model Methodology: Discuss the methodology selected, alternatives considered, and justification for the method settled upon. You should be able to explain your model to colleagues with a non-quantitative background. For less complex models, this can be as simple as documenting that the methodology is standard practice, and no serious alternatives exist.
• Data: Document the sources of data used (internal and external), any transformations performed, and any limitations or assumptions in the data.
• Final model and outputs: Document the model that was ultimately created, and define all inputs, all outputs, and any relevant details for operation. An uninvolved third party should be able to look at this section and, with the relevant data, completely reproduce the operation of the model independently.
• Model testing – Define any model performance/assumption testing being performed. Then, report on and interpret the results.
• Model Change Log – When was the documentation last updated? What was changed from the previous version? Who approved the change?

Since these documents inform senior leadership, external parties, and regulators, consistent and thorough documentation practices are imperative.

3 Change Management

Change happens—often in the form of regulatory updates and emerging industry trends, sometimes in the form of a worldwide pandemic. Institutions should position themselves to address these changes through model upgrades or redevelopment. Both large and small updates can present new risk exposures, including third-party and operational risk. Fortunately, the change management process provides a structured way to implement model-related changes and minimize those risk.

An oversight process can ensure key stakeholders agree with — and authorize — the model change while considering the priorities of those affected (such as parties external to the department who use the model, Senior Management Committees, or the Board of Directors.) Consider engaging a third-party vendor to support the process.

Third-party risk can be a special problem for models developed by vendors, housed within their platform, subject to their policies and procedures, and then updated by the vendor—especially if your institution has restricted access to the underlying logic. To mitigate this risk, require the vendor to formally communicate model changes prior to implementation. Also, periodically request results from independent reviews of vendor models, including SOC reports or audit reports in which change controls are included in the scope.

These measures help ensure that changes to models and their effect on decision making are considered by model users and appropriate decision makers.