The FDIC announced on June 7, 2017, that it would be adopting the supervisory guidance on model risk management that was previously issued by the Federal Reserve and the OCC in 2011. You may have heard about this guidance in conjunction with AML Data Validations, but with this new adoption, the Federal Reserve, OCC and FDIC banks will all be held to a higher standard regarding model oversight. This has left many institutions questioning what proper model risk is and how they move forward.
Understanding what model risk, how to build better models, and how to verify model accuracy will reduce risk and losses at your financial institution while instilling best practices that can be refined and replicated to help the organization become more secure and successful.
What is Model Risk?
Model risk is the potential for error or adverse consequence from decisions made based on misused or incorrect model output reports. Poor model reporting can lead to financial loss, poor business and strategic decision-making, and potential damage to your institution’s reputation.
There are two major contributing factors to model risk. The first is that fundamental errors within the model can cause inaccurate outputs. This can be anything from poor data imports into the model to poor set-up of the model at implementation. The second major factor relates to the use of the model itself. Data may be flowing into and out of the model correctly, however, failure to understand and use this data in the proper fashion can result in model risk.
Model Risk Assessment & Inventory
Your institution should be identifying what models are employed in your environment. Create an inventory to capture all models and risk assess each one based on the magnitude of their associated risks. While not an all-inclusive list, these common risk factors should be considered in your assessment:
- Model Complexity
- Input Volatility
- Model Use
- Financial Impact
- Business Decision Impact
- Model Design
Preventing Model Risk
There are steps you can take to reduce the chance of model risk. The most important step is to put an “effective challenge” process in place. This is done by ensuring that there is ongoing critical analysis by objective, informed parties who can identify model limitations and assumptions and apply appropriate changes. There are three areas to look at in the people who will be doing the effective challenge:
- Incentive – Having a person who did not build the model lead the effective challenge will produce better results as they will have an objective view of the model, they will not hesitate to question any of the assumptions in the model, and they will not be tempted to hide any flaws in the model that could be revealed and reflect poorly on them.
- Competence – The people overseeing the model and raising the effective challenge must be competent and have knowledge of the model process. This can be accomplished through selecting people with this knowledge as well as by providing training on how the modeling works, what the outcomes should look like, and what indicates potential problems.
- Influence – You want the people who are overseeing the effective challenge process to have influence in the institution so they can provide leadership on implementing the needed changes and be heard by the board, management, and staff.
Creating Your Models: In-House Modeling vs. Outsourced Modeling
There is no question that your institution must create the best models possible to continue operating successfully. Depending on your institution’s size and available resources, there are two ways to get the modeling done: have your internal team build the model, or outsource the work to experts. There are some things to consider when deciding to do the modeling in-house or to outsource.
In-House Modeling
Before undertaking modeling in your institution, you will need to assess if the team that would work on the model has the time and skills to build an accurate model that will produce solid output data. You will also want to consider how much control and customization is required for your modeling needs.
One of the main benefits to in-house modeling is the amount of control you will have because you are the one building the model. However, the drawbacks to in-house modeling is that you could be limited by time and availability of staff, the complexity of the model that is needed, and not having the knowledge among your team to build an accurate model.
When developing your model in-house, there are important things to consider to make sure it’s as effective as possible:
- You need a clear statement of purpose that lays out what the model should be performing.
- Make sure the components in the model work as intended, are appropriate for the business goal, and are conceptually sound and mathematically correct.
- Have a good assessment of data quality and relevance, and that the appropriate documentation is available.
- Undertake periodic testing during development to make sure that the calculations within the model are accurate, that the model is stable and robust, that limitations are assessed, and that there is constituent behavior over a range of inputs.
Outsourced Modeling
Outsourcing modeling allows your institution to seek out the best experts available to suit your needs and build the model for you without being limited by knowledge or complexity. It also means that the outside experts are working on the model, which frees up precious time for your staff to address other issues in the institution.
An outside expert brings a wealth of knowledge to the modeling process, as it’s their job to make the most accurate models possible. They not only get to know the particulars of your institution, they have also experienced and gathered a range of best practices from the many different financial institutions and can apply them to your needs.
Testing Your Model
Understanding if your model is working effectively is an ongoing process. To get an accurate understanding of the model’s accuracy and ensure that it reflects reality, you should seek out and utilize user feedback and insights and ask senior management to question assumptions and methods. If the model is not working correctly, and you are developing it internally, you may want to consider hiring a professional team to move the process forward. If you are already outsourcing the work, you can work with your current vendor or change vendors.
What is Model Validation?
All models in the institution – as well as all parts of each model – should be subject to validation. Model validation is a set of processes and activities intended to verify that the models are performing as expected and are in line with their design objectives and business uses. The frequency of the validation depends on the complexity of the model and how often it’s used. The rigor of the validation depends on the potential risk.
The validation should be performed by people who do not have oversight of the model. They should have the skills, experience, and authority to accurately validate the model and ensure that changes are made throughout the institution.
The ongoing monitoring of model validation is one of the most robust periods of testing for your models. It should include checking for mathematical accuracy, ensuring that the model is performing as expected, and assessing if it’s meeting regulatory requirements and best practices. It should be noted that one regulatory hot button right now is the override of assumptions, which can be used to manipulate outcomes. It is considered a best practice today by regulators to fully document any assumption override decisions.
The final step of the risk model validation process is the outcomes analysis. In this step, you want to compare the model’s outputs to benchmarks and industry norms, but most importantly, to the actual outputs created by the institution. This will allow you to more accurately evaluate the performance of the model. You will also want to look for some early warning metrics that indicate any flaws in your model.
Building a Strong Future by Building Better Models
Building, testing, and validating accurate models is important to your financial institution’s financial well-being. By building a strong team around the modeling process, utilizing best practices, and seeking expert consultants when needed, you can create strong models that will serve your financial institution well into the future.
