Skip to Main Content
PCBB Banc Investment Daily March 17, 2014
Banc Investment Daily
March 17, 2014

Back-Testing Nostradamus

There has been an uptick in regulator interest regarding how banks use various models and how risk is managed around such models. Regulators are concerned that the models banks rely upon don't cover a broad enough range of possible outcomes and therefore don't adequately prepare banks for plausible future events - especially in the case of asset liability management (ALM) and credit stress testing. We do our best to offer helpful solutions for bankers, so we would like to suggest you add language to your ALM policies. We suggest the writings of Nostradamus be considered when forecasting. As an added step, perhaps add Nostradamus himself to your ALCO committee.
Everyone knows Nostradamus was a French apothecary who lived from 1503 - 1566 and many believe he could see the future (would be very helpful to banks right now). However, few know he also developed a "rose pill" to protect against the plague (which didn't work), so even his prognostications are not 100%. He has so many prophesies about the future that eventually a book with 1,000 of them was published in the mid 1550's. The book is still in print almost 500Ys later and has been credited with predicting numerous major world events. To non-believers, the accuracy of his predictions may be from willful mistranslation, misinterpretation or just helpful hindsight, but few could argue there remains continuing fascination with the idea that a 16th Century occult seer could predict the future.
When trying to gauge the future you sometimes have to look to the past. Perhaps this is what regulators were doing when they re-released their 2011 Supervisory Guidance on Model Risk Management most recently. That guidance told banks to undertake active model risk management. It also indicated a good framework should include robust model development, implementation and use, effective validation, plus sound policies and controls. Regulators cautioned that there can be adverse consequences from even the best intentioned decisions based upon the output of models, so bankers should have a good process around any model used. One key concern is that models may have fundamental errors or provide inaccurate results, or they may be used inappropriately and that can lead to unintended consequences, misstatements or a false sense of security. Understanding model limitations is critical.
It has been our experience that in some banks the use of models is less pervasive so it may have less an impact on the bank's financial condition or strategic decision-making. However, for the vast majority of banks, reliance upon models runs deep and output is heavily relied upon by management.
Of special interest to regulators is the outside validation of models. This process should be independent from model development and use and specifically highlight potential risks. Those undertaking a validation process should not have a stake in the outcome either. Here, the purpose is to get a validation that verifies models are performing in line with their objectives and business purpose and as expected. The process should identify limitations and assess the potential impact of assumptions used in a given model. Here too, all of the components should be subject to validation including inputs, processing, outputs and reports. Importantly, this also includes models developed in-house and those developed by vendors or outside resources.
Validation is an important risk management process, so it should be undertaken on an ongoing basis. Baseline outcomes for stable economic times should be noted, but also incorporate the knowledge that they might paint an overly optimistic picture when more stressed economic conditions occur.
To be sure, models are wonderfully useful tools, but the information they produce is only as good as what goes in and the assumptions they rely upon. That is why it is so important to recognize model limitations and to undertake critical analysis of both what goes in and what comes out.
Models that help your bank understand its strengths and sensitivities should serve you well in preparation for the future. In the meantime, while the idea of a Renaissance apothecary on your ALCO committee may be intriguing, we think model validation might be a more useful path - plus, how can you back-test Nostradamus?