I’ve been building and validating risk models for fifteen years, and I really thought I’d seen it all. And then I started looking at sustainability and climate risk modelling – models that estimate future financial losses due to climate change, biodiversity loss and so on. But I’ll come back to those horror stories later. First let me make the more general point that people who build and use models in this area often fall into one of two traps.
Some are not used to quantitative modelling and are seduced by the precision that comes out of their shiny new model. After all, doesn’t “We have a 95% value-at-risk of CHF 1.37m” sound so much better than “We think we might lose about a million in a bad year”? The former statement might sound fancier, but it’s only more useful if it can be justified.
At the other end of the spectrum, some people are well aware that there is huge uncertainty inherent in their modelling, but they don’t think that this uncertainty matters. “All climate risk models are uncertain,” they say. This is true, but how uncertain are they, and what about them is uncertain? Both extremes lead to poor decisions, and that’s where validation comes in.
We can only make decisions using model outputs if we understand the uncertainty around them. And that means understanding the model inputs as well as the underlying assumptions and model-building decisions. We should know how the model has been tested and what controls, fallbacks, and governance are in place around it. And as obvious as it may sound, we should know exactly what it has been designed to do and who is taking responsibility, for ensuring it fulfils its purpose.
Model validation means having someone ask those questions. Someone without a horse in the race. Anyone can do this who has the necessary modelling and subject-matter knowledge, who is not involved in developing or using the model, and is not in the direct line of command. But sometimes it helps to get a truly outside view, to get a bit of (anonymised) insight into what the competition are up to, as well as into any regulatory expectations.
What we can learn from financial services
For those of us who have seen both, it’s clear that the (non-financial) corporate world can learn a lot from financial institutions about what model validation should be. But we can also learn a lot from financial institutions about what model validation should not be. Model validation should never be one quant competing with another to get an additional decimal place of precision at the expense of explainability. Nor should it be about performing the same statistical tests year-on-year in the full knowledge that the result will also be roughly the same.
What can go wrong
But I promised horror stories One major firm tried to estimate the “realistic worst case” for several different climate risks. The models used for each individual risk were fine - but they made a critical mistake with the results: they simply added all those worst-case estimates together. Each model’s worst case was based on rare events that were unlikely to happen on their own, but far more unlikely to happen all at the same time. By adding them, they ended up reporting an outcome so extreme that it would only occur once in a billion years, rather than the one-in-twenty year loss that they were aiming for. Another firm performed a perfectly good scenario analysis, and concluded that they would benefit from the transition away from carbon. But they had neglected to isolate the transition impact by also performing a similar calculation without the transition and comparing the two. A significant transition cost was masked by the firm’s aggressive growth assumptions.
A bank had built sophisticated models to assess how climate would impact the credit risk associated with different parts of their loan book. They took physical and transition risks into account under different scenarios and identified the parts of the portfolio which they considered high risk. But they neglected to take the maturities of the loans into account. This led them to believe that their main exposure to climate risk – something which generally unfolds over decades – was through a portfolio with a six-week maturity.
What next?
So, if you’re nervous about your climate models — and if you’re not, then you probably should be — then I recommend a robust model validation. Feel free to give me a call!
