08 July 2009

Published in: Capital - models, Risk governance

The modellers, not the models, failed

We should remember Nobel laureate Robert Merton's warning that "models are not at all precise in their application to the complex real world," argue Athula Alwis and Dave Ingram.

Have you ever heard anyone say that their car got lost? Or that they got into a massive pile-up because it was a 1-in-200-year event that someone drove on the wrong side of a highway? Probably not.

But statements similar to these have been made many times since mid-2007 by CEOs and risk managers whose firms have lost great sums of money in the financial crisis. And instead of blaming their cars, they blame their risk models. In the 8 February 2009 Financial Times, Goldman Sachs' CEO Lloyd Blankfein said "many risk models incorrectly assumed that positions could be fully hedged . . . risk models failed to capture the risk inherent in off-balance sheet activities," clearly placing the blame on the models.

But in reality, it was, for the most part, the modellers, not the models, that failed. A car goes where the driver steers it and a model evaluates the risks it is designed to evaluate and uses the data the model operator feeds into the model. In fact, isn't it the leadership of these enterprises that are really responsible for not clearly assessing the limitations of these models prior to mass usage for billion-dollar decisions?

But humans, who to varying degrees all have a limit to their capacity to juggle multiple inter-connected streams of information, need models to assist with decision-making at all but the smallest and least complex firms.

Merton's caveat

Models can synthesize reams of complex streams of information into trends. Models (like humans) can find trends where none exist. Models can help to answer complex "what if" questions that are always going through a business manager's mind. But those answers are only reliable if the modeller understands the warning Robert Merton gave during his Nobel Prize acceptance speech in Stockholm in 1997:

"The mathematics of financial models can be applied precisely, but the models are not at all precise in their application to the complex real world. Their accuracy as a useful approximation to that world varies significantly across time and place. The models should be applied in practice only tentatively, with careful assessment of their limitations in each application."

Models are by nature based upon simplifications of the complex real world. A model of the world that did not contain simplifications would need to be as complex and as large as the world. A good modeller will spend a significant amount of time deciding which aspect of reality to reflect in detail in the model, which to simplify and which to leave out completely.

Freddie Mac, according chairman John Koskinen, tested its mortgage portfolio before the crisis to see what would happen if house prices dropped by 5%-10%. Actually, that has been the national average historically. Unfortunately, the modellers or lenders did not recognize that the housing bubble has not been this large in many decades. So, no-one bothered to test what would happen if house prices dropped by 20%-30%. The catastrophe could have been at least recognized at an earlier stage.

Models are a necessary but dangerous tool. A mathematical tool cannot model human behavior.

What can we learn?

A mathematical model is a tool. It cannot and should not replace the practitioner's experience, judgement and business intuition. The major strategic decisions should be guided by the business knowledge and common sense of experienced business leaders, not by models. A model must reflect business realities as closely as possible. Using inappropriate models mechanically without exploring the applicability has been a serious issue that must be addressed. For example, Dr Nouriel Roubini, who predicted the impending US and global meltdown in September 2006 at an IMF economists' meeting, was criticized for not using mathematical models. Roubini's approach actually combines mathematical modeling with his experience, judgment and business knowledge. In other words, Roubini is comfortable with the fact that models by nature are an abstraction of the real world and therefore, unable to capture all its complexities. This holistic approach leaves enough room for judgment and nuances.

It is important to use multiple risk metrics and models when engaged in a risk analysis. These could include value-at-risk (VaR), conditional tail expectations (CTE), volatility, scenario testing, stress testing, etc.

Let's think about the most popular of these risk metrics. VaR as a risk metric is easy to use and widely accepted (even by regulators). It is supposed to give you the amount of loss in an extreme case. On the other hand, the absolute value that represents VaR is sensitive to many parameters used in the modelling process. VaR can be manipulated and tends to underestimate worst-case scenarios. In addition, VaR under certain circumstances seems to disregard tail dependencies. A decision strictly based on VaR as a standard can be a very dangerous one. It seems that in the period leading up to the current crisis many key decisions were.

The assumptions used in any model should be validated by business practitioners. It is imperative that analysts and modellers understand the market conditions, coverage and business processes rather than independently selecting assumptions for models in a vacuum. It is pretty clear now that many of the drivers of the mortgage vehicle did not have a clear idea about road conditions and how slippery the route could become during a storm.

The simplifying assumptions should be evaluated and periodically re-evaluated for validity. Historically, some of the best minds in the world have worked decades to simplify assumptions in well known models like the Black-Scholes option pricing formula. Over-simplifying models without a thorough assessment of its economic implication could lead to disastrous scenarios. James Boness developed a precursor to the Black-Scholes option pricing model in his PhD thesis in 1962. The key difference between the Black-Scholes model and the model by James Boness is the recognition by Fischer Black and Myron Scholes that the risk premium for the option is actually embedded in the stock price. Therefore, one can use the risk-free rate in the option pricing model. That simplification materialized in 1973, 11 years after James Boness published his model.

It is imperative that analysts and modellers understand the market conditions, coverage and business processes rather than independently selecting assumptions for models in a vacuum.

It was as recently as April 2000 that David X Li published his Riskmetrics working paper (the first draft was in September 1999) presenting his Gaussian copula model to price CDOs. It is highly doubtful that enough time had passed to carefully evaluate the simplifications of Li's model. Yet the CDO market went from $275 billion in 2000 to $4.7 trillion in 2006.

As a simplifying assumption Li considered correlation to be constant. However, correlation -- in other words contagion risk -- is anything but constant. It tends to increase materially under scenarios of extreme financial stress. Another major assumption Li used was the use of credit default swap prices to parameterize his model. This simplification allowed Li to circumvent obtaining historical data to quantify his risks. However, the pricing Li used was limited to stable pricing during an economic expansion. It completely misled practitioners about the downside risk during times of economic contraction. Moreover, Li's model was not designed to handle a paradigm shift such as the change in direction of the overall economy symbolized by material change in house prices.

The usefulness of the Gaussian copula is that it allows modellers to describe the joint behavior of risks when they know the individual behaviour of parts.

The models should be applied in practice only tentatively, with careful assessment of their limitations in each application The main limitation of Li's model was known from the beginning. The Gaussian copula by design reflected the dependency (i.e. correlation: the likelihood of bad things happening at the same time) in the body of the data. It did not address the likelihood of many extreme events happening at the same time or in sequence (i.e. tail of the data). However, for the first time the quants had a simple enough model that they could programme efficiently to price credit derivatives.

David Li did try to raise an alarm when he said: "The most dangerous part is when people believe everything coming out of it. Investors who put too much trust in it or don't understand all its subtleties may think they have eliminated their risks when they haven't."

Li got the idea for his model from a concept in life insurance called "broken heart." It seems an elderly spouse is not expected to live too long after the death of a partner if it they had a long and successful marriage. In that regard, looking for correlations in the body of data makes good sense. In fact, applying this concept using a Gaussian copula in life insurance analysis is a necessary step. However, this thinking is fundamentally flawed when the real risk is correlation under stress. Under duress, extreme events tend to line up and move in the same direction, a phenomenon about tail risk that is not captured by a Gaussian copula.

David Li did try to raise an alarm when he said [of his model]: "The most dangerous part is when people believe everything coming out of it."

Use actual original data. The lack of knowledge of underlying risks played a major part in bundling and rating of mortgage-backed securities during the current financial crisis. It is useful to know the limitations in data such as number of data points and whether the data covers a full business cycle. If there had been adequate knowledge of underlying data, it is possible some investors would have avoided CDOs made of sub-prime mortgage pools.

The data that goes into models should be validated, scrubbed and compared to at least one other independent source. The axiom of "garbage in, garbage out" is a timeless statement. Inaccurate data has been the cause of many incorrect projections during the current financial crisis.

Regular review/upgrade of models and underlying technologies has to be carried out. For example, the original Black-Scholes model was upgraded numerous times. At times, it was due to work by other academics to simplify assumptions. However, there were occasions when the model needed upgrading simply because of new technology.

Model correlation and systemic risk (risk is not randomly distributed; you cannot escape it). The current financial meltdown could not be a more powerful reminder to the mathematical modelling world of the dangers of correlation and systemic risk.

Things change quickly. Everyone uses the same or very similar models. This results in herding behavior which itself changes the risks being modeled. New information needs to be incorporated immediately into models to avoid becoming the victim of the next problem. The last one to update their risk model will be the one who accepts the worst risks when the leaders have already fled the market.

So even if you recently did get lost, do not blame your car. Fix the problem by asking for directions, looking at maps, consulting a web map site or buying an electronic navigation system. Then your car will not get lost. And understand the limitations of your car -- it is not designed to fly!

Athula Alwis is Associate Vice President & Actuary, Freedom Specialty, a Nationwide Company, in New York, and Dave Ingram, CERA, FRM, PRM, is Senior Vice President, Willis Re, in New York.

This article is based on a presentation by Athula Alwis at the annual Export Credit & Political Risk conference in London in February 2009 while he was an employee of Willis Re Inc.

Back to top