Friday, November 28, 2008

Black Swans

As I am away from my personal Difference Engine today, and yet not without access to a Difference Engine connected to the Aethernet, I pondered a topic about which I could write.

It occurred to me that uncertainty plays a large role in everyone's life, yet individuals seem powerless to come to grips with how to evaluate uncertainty. Even corporations that employ very smart people whose job it is to think about how to evaluate risks often have blind spots on the subject. Insurance companies regularly insured large swathes of hurricane-prone cities. The firms understood inidividual risks well, and used the fact that those risks tended to be independent of one another - the odds my house burns down does not depend on whether your house, two blocks away, burns down - to calculate the premium needed to make those risks profitable. Yet the firms failed to understand that certain kinds of risk are highly correlated: if the levees holding back Lake Ponchartrain in New Orleans fail, all the houses below sea level in the path of the water are at risk. Similarly, when the insurance company AIG insured mortgages across many banks, the firm, so the story goes, priced that insurance to reflect the odds that some mortgages would not be repaid and, under some circumstances, enough mortgages would not be repaid that some banks would fail. Yet AIG failed to understand that certain risks in the mortage market were correlated with one another - in particular, the risk that all the banks had large holdings of mortgages whose likelihood of repayment depended cruicially on housing prices continuing to increase. When the bubble burst, all those mortgages were at risk all at once.

Oops.

Which brings me to an interesting, if ultimately flawed, book, The Black Swan: The Impact of the Highly Improbable, by Nassim Nicholas Taleb (2007). In an entertaining and largely non-technical manner, Taleb notes that our ability to calculate and assess risk depends on having already observed data on that type of risk. For example, insurers have had years of experience with houses burning down, or being burglarized, or pipes bursting, or even widespread wind damage, and thus could price those types of risk appropriately. But insurers had no experience with levees failing, causing a city to flood, so they failed to price that risk appropriately. Taleb notes that those risks are inherently unknown and impossible to predict, yet, in aggregate, almost certain to occur. That is, some important but low-probability event is likely to occur to disrupt the status quo ante.

In the end, Taleb's advice is to prepare for the inevitable "black swans" - highly improbable events that cause dramatic changes - by minimizing exposure to bad events and maximizing exposure to good ones. The problem with that is (at least) twofold: first, if one had any idea of what could happen, those events would not be completely unpredictable, so deciding what constitutes a reasonable limitation of exposure to bad events is itself unknowable. Second, the implication of that way of thinking seems to suggest playing lotteries, or betting big on a single number on the roulette wheel, or ignoring your firm's trading guidelines and risking the company on a single stock market play. Some small number of people will become very wealthy this way; far more will find themselves in the modern equivalent of pauper's prison. The same thing applies on the down side: we have no way of assessing how often aircraft are destroyed in flight by spontaneous combustion of passengers, but minimizing that risk by refusing to fly seems like excessive caution.

The black swan of the title refers to the story that, prior to European discovery of Australia, Europeans "were convinced that all swans were white, an unassailable belief as it seemed completely confirmed by empirical evidence." Finding a black swan disrupted that firm conviction: "One single observation can invalidate a general statement derived from millenia of confirmatory sightings of millions of white swans. All you need is one single (and, I am told, quite ugly) black bird."

Indeed. But the lesson, it seems to me, is to be aware that one's model of how the universe works is in constant need of updating as new information flows in. Being too cocksure is, at best, a way to become a laughing stock, and, at worst, can get one killed. Beyond that, however, the lesson of the black swan is unclear.

In this reality of ours, black swans do exist - or, at least, one did exist, as evidenced by the photograph below:


What one is to make of this is far from clear. Nonetheless, the existence of a single black swan can open one's eyes to endless possibilities, such as the existence of floating inverted pyramids on top of which stand a seemingly infinite number of noobies in "away" mode.
Completely unpredictable, and, yet, there you are.

No comments: