Standing Inside the Outside Box

In his book Empirical Model-Building and Response Surfaces, George Box made the now semi-famous statement "All models are wrong, some are useful." In summing up mathematical modeling and statistics, this comes about as close to ground truth on the subject as possible. Aside from those of us suffering from the side effects of cool-aide poisoning (dispensaries have sprung up at universities across the country) many of us who have worked through the detailed mathematics that underlie statistics and formal modeling realize the metaphoric nature of the structure and the perilousness of the assumptions behind it. Despite that, we continue to oversell the "truth" of what we do.

We excuse this behavior with a simple expedient; give the customer what he wants. The general wants an answer, so he is provided with a point solution to a broad spectrum question. He’s happy, we’re happy, everyone’s happy. At least until they start bringing the boys home in boxes. But don’t worry, how will the bullets in those corpses ever be traced back to that product handed to the general. It’s what he wanted, after all. Conscience clear.

Not in my book.

We're accosted on all sides. In the press we continually see where someone or other has invented some new method for predicting the future (which usually turns out to be something already done wrapped in new clothing, e.g. Gourley, BMD, etc.) and the money starts to flow. Hard not to be envious of that.

When we read reports about some new wizbang method for predicting the future, our first reaction should not be one of optimistic hope and excitement. Instead, our first impulse should be to reach for our crap detector and to start figuring out what it's doing, what context it works within (and what context it doesn’t), and then to firmly place it within our hierarchy of tools in such a way that it is optimized to perform only on the issues on which it is effective.

Our other challenge comes in the form of poorly defined problems being posed by the technically incompetent (who, unfortunately, usually hold the checkbook). "If I could predict what the terrorists were going to do next, I could stop them and better defend the country." How many times have you seen some formulation of that sentence/sentiment?

It's mostly our fault. In the mad dash for social sciences to gain respectability by donning the trappings of the scientific method, we've made promises we can’t actually keep. Of the list of things that all good models are supposed to do, predict is one of them. This comes right out of formal methods 101, and it's based upon a gross misunderstanding of reality. As Stephen Downes-Martin has so effectively pointed out, F=MA is a very effective model at predicting things, except that it is not a social science model.

In quantitative academic literature on civil wars, insurgencies, and non-state actor violence we see a broad range to empirical evidence used to back all sorts of claims. One of my favorites is the notion that suicide bombings are primarily the result of occupations in foreign countries. As a result of this correlation, Robert Pape has recommended rather broad policy prescriptions that have gotten traction in some circles. Except that the correlation is so obviously spurious and driven by a particular political perspective. Beyond that, try and reproduce the results. You can't. Pape won’t share his data (I’ve asked him more than once for it, and so have other professors. Pape won’t even respond to the request). Another favorite is the work of James Fearon and David Laitin out of Stanford. They claim to have empirical evidence that the presence of mountainous terrain is a primary determinant for insurgency. This author has gotten their data and run their regressions. If the original R Squared value of low twenty percent doesn’t give you pause, then when I tell you that if you take the lagged dependent variable out of the model the whole thing collapses it should at least raise an eyebrow.

But beyond the shortcomings of quantitative work being done in the social sciences (there is good work out there, it’s not a complete calamity), we have a problem with approach. "If I could only predict what the terrorist is going to attack next" is really not the problem we need to be thinking about. What if we instead ask "what can the terrorist do that can really hurt me?" and it’s corollary "what are the responses to bad events that are most appropriate?" These are questions we can actually answer with some degree of confidence, and by doing so, mitigate against their occurrence. Similarly, if I ask "what COA will defeat whatever Red does against it?" I have an intractable problem. But instead what if I ask "How can Red defeat my COA?" then I have a problem that is more tractable by not relying on attempting to predict what he might do, and instead causing me to think about making my own actions more flexible and responsive to potentially changing situations. The wargame response to the latter question is not to have accurate predictive PolMil models informing the game, but rather to harness the creativity of a large number of red cells as they attack a single Blue COA.

By understanding and cataloging our own vulnerabilities, we don’t get a tool to predict, but we do get a tool that anticipates and mitigates. But there’s no panacea here. We live in a sea of risk, and there’s no real way to create certainty out of chaos. In trying to form a future based upon predicted outcomes we create a system highly vulnerable to black swan events that could easily destroy the very future we wish to create. But by anticipating events by understanding what we are vulnerable to and then mitigating against them, we create a ship that can weather most storms.