Black Swans

Sometimes seemingly unrelated conversations can come full circle and illuminate the bulb of fresh thinking. In a recent issue of F&M I penned a brief editorial about my experiences at a MORS conference. To my surprise, someone actually read it. Somewhat less to my surprise, that someone disagreed with my findings.

In the editorial I mentioned two related concerns that I had about the complete dependence of professional wargame designers on computers and the quality of research that was going into them. The first was the black box phenomenon, in which a user has to simply take the game qua training tool at face value. He has no way to know or verify that the assumptions, theories, and research in the game are valid. He may only use the interface and get a result.

The second was that some of the research I actually saw being incorporated was simply invented. Now presumably that was only a temporary situation, but, given concern number one above, how would I ever know? My basic point in the editorial was to shill for board wargames as a serious tool because they did not suffer from the lack of transparency problem, therefore could be evaluated for content as well as utilized for results.

The fellow I mentioned above who took some umbrage to my comments was Michael Ottenberg, Principal Military Operations Analyst for AT&T, currently working in the Office of the Secretary of Defense, and highly respected in the field. I was honored to find that he, in fact, had read Fire & Movement, and what is more, will be sending me a rebuttal piece for publication in the near future. I greatly look forward to that.

Mr. Ottenberg’s point of view was that the wargames that are produced in the field are highly researched and vetted affairs, and that cases where assumptions, theories, or research are simply made up don’t exist due to the levels of scrutiny involved. He and I discussed this point in some detail at a later MORS conference. Yet, another event occurred at this MORS conference that is of interest, but first I must regress a few years.

Sometime in 2005 I was sitting in a classroom listening to a lecture on how bureaucracies changed over time. To be honest, I don’t remember the lecture in enough detail to recount the theory. I can only remember thinking that whomever dreamed it up had never spent a day trying to get a piece of paper submitted at a local motor vehicles office. What was important about this lecture, however, was that I was so struck by how wrong I thought the theory was that a new one occurred to me on the spot. I coined it SHF Event Theory.

My theory was simple. Bureaucracies have a vested interest to remain unchanged. They must continually justify their existence, well beyond the period when the need for whatever public good they provide has ceased to exist. Thus, a bureaucratic institution remains monolithic, even though they exist in a dynamic environment. The longer it remains unchanged, the less relevant it becomes to the current status quo. What causes change are SHF Events. These are unpredicted, unanticipated major events that upset the system to such an extent that the inadequacy of the existing bureaucratic structure is revealed, and it is changed. Thus, the only thing that drives bureaucratic, and thus institutional change, is SHF Events.

At the time I had several other irons in the fire, so I shelved the idea for later development. Fast forward to the MORS conference where I’m sitting next to Michael Ottenberg watching a slide presentation on developing a discipline of study for wargaming, when a slide comes up showing a book called The Black Swan: The Impact of the Highly Improbable by Nassim Nicholas Taleb. This is a book that Peter Perla has been recommending wargame designers and developers read for some time, and so, like a good soldier, I went to the book store and procured a copy. Much to my surprise and amazement, here was my SHF Event Theory applied not just to institutional change, but to the entire course of history. Taleb’s position is that all of history is not driven by previous events, but rather by, what he calls, black swans, or completely unpredicted, unanticipated major events that radically upset the system (sound familiar?).

This idea poses some interesting intellectual challenges. First of all, it puts our whole research paradigm in question. In predictive analysis, we depend entirely upon past data trends to predict the future. Even when we develop systems to model larger dynamic systems, we are faced with the inadequacy of using a closed system to model an open one (reality). Commonly when we utilize data in these systems we play statistical tricks, such as logging variables to reduce the effects of trending or multicollinearity, and particularly the biasing effects of outliers. Often, we drop these outliers altogether so that the model behaves “reasonably.” But if Taleb is correct, it is these very outliers that are driving the system. The sad truth is that we have no accepted way to deal with that. There are likely to be quants (“quants" are the loving term applied to folks who do quantitative research, such as myself) who will claim that they can be accounted for, but I challenge them to show me the econometric text that contains the method. Our complete inability to predict even the likelihood of a black swan event would demonstrate the inadequacy of such techniques. The truth is that we’re still largely using very complex methods to essentially say that “it’s sunny today so it will probably be sunny tomorrow.”

I’m still reading The Black Swan, so I don’t know what Taleb’s conclusions are going to be yet, but I can take a guess. For some time I’ve said that the quantitative vs. qualitative debate in academia and elsewhere has to be the stupidest disagreement I’ve ever witnessed. The two are symbiotic. Although black swans are probably not ever going to be predictable (after all, then they wouldn’t be black swans), we can reasonably anticipate where there are vulnerabilities to them. For instance, while we could not have predicted the events of 9/11, we certainly could have anticipated our vulnerability to such an attack. The work of anticipating these vulnerabilities can only be accomplished in the context of both qualitative and quantitative methods. What’s more, I think that wargaming offers a particularly good venue for doing exactly that. In fact, I think that wargaming, if given the opportunity, would excel at this particular task.

Unfortunately, we have some serious problems of our own to overcome. Let me regale you with a personal experience to demonstrate the point. Last year we at MCS Group submitted our game Battle for Baghdad to the Serious Games Showcase and Challenge. Not only did we not place, we weren’t even allowed to compete. Why? Because Battle for Baghdad is a board game. Now we didn’t submit the game to the Serious COMPUTER Games Showcase and Challenge, and a careful reading of the entry restrictions mentioned nothing about the game having to be computerized. The implication is, of course, clear. A board game obviously is not considered a serious game. In fact, we were greatly encouraged by the panel to convert the game to computer and resubmit the next year. We’re not going to.

Battle for Baghdad is a game about various political factions (who control military and other forces) vying for control of the city of Baghdad during U.S. occupation. The entire point of the game is to be able to creatively deal with the other faction heads to negotiate or strong arm your way to your own particular factional goals, while maneuvering to prevent other factions from reaching theirs. The game simply cannot be done remotely on a computer without losing much of the point of it. All of this was completely lost on the panel. It wasn’t a computer game, so it wasn’t serious.

This is a serious case of, what I call, glass navel syndrome (you figure it out). Here is a group so completely married to a particular media that they can’t even acknowledge that some other media presentation may have something important to say; which is the same as the quant vs. qual debate. One thing I understood from Michael Ottenberg was that the games he works on and, by implication, that are generally done within the professional wargame industry, must have every assumption vetted with real data. The implication is that where data doesn’t exist, potential interactions are simply dropped or ignored. As a quant, I know that most data is incomplete, that we have relatively little data to work with on most things, and that often the data that we do have is simply crap. Therefore, games that must endure vetting to that extent must face severe limitations in their capacity to teach, but especially predict. Such must be especially the case when dealing with games that deal more with the social science end of the spectrum, as opposed to those that are modeling the impact of a guided missile.

The reality is that many of the things we seek to model in a wargame simply can’t be measured quantitatively. Battle for Baghdad is a prime example of this fact. The game relies entirely upon qualitative research, case studies, and regional history to synthesize its message. What’s more, Battle for Baghdad allows for the occurrence of black swans through creative play, which is something that will never happen no matter how many times you fire up your copy of Joint Operations.

As a community of practitioners I strongly believe that we have to abandon our particular methodology affiliations and become generalists, open to all methodologies: quantitative, qualitative, and even to black swans. Wargames are a hybrid methodology that dictates a broad spectrum approach. Not doing so leaves us vulnerable to SHF Events. And in case you’ve been wondering, it stands for Shit Hits the Fan.