Analysis and the Innovation Imperative

The defense community is recently (and not-so-recently) awash in calls for innovation to ensure that we are militarily postured to meet future challenges and possess the agility necessary to turn toward those challenges we did not anticipate.

For example, we have the Pentagon's Third Offset Strategy and Air University's call for "Airmen to offer innovative solutions to address problems facing the Air Force in a time of increasingly daunting global and fiscal challenges." We also see what B. J. Armstrong describes as a small movement

... growing across the defense community which realizes that the challenges of the new century are going to require innovative and creative solutions. Parts of this movement, inspired from the junior ranks of our services, look to embrace the ideals of innovation and entrepreneurship from the business world. These dedicated women and men recognize that the budget, manpower, and resource challenges in a post-war drawdown mean that new ways of doing things will be required.

The Red Queen and Alice

The ways in which this small (but growing) movement manifests are legion, including formal organizations such as the Defense Entrepreneurs Forum and  CIMSEC, online venues for the exchange of ideas (The Strategy Bridge, War Council, The Constant Strategist, etc.), and the growth of peer-mentoring communities of practice (whether on the model of Scharnhorst's Militarische Gesellschaft or something less formal). I am a huge fan of these efforts, and I think the powers-that-be should support them in any way they can. They will, I believe, be a critical part of creating innovation going forward.

Why should the powers-that-be support such an effort, though, especially in the face of limited resources? In times of plenty, we have the resources but lack the imperatives for innovation. In times of need, we lack the resources but the necessity is much more clear. This is the conundrum.

That innovation (or at least the ability to innovate) is necessary in the abstract seems self-evident (and if not self-evident then compelling arguments can be made for its necessity). The world changes around us. There are adversaries, potential adversaries, allies, and potential allies all about us who all seek to increase their own relative advantage, and every change they make (whether they intend it or we like it) affects the calculus of our own continuing advantage. We are all trapped in a Red Queen Race, and survival depends upon our capacity to find new solutions to new problems, more efficient solutions to old problems, and the creation of new problems for our competitors.

But innovation for the sake of innovation is a mistake. The folks at the Havok Journal make a fine argument that not all "outside the container" thinking is worthwhile. Paraphrasing egregiously, they claim that the container was built by someone for some reason, sometimes the container is just fine, and those looking to operate outside its confines often "don’t really understand the fundamentals of [their] profession and don’t want to take the time to learn." (The full article is well worth reading, and I highly recommend it. My self-serving paraphrasing does not do it justice.)

Art, Science, and Engineering

There is a real and important tension here, described in the fields of expert systems and evolution as a trade-off between exploration and exploitation. If a particular strategy works, it makes sense to exploit that strategy. But we should also explore the strategic environment for approaches that work better and for changes in that environment that might compromise the approaches we've exploited so successfully. A nice metaphor for this tension has been described elsewhere in this space as the interaction between art, science, and engineering. (I won't lie. The parallel with Clausewitz's wonderful trinity and the chaotic dynamics of the three-body problem add an undeniable attraction to the model.)  The balance between these is the difference between evolutionary success and failure.

So, why post this thought in a forum all about analysis? I thought you'd never ask.

We (analysts, mathematicians, statisticians, modelers, computer scientists, etc.) are a part of this environment, too. Our worldviews and tools must be no less adaptable than those of the doctrine writers, planners, and strategists of the world. More important (and a little frightening) is the possibility of reified and static analytic ideas becoming framing concepts for the rest of the strategic world.

Optimality is a favorite and appropriate example. For what do we optimize force structure? A world and a worldview. What is the consequence if either situation fails to meet our assumptions? Something less than optimal. How do we minimize the probability that failures of optimality (or analysis in general) are catastrophic? Welcome to the problem of framing analysis in a way that supports the decision needs of leadership in a way that meets the needs of today, tomorrow, and the future. And welcome to the need for innovation in our own community.

So, what does this all mean? How are art, science, engineering, innovation, expertise, exploration, and exploitation to be managed? Not surprisingly, I have some thoughts on the matter:

1. Expertise matters. The problems we face as military analysts are trivial in neither their costs not their consequences. If military operations research is to synthesize inputs and techniques from diverse disciplines (math, statistics, economics, computer science, etc.) expertise in those disciplines is important. Destruction and creation can produce positive results by accident, but will produce innovation far more reliably if the agents involved really understand the underlying philosophical and technical principles.
2. Diversity is more than important. Without diversity of experience and expertise, it is difficult to find and exploit the skis, handlebars, outboard motors, and tank treads in our experience to create effective snowmobiles. This is an interesting concept for military operations research. Ours is an academic discipline that is inherently interdisciplinary, but an interdisciplinary field will always struggle with sufficient isolated expertise to facilitate effective interdisciplinary exploration.
3. Tolerance for individual failure is critical. Evolution and innovation are bottom-up processes, and there must be room for both exploration and exploitation. In an ecosystem, failure is fatal, but that is a system relying on chance to create the necessary genetic and phenotypic variations that facilitate adaptation to changing environments at a population level. The loss of individuals is not important, since the adaptation is neither social nor volitional. On the other hand, analytic innovation is a volitional act by rational social agents, and intellectual variation leading to dead ends cannot lead to deadly individual ends (unless the objective is abject conformity). This is the primary purpose of a community/institution in the context of organizational innovation. It exists to exploit the known, incentivize exploration, and protect the explorers from censure. We need to be free to disagree, argue, and explore.

For our community these observations result in some imperatives for action. Simply put, a structure that prizes expertise and diversity in that expertise (not expertise in that diversity) while facilitating exploration of new ideas and analytic approaches is what we should seek. 

Commission for Military
Reorganization at Konigsberg, 1807

This blog notwithstanding, I'm left wondering why there aren't analytic equivalents of CIMSEC, The Strategy Bridge, War Council, The Constant Strategist, the Militarische Gesellschaft, etc. I'm left wondering why so many of our analysts attend the same school (singular) to receive the same graduate education. I'm left wondering if we will find a way to be relevant in the face of a future that looks decidedly unlike the world in which our "discipline" emerged.

The Right Answer?

It is strange how serendipity occasionally intervenes to link multiple lines of thinking and the conversations that go along with them and thinking on a subject crystallizes. In June 2014, Harvard Business Review published an article titled "Why Smart People Struggle With Strategy." The piece begins thus:

Strategy is often seen as something really smart people do — those head-of-the-class folks with top-notch academic credentials. But just because these are the folks attracted to strategy doesn't mean they will naturally excel at it. The problem with smart people is that they are used to seeking and finding the right answer; unfortunately, in strategy there is no single right answer to find. Strategy requires making choices about an uncertain future. It is not possible, no matter how much of the ocean you boil, to discover the one right answer. There isn't one. In fact, even after the fact, there is no way to determine that one’s strategy choice was “right,” because there is no way to judge the relative quality of any path against all the paths not actually chosen. There are no double-blind experiments in strategy.

When this crossed my digital desk this week, I was reminded of a slew of recent articles on the Service Academies (here, here, and here) and a great discussion that ensued on The Constant Strategist over the questions raised in the first of the linked articles. Here, two questions matter: What was the crux of the original issue and where have I landed in the overarching questions?

The initial online debate in this exploration centered on the curriculum most appropriate to the education of military officers. What should be the emphasis in a liberal education intended to develop them deliberately? There are two--one might call them adversarial--camps in this debate centered on the relative importance of the sciences and the humanities. I have always found myself standing athwart the apparent chasm between these two positions. As a military analyst with too many graduate degrees in math, I have enormous sympathy for the technical side of this debate (perhaps selfishly, since another position would invalidate much of my education and professional life). But I began my life as a student of English Literature and I spent a formative interlude as a graduate student in military history and strategy, and I know the technocrat's approach to conflict and strategic planning is problematic. But since it is hard to ask that everyone know everything, what is the right answer?

In the end, I think appropriate diversity is the answer, at every unit of analysis from the individual to the population. The trick is to ensure both expertise in the population (i.e., someone somewhere has spent a life studying the topics of interest) and familiarity in individuals (i.e., we have all studied enough of the other that we can speak a common language and seek useful metaphors). That means we should encourage and incentivize both expertise and exposure in a variety of disciplines--math, statistics, physics, chemistry, engineering, history, anthropology, theology, literature, etc. But there's a catch.

In our world of military operations research analysts so well trained in seeking optimality the idea of external familiarity really matters, and this is why I'm writing. There is a distressing and problematic bias in the technical fields toward the existence of a "correct" answer, not unlike the assertion in the Harvard Business Review piece. It's how we're trained. In our education, there exists a provably true answer (at least within the constraints of our axiomatic systems) to most of the textbook questions we answer as we learn our trade. This is, in fact, part of the reason for my own shift once upon a time between literature and math as a chosen field of study. Certainty has a certain comfort and led to fewer arguments between student and teacher.

I do NOT want to encourage anyone to not study the sciences, operations research, or (my own love) mathematics. And I do NOT want to encourage avoidance of the the less technical disciplines. Rather, what I want to encourage is an appreciation of contingency in the application of the technical disciplines and a rigor in the application of the non-technical disciplines, especially the context of militarily relevant questions.

Why? Fundamentally because our models and computational tools are by definition rife with assumptions. What if one or more of those assumptions are wrong? What if we forget some minor idea that turns out to be critical? What if our axioms don't work? (I'm looking at you, Economics.) What if optimality itself is a chimera?

For us, reading history of various kinds and actively considering the question of how our forebears (analytic and otherwise) erred is perhaps a useful remedy. The problem is not that one is smart or not. The problem is how and what one studies and with what intent.

The proverb says, "Iron sharpeneth iron." A suggested circular addendum to this wisdom is that the humanities temper the steel of the sciences while the sciences sharpen the analytic edge of the humanities.

Seeking Truth

The last few posts I've penned for this forum (here and here) have danced around the edges--and occasionally jumped up and down on--the notion that we humans are flawed, cognitively compromised, and subject to some intrinsic constraints on our ability to see, understand, communicate, and act on the truth. Though this is not a new soapbox, I hadn't realized that this notion had taken over my writing and become as strident as it had. Then a good friend asked a simple question, and I found myself wrestling with the consequences of the human cognitive silliness on which I've been recently focused and what it means for truth in general and, perfectly apropos of this forum, truth in our analytic profession.

So, what poser did my wise friend propose? He offered three alternative positions based on the existence of truth and our ability to know it:

1. There is a truth and we can grow to understand it.
2. There is a truth and we cannot understand it.
3. There is no truth for us to understand.

(Technically, I suppose there is a fourth possibility--that there is no truth and we can grow to understand it--but this isn't a particularly useful alternative to consider. As a mathematician and pedant by training and inclination, though, it is difficult to not at least acknowledge this.) 

The question is then where I fall on this list of possibilities. It's an important question, if for no other reason than where we sit is where we stand, and it becomes difficult to hypocritical to conscientiously pursue an analytic profession if we believe either two or three is the case. Strangely, though, I found this a harder question to answer than perhaps I should have, but here is where I landed:

At least with respect to the human physical and social universes with which we contend, there is an objective truth that is in some sense knowable and we, finite and flawed as we are, can discover these truths via observation, experimentation, and analysis.

In retrospect, my position on this question should have been obvious. I've been making statements that human cognition is biased and flawed, averring that this is a truth, and I believe it to be one. We can observe any number of truths in the way humans and the universe we occupy behave. I find, on refection, though that there is a limit to this idea. Specifically, we can probably never know with precision the underlying mechanisms that produce the truths we observe. We may know that cognitive biases exist and we may be able to describe their tendencies, but (speaking charitably) we are unlikely to ever have an incontrovertible cause-and-effect model to allow us to interact with and influence these tendencies in a push-button way.

So, the trouble I have with truth is that we apply truth value to the explanatory models we create. Since these models are artificial creations and not the systems themselves they must, by definition, fail to represent the system perfectly. Newtonian theories of gravity based on mass give way to relativistic theories of gravity based on energy. In some ways one is better than the other, but neither is true in a deep sense. Our models are never true in the larger sense. They may constitute the best available model. They may be "true enough" or " right in all the ways that matter." But both of these conditions are mutable and context-dependent. In a sense, I find myself intellectually drawn to the notion that truth in the contexts that matter to us professionally is an inductive question and not a deductive one.

In the end, I'm actually encouraged by this reflection, though the conclusion that models are and must be inherently flawed results in some serious consternation for this mathematician (soothed only by the clarity with which mathematicians state and evaluate our axiomatic models). I understand better what I'm seeking. I understand better the limitations involved. And, at the risk of beating a dead horse, I am more convinced of the need to put our ideas out in the world. This reflection might never have taken place if not for Admiral Stavridis and his injunction to read, think, and write.

Belief, Dissonance, and Difficulty in Analysis

René Descartes

In a recent conversation on The Constant Strategist, an acquaintance offered the insightful observation that a lot of reading is really important, but a little reading is actually harmful as people may be taken with the belief that the one book they've read on a subject is the last word rather than what should be the first word on the subject. In this case, the particular subject was the importance of cultural and contextual understanding as an important (if not necessary) prerequisite for effective strategic engagement with another society or nation. But this thought led to a broader reflection on the theory of knowledge (or at least one aspect of the theory of knowledge), cognitive dissonance (with all its myriad side effects), and what these two things mean for analysts.

Baruch Spinoza

Several years ago, I first read a marvelous paper by Daniel Gilbert titled "How Mental Systems Believe" (that you can find online here). The gist of this article is a contrast between theories of learning described by René Descartes and Baruch Spinoza. In a nutshell, Descartes believed that one must first comprehend an idea before one can assess the truth of that idea. In other words, "comprehension precedes and is separate from assessment." Spinoza, on the other hand, dismissed the Cartesian distinction between comprehension and assessment, arguing "that to comprehend a proposition, a person had to implicitly accept that proposition." Only once an idea has been comprehended and believed is it possible to engage in a secondary and effortful process to either certify the idea as true or actively reject its truth. The evidence presented by Gilbert suggests that human beings are, for any number of reasons, Spinozan systems rather than Cartesian systems (or, at the very least are not Cartesian and may be some other type of system in which acceptance is psychologically prior to rejection).

This is all very interesting, but why does it matter? Perhaps the most important answer to that question is an oddity of human cognition commonly known as cognitive dissonance, the mental stress that results from holding "two or more contradictory beliefs, ideas, or values at the same time or" confronting "new information that conflicts with existing beliefs, ideas, or values." How do humans respond to cognitive dissonance? Robert Jervis has a good deal to say on the effects of cognitive dissonance in the milieu of international relations, noting that dissonance "will motivate the person to try to reduce dissonance and achieve consonance" and "in addition to trying to reduce it, the person will actively avoid situations and information which would likely increase the dissonance" so that "after making a decision, the person not only will downgrade or misinterpret discrepant information but will also avoid it and seek consonant information." This gives us such favorite phenomena as the Dunning-Kruger Effect (where the uninformed and unskilled rate their knowledge higher than is objectively accurate), the Backfire Effect (where in the face of contradictory evidence beliefs get stronger rather than weaker), and oh-so-many-more. So, if we must accept as true a concept if we are to understand it, as Spinoza indicates, subsequent rejection of the newly-learned concept is not just effortful but in some sense super-human. This means that reading more may not be useful, and for an inveterate reader this is disheartening. But ...

In another recent post, I raised the idea (shamelessly copied from an analyst far more insightful than I) that the essence of the analysis profession is to understand things and explain them to others. If the very act of learning and understanding drives us to error, though, what are we to do? Does this mean we should throw up our hands and abandon the search for truth? Of course not. But when my acquaintance suggested that we read more, he was only half right. We must absolutely read and study more. But we must also:

• Actively seek out positions different from our own. This includes red-teaming ourselves and exploring the results if each, every, and any combination of our assumptions are wrong. Since, by definition, each assumption we make must be necessary for planning or analysis, changes in those assumptions should change our analysis (else we would state them as facts and not assumptions) and generate a better view of the decision space.
• Train our analysts (and ourselves) as early and as constantly as possible that the mental models we have of the world are themselves assumptions, and then refer to the previous point. This should go a long way toward mitigating the Law of the Instrument (when I have a hammer, problems seem to resemble nails).
  
• Take nothing personally in the search for truth on which no one among us has a monopoly. 

There are probably other things to do, but this seems a good start. The natural question, then, is how we go about doing these things. One answer is simple, but (to shamelessly appeal to the authority of Dead Carl), "Everything is very simple in war, but the simplest thing is difficult." 

I submit that the first, second, and third steps in this journey of a thousand steps are, in the words of a favorite maxim from ADM Stavridis, to read, think, and write. Reading widely brings us new ideas, providing new positions, information, and perspectives (if we consciously seek non-confirmatory writings). Thinking is all about taking in the new information and new models, acting on the assumption that our own might be wrong, and looking for new and informative results. Writing facilitates both of these by putting our thoughts out in the world where they are subject to criticism from those not subject to our own biases, and it is these contradictory views we must learn to cherish since it it easy to find agreement (via confirmation biases if in no other way) but hard to find and use constructive disagreement. Public writing is a way (though not the only way) to find this input. (A disciplined red team can do so as well, as can other well meaning and trusted colleagues.) The final injunction, to take nothing personally, is important. This is an iterative process. If we read, think, and write we start on the right (write?) path, but if we then allow offense to drive use from the debate we will lose the gains we seek and must have.

Vincit omnia veritas ... but analytic truth is first a foe to conquer.

Data Worship and Duty

If you spend more than a few minutes working as an analyst--operations, program, logistics, personnel, or otherwise--it is almost inevitable that some wise military soul will offer trenchant historical lessons about undue trust in analytics for decision making derived from the performance of Robert McNamara as Secretary of Defense. Too often, these criticisms are intended to deflect and deflate criticisms and conclusions of analysis without addressing the analysis itself (an ad hominem approach without so much of the hominem). But that doesn't mean there aren't common mistakes made in the conduct of analysis and worthwhile lessons to be learned from McNamara.

This short article from the MIT Technology Review is a bit old, but it also makes a number of useful points. The "body count" metric, for example, is a canonical case of making important what we can measure rather than measuring what's important (if what is important is usefully measurable at all). Is the number of enemy dead (even if we can count it accurately) an effective measure of progress in a war that is other than total? So, why collect and report it? And what second-order effects are induced by a metric like this one? What behavior do we incentivize by the metrics we choose, whether its mendacious reporting of battlefield performance in Vietnam or the tossing of unused car parts in the river? 

There's something more fundamental going on in the worship of data, though. We gather more and more detailed information on the performance of ours and our adversaries' systems and think that by adding decimals we add to our "understanding." Do we, though? In his Foundations of Science, Henri Poincaré writes:

If we could know exactly the laws of nature and the situation of the universe at the initial instant, we should be able to predict exactly the situation of this same universe at a subsequent interest. But even when the natural laws should have no further secret for us, we could know the initial situation only approximately. If that permits us to foresee the subsequent situation with the same degree of approximation, this is all we require, we say the phenomenon has been predicted, that it is ruled by laws. But this is not always the case; it may happen that slight differences in the initial conditions produce very great differences in the final phenomenon; a slight error in the former would make an enormous error in the latter. Prediction becomes impossible and we have the fortuitous phenomenon. 

Poincare is describing here what would later be dubbed the butterfly effect for nonlinear systems (with the comparison to predicting the weather made explicit in a later chapter). In systems such as these, chasing data is to pursue a unicorn and the end of the rainbow. Rather, it is structure we should chase. Modeling isn't about populating our tools with newer and better data (though this may be important, if secondary). Rather, modeling is about understanding the underlying relationships between the data.

We often hear or read that some General or other should have fought harder against the dictates of the McNamara Pentagon, but one wonders if perhaps such a fight is also the duty of a military analyst.

Know your history ...

A friend who knows my leanings toward math and statistics -- and who understands my professional inclination to read, study, and apply them to military problems -- recently sent me a link to a wonderful article from The Economist, "They also served: How statisticians changed the war, and the war changed statistics."

Aside from the laudatory mention of George Box, whose assertion that "all models are wrong, but some are useful" has done more damage to the profession than any other single statement, this should be essential reading for the members of our smallish profession. Off the top of my head, I can think of at least two other works (other than the marvelous titles already described on this blog as "essential reading" and "books of interest") that should be part of our essential education as military analysts:

The Science of Bombing: Operational Research in RAF Bomber Command by Randall T. Wakelam. Much of our professional identity as a community comes from the mythology of operations (or Operational) Research and its application to the problem of civilizational survival in the Second World War. It seems a good idea to read the actual history of the people, techniques, politics, decisions, and decision makers involved in that history.

Thinking About America's Defense: An Analytical Memoir by Glenn A. Kent, David Ochmanek, Michael Spirtas, and Bruce R. Pirnie. Whether it's the mathematical techniques, the influence of political/historical context on problems of interest, or something more personal, this is an important work for military (especially Air Force) analysts.

There are more, of course. It's hard to tell what the next problem posed to a military analyst will be, so our educations must be necessarily broad. The study of mathematics, statistics, PPBE, doctrine, military history, international relations, leadership, management, theories of innovation, etc., are all important. In this case, though, the question is about the central and defining history of our communal story, our mythology.

What else should we read?

The Analytic Profession in One Tweet

Last week, a blog post by an Army strategist appeared on The Bridge (a marvelous blog that I highly recommend to military professionals of all stripes) that posed the following question:

"How would you define the art and science of our profession in one tweet?"

In this case "our profession" referred to the profession of arms, and the author put forward a compact solution with an attendant explication of his reasons that answered the challenge nicely. There is, I think, more than a little value in an effort like this one, and cutting away the chaff and getting to the heart of who we are, what we do, and why we do it is more than just an interesting intellectual exercise. If done well, it provides a clear and memorable vision that communicates to those on the outside what we do and to those on the inside why and how we do it (whatever "it" might be), in this way creating a professional community centered on the vision. This clarity of vision then has any number of second-order effects on prioritization, training, recruiting, etc., and the effort to create it can pay incredible dividends.

As a member of more than one professional community, though, this line of thinking led me to wonder, "How would you define the art and science of our military analytic profession in one tweet?" I frequently use the phrasing below when discussing the career field among the analysts with whom I work, though I can't claim credit for its composition. Those who know Mike Payne will recognize it and have likely been part of the ongoing conversation that led to it, but the words are his:

"Analysts learn how things work and explain it to others, usually in relation to other things and often quantitatively."

This definition (with 22 characters to spare) captures several critical characteristics of the analytic profession. 

1. It is general. In many cases, we don't have the luxury to consider ourselves as ISR analysts, force structure analysts, operational assessment analysts, etc. Rather, our particular skills will be applied to whatever question is relevant to leadership. 
2. Learning how things work is interesting as a standalone activity, but productive of nothing. Communicating the things we learn to those responsible for making decisions is a critical element of who we are as a community. 
3. Not all analysis is quantitative. There are some tools available to the community of military operations research professionals (mathematical, simulation, etc.) that are in some sense unique, but these are not a sine qua non for analysis. Consider, for example, the analysis given by Graham Allison and Philip Zelikow in their iconic book Essence of Decision. Nary an equation is to be found, but it's difficult to dispute that they are seeking to understand a system and explain it to others who will make decisions. Analysis is something done by analysts, and it is independent of the tools used (except for tool between the analyst's ears). 
4. The systems we study are not isolated, and understanding how they are coupled to other systems is vitally important, both to understand the constraints and restraints imposed by the environment and to illuminate non-proximate effects that may result from changes in the system under investigation. 
5. It's worth noting what this definition does not do. The word "answer" does not appear, for example. Most questions of interest, do not have clean and precise answers, for example, or they have multiple answers that each have merits making them equally palatable but qualitatively different. So, it is generally not possible for analysis of an interesting problem to produce a single, incontrovertibly trues, and perfectly optimal answer. Thus, we explain to senior leaders how the system works to help them better understand the decision space before them, but we rarely provide answers and to chase these chimerae is ... problematic.

That's my 140-character contribution.
Merf

PS ... There's a fairly robust discussion of this question on Facebook among some of the participants in this thread. You can find it here.