Ruminations on Path Dependence and Archimedes

A few days ago, I collided at a dead run with one of the hazards--one might go so far as to call it one of the nightmares--of working in the business of analysis. Having conducted a thorough assessment of a problem, applying every tool available and appropriate, carefully weighing every assumption, considering every possible solution, and expending significant manpower, a methodological course of action emerged that incontrovertibly increases both effectiveness and efficiency in the process under investigation. The results were presented to the customer, and the answer came back: "Thanks for the great effort. We love the solution you've given us. It's brilliant! We're going to stick with our current process."

I've been in this business for nearly half my life, and I shouldn't be surprised by this outcome. In this instance, though, I was floored. Then, after a few hours of frustrated muttering, I had an epiphany (or calmed down enough to recognize the dynamics in play). There was a reason for the outcome, and understanding the reason both soothed my frayed analytic psyche and helped me to understand the way ahead. That reason? A favorite concept from economics: path dependence.

So, what is path dependence? The simplest possible expression of the concept is the statement that history matters, but this doesn't do the idea justice. In path dependence, history matters in particular mathematical ways and with particular mathematical and practical consequences. Essentially, though, it boils down to the notion (contrary to classical economics) that positive feedback mechanisms create multiple stable equilibria in dynamic social, technological, and economic systems. (Scott Page describes multiple forms of path dependence in a wonderful essay on the subject, but the naive notion given here is sufficient for most purposes.)  Accidental perturbation (or human decision) at critical junctures may nudge a system's trajectory toward an outcome that is unforeseen and sub-optimal (as many, most, or all of the stable equilibria may be). If one is in a tautological mood, one might then say that these stable equilibria are difficult to escape, but the point is that positive social, technological, and fiscal returns incentize stability.

Examples of path dependent behaviors are everywhere, if we take the time to think about it. Arbitrary coherence--the idea that an arbitrary initial price affects the long-term price irrespective of intrinsic value--is a classic path dependence phenomenon. A favorite military example is low-observable technology in the Air Force's force structure. A little stealth incentivizes improvements in adversary radar that encourages more stealth, and positive feedback takes over. This is a gross oversimplification of the many social factors in play, and it leaves out a number of positive feedback mechanisms, but it illustrates the idea. (I should also note that the situation that smacked me in the face with 'thank you for your interest in national defense' had nothing to do with this example or with questions of force structure.)

So, why am I comforted by this idea? There are two reasons.

First, it helps to know that adherence to the status quo is in some sense and in some cases independent of whether the status quo is the best position available. This realization provides a partial response to the question, "What did we do wrong?" (The other parts of that response involve a close examination of the analysis to make sure we hadn't missed something critical.) We are where we are for reasons of history, and the equilibrium is stable for any number of systemic reasons. We can be both right and rejected.

Second, it gives me hope and encourages me to continue laying the intellectual groundwork for the position. Path dependence and positive returns depend on context, but a fundamental characteristic of complex adaptive (social and technological) systems is that the context is always in motion. There may come a time when the status quo is no longer attractive. There may come a time when the costs of achieving escape velocity are within reach. There may come an opportunity, and if the work is done in anticipation of that opportunity we'll be in a position to exploit it. In the end, much of the great analysis we do is anticipatory. That is, if we wait for the question to be asked (or the opportunity to prevent itself), the analysis to support the decision will almost certainly be too late. Anticipatory analysis creates the lever, and we simply wait for a fulcrum to present itself.

No matter how brilliant an idea may be, no matter how solid the recommendation, it won't always change the world. And the reasons it won't are not always rational or in our control. But that's not a reason to not do the work. In some ways, good work is its own reward (for me). More importantly, doing the work is how we change the context and prepare for the opportunity to move the world.