Ladies Tasting Tea, Determinism, and Comfort With Contingency

Karl Pearson, Public Domain

At the encouragement of a statistician friend of mine, I recently read David Salsburg's book, The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century. This was a delightful read in a hundred ways, especially in it's effort to humanize the statistical luminaries of the 20th century, and I highly recommend it. That said, there was one idea--it seems, in fact, as if this idea is a main objective for writing the book--that left me wondering whether the author had taken leave of his senses. Bear with me while I work through this idea and it's implications for analysis.

Here is what Salsburg has to say:

Over a hundred years ago, Karl Pearson proposed that all observations arise from probability distributions and that the purpose of science is to estimate the parameters of those distributions. Before that, the world of science believed that the universe followed laws, like Newton's laws of motion, and that any apparent variation  in what was observed were due to errors ... Eventually, the deterministic approach to science collapsed because the differences between what the models predicted and what was actually observed grew greater with more precise measurements. Instead of eliminating the errors that Laplace thought were interfering with the ability to observe the true motion of the planets, more precise measurements showed more and more variation. At this point, science was ready for Karl Pearson and his distributions with parameters.

The first thing that troubles me is a strange, Platonic (metaphysical) realism in the statement that "observations arise from probability distributions" as if these distributions were actual things rather than mathematical characterizations of the observed probabilistic behavior of actual things (or the probabilistic observations of the deterministic behaviors of actual things). At the risk of labeling myself some sort of radical nominalist, this seems an odd and difficult pill to swallow. This does not mean, however, that Pearson's effort to shift our attention from individual observations to more fundamental concepts of parameters that describe the totality of observations is problematic. It only means that the notion of an unobservable pure distribution of which we observe only imperfect shadows is an infelicitous representation of Pearson's work. So, this is not the major objection to Salsburg's purpose and point, but it is the (shaky) foundation on which he proceeds to build his house, and it is the house that presents the more significant problem.

Salsburg seems to fundamentally misunderstand the concept of Kuhn's paradigm shift, the accumulation of anomalies (i.e., the growing differences between observations and expectations in planetary motions, in this case), and the relationship between these phenomena and the philosophical positions of determinism and probabilism. (Incidentally, he also seems to reify this model of science, but that's a problem for another day.) The increasing variation from prediction lamented as a flaw of worldview is in fact such a flaw, but not a flaw in determinism as such but rather a flaw in the model of planetary dynamics as derived from Newton's laws of gravity and motion. The model of planetary motion was wrong--as models are--and this manifested more clearly once methods of measurement improved. This leads not to a revolution in probabilistic worldviews but rather a revolution in the model of gravity and planetary motion (i.e., relativity). So, while the errors of measurement are probabilistic, the source of changing error is systemic. These are different, and need to be treated differently (one statistically and one deterministically).

Henri Poincare
Public Domain

That means there is no fundamental disagreement between the worldviews--probabilistic and deterministic--that Salsburg sets in opposition to each other (at least as he's characterized them ... there are deeper philosophical divides, but Salsburg is really a determinist in disguise). Henri Poincaré writes in Chapter IV of The Foundations of Science that "we have become absolute determinists, and even those who want to reserve the rights of human free will let determinism reign undividedly in the inorganic world at least." He then goes on to discuss in detail the nature of chance, or "the fortuitous phenomena about which the calculus of probabilities will provisionally give information" and describe two fundamental forms of chance: statistically random phenomena and sensitivity to initial conditions. He writes:

If we could know exactly the laws of nature and the situation of the universe at the initial instant, we should be able to predict exactly the situation of this same universe at a subsequent instant. But even when the natural laws should have no further secret for us, we could know the initial situation only approximately.

Since we can know the exact condition of the universe only approximately (because we are finite, because humans have freedom of non-rational choice, becuase we are irrational and our models shape our observations, because Heisenberg dictates that imprecision is fundamental, etc.) all phenomena are thus to some degree or another functionally probabilistic for even the most determined determinist.

Carl von Clausewitz
Public Domain

The form of chance observed is then a product of the underlying dynamics and laws of the system under observation. Are we dealing with statistically random phenomena in which, when we have eliminated large and systemic errors, "there remain many small ones which, their effects accumulating, may become dangerous" and produce results attributed "to chance because their causes are too complicated and too numerous?" (The similarity to Clausewitz's discussion of friction is no coincidence.) Or are we dealing with nonlinear phenomena in which the single small error (or the butterfly flapping it's wings) yields outcomes all out of proportion to the error? Is there a structural reason for the particular distribution we see in the chance behavior? And what parameters describe these distributions?

These are important questions for analysts, with important implications. We bound our systems in time, space and scope for the purposes of tractability, introducing error. We make assumptions regarding the structure of our systems (analogous to the application of Newton's laws to planetary motion), introducing more errors. We measure, anticipate, and assume all manner of inputs to our analytic systems, introducing yet more error.

So what does this mean for us? As analysts we must everyday ask ourselves, "What errors are we introducing, what is their character, what is their structure, and how will they interact with other errors and the system itself?" And we must become comfortable with facing these uncertainties (something occasionally difficult for those of us with too many math classes under our belts).

Reading, thinking and writing about something for analysts to consider.

Data Worship and Duty

If you spend more than a few minutes working as an analyst--operations, program, logistics, personnel, or otherwise--it is almost inevitable that some wise military soul will offer trenchant historical lessons about undue trust in analytics for decision making derived from the performance of Robert McNamara as Secretary of Defense. Too often, these criticisms are intended to deflect and deflate criticisms and conclusions of analysis without addressing the analysis itself (an ad hominem approach without so much of the hominem). But that doesn't mean there aren't common mistakes made in the conduct of analysis and worthwhile lessons to be learned from McNamara.

This short article from the MIT Technology Review is a bit old, but it also makes a number of useful points. The "body count" metric, for example, is a canonical case of making important what we can measure rather than measuring what's important (if what is important is usefully measurable at all). Is the number of enemy dead (even if we can count it accurately) an effective measure of progress in a war that is other than total? So, why collect and report it? And what second-order effects are induced by a metric like this one? What behavior do we incentivize by the metrics we choose, whether its mendacious reporting of battlefield performance in Vietnam or the tossing of unused car parts in the river? 

There's something more fundamental going on in the worship of data, though. We gather more and more detailed information on the performance of ours and our adversaries' systems and think that by adding decimals we add to our "understanding." Do we, though? In his Foundations of Science, Henri Poincaré writes:

If we could know exactly the laws of nature and the situation of the universe at the initial instant, we should be able to predict exactly the situation of this same universe at a subsequent interest. But even when the natural laws should have no further secret for us, we could know the initial situation only approximately. If that permits us to foresee the subsequent situation with the same degree of approximation, this is all we require, we say the phenomenon has been predicted, that it is ruled by laws. But this is not always the case; it may happen that slight differences in the initial conditions produce very great differences in the final phenomenon; a slight error in the former would make an enormous error in the latter. Prediction becomes impossible and we have the fortuitous phenomenon. 

Poincare is describing here what would later be dubbed the butterfly effect for nonlinear systems (with the comparison to predicting the weather made explicit in a later chapter). In systems such as these, chasing data is to pursue a unicorn and the end of the rainbow. Rather, it is structure we should chase. Modeling isn't about populating our tools with newer and better data (though this may be important, if secondary). Rather, modeling is about understanding the underlying relationships between the data.

We often hear or read that some General or other should have fought harder against the dictates of the McNamara Pentagon, but one wonders if perhaps such a fight is also the duty of a military analyst.