The Zealot and His Disciples

Two years prior to the fateful board meeting, the analyst and the PM were in the same meeting foreshadowing the showdown yet to come. We both had the opportunity to be in that very same room. Let’s listen in. “I’m a believer and a zealot and I will tell you right now that I am here to influence you.” These were the words from the PM as they sat at the conference table, trying to shake off the early morning with coffee. Within 5 minutes they were both feeling like believers as well.

Were they drinking coffee or were they drinking grape Kool-Aid? His words were powerful and resonated with the sound of a corporate hero, someone who had been with the company many years, had fought many battles, and was promoted to his position on the basis of the trust he had gained from the corporate structure. “My job is to produce things of great value for the corporation.” For those of us languishing behind stacks of our paper on our desks we have heard the tales of the great ones, those in the field or the lab either landing the big contract or patenting the next invention that will earn the corporation it’s next billion dollars. And here, this morning standing before them was a legend – or so he said.

The charisma of such a hero – those who brave the depths of the commercial marketplace to take the fight to our enemies – gives us another glimpse of the excitement that is capitalism. Instead of being face to face with the empire builder in the form of someone creating and growing an organization as mentioned previously, we were now being addressed by the PM also known as a Product Manager for a corporate product line. They to are empire builders but typically they have a reason that transcends growth for the sake of the seats on an organizational chart – and they will tell you this fact. They transcend the organization because they bring value with a product – the holy grail of the organization in their minds.

Something of such value that everyone else should bow down before them, as they, guardians for the chalice, the Knights Templar by another name, The Zealot as was his nickname out side of work, enters the room. As the Zealot spoke he gave credit to those above him. “Our CEO is a visionary,” was a powerful sound bite. “The Chief has a profound vision of the future,” was another. It seemed to me, however, that those we label as visionary are the leaders who we believe are preaching closest to our own view of the world. We believe this particular PM indeed has a vision, however after a 50 minute sermon I’m not sure he knew that his vision is really about keeping investment dollars flowing into his program and not about winning a war in the corporate market place.

But this blog is not about a zealot trying to keep his program funded. If we had a nickel for every PM we’ve had to listen to ramble on about their product… rather, this essay is about truth. Maybe the PM is right; maybe he is wrong, the question is can we find out? This essay is about how analysis in an uncertain world can be used to shape decisions – and influence decisions the correct way. The promise of analysis has always been about removing bias, removing uncertainty, and delivering the right answer to a decision maker. What we are calling discovering the truth.

In reality our analysis is used to influence friends, bolster intuition, and defend decisions that were made for reasons other than what was revealed our analysis. Let’s get right down to it. Human beings are trained and equipped from preschool to make simple decisions. They can compare two things. Round pegs and square holes. Right choices and wrong choices. Good people, bad people. It’s easy to compare two things. It’s not easy to compare multiple and complex competing alternatives in an uncertain environment. It is easy to change analytic methods as we continually move between mathematical models. It’s not easy to use the right model or even use it correctly in a such a way as to produce analysis that a decision maker can use in a credible manner that can width stand significant scrutiny.

It’s easier to win friends with political rhetoric then it is to change minds with logical arguments, even with the backing of sound analysis. So friends we tell you today that we are also here to influence you – and run the risk of being exposed for our very own brand of zealotry – perhaps we are indeed worshiping at our own holy grail. But, we have a vision related to the future of analysis and its about discovering the truth. It’s about how we might go about producing useful analysis – analysis that helps a decision maker reach a conclusion for the right reason, not for the wrong reason. It’s about helping the decision maker reach the right conclusion by making them a part of the analysis – not a recipient of the analysis product. It’s about helping the human understand complex alternatives in order to make an informed decision. It’s about discovering the real truth, not the truth handed down to us by some dime store philosopher preaching his form of the Gospel with legions of Cool-aide drinking disciples handing out samples.

Good analysis will discover the truth and make it available to not only the initiated but the casual observer as well. It will be presented in clear and intuitive terms – not cloaked in rhetoric and dressed up by the sales man telling you he knows the truth and has done your thinking for you.

The next principal is simple. The Zealot always believes they must do our thinking for us – “You can’t handle the truth” as Jack Nicholson exclaimed at the end of the movie, A Few Good Men. But you can handle the truth – when the truth is revealed it doesn’t require lipstick to make it look good. When revealed the truth is self evident –it is our job as analysts to strip away the hype and find what truly matters. What are the meaningful measures that must be taken into consideration. Not what are the good things that people want to hear, taste, and smell. Tell them the truth, they can handle it, and they will appreciate it.

The Empire Builder

Most people with problems to solve are in the middle of building something. Sometimes it’s a small empire, sometimes it’s a large one. But it is, in almost every case, an empire. It is, in our country, the capitalist way. The bigger, the better. If you are not growing, you are dying. There are many clichés. Many will confess that they are building an empire, some will deny it vehemently, some will not realize they are even doing it. With very few exceptions, they all are in the process of building. Just to note, there is nothing inherently wrong with empire building and there is nothing inherently wrong with falling into one of the three categories of empire builder, depending on what you do with your empire might have ethical implications but that is not the subject of this blog, that comes later.

This blog is about pursing the truth in order to solve problems. If the empire is the problem it is quite conceivable that the builder will not be able to handle the truth, which might be the case where there is, in fact, something inherently wrong with the empire, but hopefully that is a remote case. More than likely, it is the pursuit of something other than the truth in the blind desire to build the empire that will result in something that is ethically questionable. There will be much more on that case later as well. For now, however, you have just received a phone call because there is a real or perceived problem to be solved and you are the solver of problems, although—professional analyst sounds and looks better on your resume. As you put down the phone you realize it’s almost time to go to work. But first you want to read a little more about empires.

We know the Rise and Fall of the Roman Empire, not because we have all read it, but because we know time and time again that history repeats itself. It’s not just that way with Dynasty’s. It is the ubiquitous the cycle of life (sorry Lion King). Everything around us is either growing or dying. Look around you – are you trapped in urban sprawl as your city planners struggle to decide if they should approve yet another tract of housing or strip mall? Is your social group attempting to attract new members and/or seek donations? Is your company trying to land new contracts by pursing business with new clients? Is your boss trying to increase the size of your division? You haven’t been called necessarily to solve any of these problems; you have been called to help a very specific client with a very specific problem.

But your boss did mention that it was an important client whose business was very important to the company. Should you know anything about that client? Should you ask your boss more about the client before you respond? Perhaps you should ask your boss just how important and how long they have been a client? Perhaps you should ask if your boss knows about the problem to be solved. Chances are, you now know everything your boss knows. It is important, as you move forward, however to know from where you came and to where you are going. Why? Because the problem is about to become your problem. And if you are to attempt a solution that allows you to pursue the truth and maintain your integrity you will need to know your level of independence from the issues.

Understanding your level of independence is our second principal for pursuing the truth. And it is why you must know your place in the empire. Almost everyone is inside the empire or nested empires – it is practically impossible to not be connected in some way. Fortunately however, for now it is sufficient to simply understand you’re level of independence as you begin to assess the problem. You must reassess, however, throughout your work, this level of independence. If you don’t know where the lines are, you will no doubt become part of the problem itself and hence part of the solution. The truth will have been lost before you even leave your office.

The Analyst: Bold--Gusting to Arrogant

A colleague once told me that an analyst must be bold--gusting to arrogant. Those are some of the wisest words I’ve ever heard with regard to this profession –that of the professional analyst. Typically, or stereotypically, we are meek, akin to the computer geek or CPA. We can be seen perhaps lunching with the engineers, all of us sporting pocket protectors with outdated neckties and short sleeve shirts. A great professional analyst, on the other hand, must by definition be bold and not meek. It is their job to question everything, and then with the important knowledge gained, help those who have hired them to either solve their problems or answer their questions.

Unfortunately, to do this correctly, we must gain considerable knowledge in the pursuit of an answer. Often we become so knowledgeable that our understanding exceeds that of our employers. If we show our cards too early, we can be labeled as arrogant. By definition though, if we indeed have a solution to our employer’s problem, we are smarter, at least on the subject under consideration. Therefore we must be bold to be heard.

If, however, there is something missing, and we are not smarter than our employers and our solution is either inadequate or is rejected, we have either failed to understand the problem completely or we were hired to help with a problem that was not intended to be solved (Let’s call this the Gauntlet for you Clint Eastwood fans). If either happens we work harder, quit, or slip into analyst purgatory. Analyst Purgatory or AP is that waiting room on the way to Hell where we might get a second chance. When we find ourselves, as we inevitably do, staring at the bleak walls of AP, we still have options we can take.

We can join with the devil thereby completely sealing our fate by becoming the chambermaids of our master to assist him with his evil plans. We can become prostitutes by selling our services to the highest bidder, remaining neutral on ethics, but affirming the remainder of our lives will continue to exist in the AP of our creation. We can leave the profession completely. Or, as I would like to be the option you have chosen exhibited by your desire to participate and contribute to this blog, you can attempt to elevate our profession, with me, above the charlatans and whores that practice all around us.

We must pursue truth in everything we do. We must leave no stone unturned in the pursuit of the truth; to strive for less would be to strive for something incomplete, and most likely something to label as incompetent. For if our recommendations are followed, and they are not the truth, what is left is blind luck. We might as well have rolled the dice, kicked up our feet, took a nap, and waited for a paycheck. But to pursue the truth, to really find the answers to the questions we seek, we must be bold, very bold. To do so requires rock solid faith, abundant energy, and complete competence. When you have faith, energy, and competence you are bold – perhaps not yet gusting to arrogant, but you will gust soon, perhaps alienating those around you, perhaps winning the respect you deserve – there will be time for humility after you’ve solved world hunger.

In the mean time there is no room for the incompetent in our profession. Why would those with problems to solve choose to hire the incompetent? That would be the height of irony. It unfortunately, is not just a good question, it is a reality that happens everyday, and there are reasons it happens, some, like the sinister Gauntlet will be addressed in the days and weeks to come. Other reasons will remain a sacred mystery, but most, fortunately are simply the landscape of a profession that has forgotten how to discover the truth.

We are all ignorant when we first come upon a new problem to solve; as we start we know very little about the problem. There are other essays and professional works that can help you discover the problem and how to define things, then later how to solve things. This blog is not about the analysis process proper. This blog is about seeking the truth. So as we begin each problem anew, ignorant but hopefully with motivation, it is our next step that will dictate competence or incompetent behavior. We cannot predict which. Nonetheless, our first principle should be clearly stated here--

Discovering the truth is not easy, but if you strive for it, pursuit of the truth keeps you closer to path of competence rather than the one of incompetence. However, there are no guarantees.

So let’s begin boldly and gust arrogant as we go along.

Frameworks for Understanding

Orge Here --

Thank you Mooch for the opportunity to learn and share. Though I have long been an amateur thinker, the job title I have had for the past two years, "management analyst" asserts that I am now a professional thinker. Consequently, it is in my best professional interest to learn from my peers and avail myself to them, to the extent I am able. For now, I will not detail my background, because my current thought is that I want the ideas that I present to stand on their own. As you will see, they do require critique, maturing, and possibly expansion.

So, friends, without further introduction, here is one of my thoughts on "analysis." Analysis being defined as: seeking the truth of a particular matter with intent to understand in order to make useful decisions. To think productively, we need to understand the frameworks we use (one will not do, for reasons that appear intuitively obvious to me – let me know if you disagree; and to omit any one puts daylight between our analysis and reality/truth). Please let me know what other frameworks I have over looked. I will grant that they overlap, but I consider each a primary driver for how things work.

Frameworks:
Deterministic (because some things happen because another event caused them)
Random (because some things happen on a randomly distributed basis)
Chaotic (because some things happen on a non-random, non-periodic, unpredictable basis) Deliberate (because of free will)
Bias based (because we are human)

My goal is to increase in knowledge, understanding, and (some day) wisdom. Ogre

Lies, damn lies, and ...

“There is always a well-known solution to every human problem—neat, plausible, and wrong.” H.L. Mencken

On the subject of analytic and scientific truth…I haven’t entirely sussed out what this means to analysis, to analysts, and to me, but it leaves me questioning an awful lot of the things I’ve seen, read, and done. If nothing else, it leaves me with an even greater skepticism than I had when I woke up this morning.

I've been reading a little book titled Wrong: Why Experts Keep Failing Us--and How to Know When Not to Trust Them, by David H. Freeman. In this case, experts refers to “scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, high-powered consultants, health officials, and more”—pretty much everyone who offers advice or conclusions in other words—and the book is all about the many and varied ways they (and we) get it…well…wrong most of the time. According to Freeman, we live in a world of “punctuated wrongness,” a world where, according to one expert (the irony here is intentional on my part and acknowledged on Freeman’s), “The facts suggest that for many, if not the majority, of fields, the majority of published studies are likely to be wrong…[probably] the vast majority.” This is a pretty stunning claim. In fact, if I think about this issue as a mathematician—the area of emphasis for most of my formal training and publication—I’m simply staggered by the claim. But my field is a little special I suppose, since “truth” (within the axioms) is pretty easy to spot. We may be the only discipline wherein one can actually lay legitimate claim to prove anything since ours is probably the only completely deductive intellectual endeavor. (That still doesn't mean we have any greater access to Truth, though.) In other fields of inquiry, the fundamental process is inductive—observe, hypothesize, observe, adjust, observe, adjust, etc.—and claims to proof are problematic in the extreme—which doesn’t stop anyone and everyone from using the phrase “studies show” as if they’re quoting from the Book of Heaven. But I also have a fair bit of training in statistics—both on the theory side and in applications—and one of Freeman’s explorations of “wrongness” really hit home.

Why do we use statistical methods in our research? Basically, we want to account for the fact that the world—as we observe it—is stochastic (although whether it is fundamentally stochastic might be an interesting debate) and ensure the measurements we make and the inferences derived from those observations are not (likely to be) statistical flukes. So, when we make a claim that some observation is “statistically significant” (not to be confused with a claim that something is “true”—a mistake we see far too often, even in our professional crowd) we mean there is some known probability—the level of significance—that we'll make a (Type I) mistake in our conclusion based on observing a statistical fluke. So, for example, a level of significance of .05 indicates (kinda sorta) a 5% chance that the results observed are the result of chance—and that our inferences/conclusions/recommendations are “wrong.” 1 in 20? Not so bad. How do we make the leap from there to “the majority of published studies are…wrong?”

As an exercise for the student, suppose 20 teams of researchers are all studying some novel hypothesis/theory and that this theory is “actually” false. Well, (very roughly speaking) we can expect 19 of these teams will come up with the correct ("true negative") conclusion and the 20th will experience a “data fluke” and conclude the mistaken theory is correct (a "false positive"). With me so far? Good. The problem is that this makes for a wonderful theoretical construct and ignores the confounding effects of reality—real researchers with real staff doing real research at real universities/companies/laboratories and submitting results to real journals for actual publication. Freeman has estimates from another set of experts (again with the irony!) indicating that “positive studies” confirming a theory are (one the order of) 10 times more likely to be submitted and accepted for publication than negative studies. So, we don’t get 19 published studies claiming “NO!” and one study crying “YES!” We see 2 negative studies and 1 positive study (using “squint at the blackboard” math)...and 2 out of three ain’t bad. (Isn’t that a line from a song by Meatloaf? I think it’s right before “Paradise by the Dashboard Light” on Bat Out of Hell. Anyway…) The other 17 studies go in a drawer, go in the trash, or are simply rejected. Cool, huh? Still…we don’t have anything like a majority of published studies coming out in the category of “wrong.” In the immortal words of Ron Popeil, “Wait! There’s more!”

Statistical flukes and “publication bias” aren’t the only pernicious little worms of wrongness working their way into the heart of science. “Significance” doesn’t tell us anything about study design, measurement methods, data or meta/proxy-data used, the biases of the researchers, and a brazillion other factors that bear on the outcome of an experiment, and ALL of these affect the results of a study. Each of these are a long discussion in themselves, but it suffices to say “exerts agree” (irony alert) that these are all alive and well in most research fields. So, suppose some proportion of studies have their results “pushed” in the direction of positive results—after all, positive studies are more likely to get published and result in renewed grants and professional accolades and adoring looks from doe-eyed freshman girls (because chicks dig statistics)—and suppose that proportion is in the neighborhood of an additional 20%. Accepting all these (not entirely made up) numbers, we now have 5 false positives from the original 20 studies. If all five of the “positive” studies and the expected proportion (one tenth) of the “negative” studies get published, we expect to see 7 total studies published, of which 5 come to the wrong conclusion. 5 of 7! Holy Crappy Conclusions, Batman! (Don't go reaching for that bottle of Vioxx to treat the sudden pain in your head, now.)

Freeman, following all of this, goes on to warn we should not hold science as a method or scientists themselves in low regard because of these issues. They are, in fact, our most trustworthy experts (as opposed to diet gurus, self-help goobers, television investment wankers, and other such random wieners.) They're the very best we have. Scientists are at the top of the heap, but “that doesn’t mean we shouldn’t have a good understanding of how modest compliment it may be to say so.”
CUMBAYA! It’s no wonder we poor humans muddle through life and screw up on such a grand scale so often! I need a drink, and recent studies show that drinking one glass of red wine each day may have certain health benefits…

Did Boyd Get Entropy Wrong?

Orge here -- I suspect that Boyd, as bright as he was, and as much as I respect what he accomplished, is not so good on thermodynamics. While I don't intend to discuss entropy (the part of a thermodynamic equation that shows irrevesability and how far the process is from the ideal adiabatic process) at length, I want to ask if anyone else remembers fom their thermo days the concept of entropy and agrees that Boyd got it wrong. I figure that if this discussion can teach me in this, perhaps it will be a good forum to learn other, more important things.

Send in the Clowns

Potentially everyone is a problem solver. We all have to make decisions about things in our life be they major decisions like the purchase of a car or house, life changing decisions like whether to get married or not, change jobs, or have children, or very simple decisions like should you get out of bed today, OK sometimes that's not a simple decision. But at the end of the day, we really hope that the decisions that we make are the right ones. Unfortunately the Program Manager introduced in this blog on Sept 8th might come across as overly bad or someone with nefarious intentions. That could be true but most likely his behavior could simply be the result of bad decision-making.

It's painful to know we make bad decisions, sometimes very bad decisions, and we tell ourselves if we only knew this, or we only knew that, we would have decided differently. It hurts even more if we are exposed for our incompetence. The point is, whether we know it or not, we have been solving problems and making decisions all our lives. Marketers know that humans in general would rather move through their day making as few choices as possible – for some reason our primordial genetic make-up wishes for us to form habits. It probably has something to do with safety. If I always eat the berries from this tree I will not die, or something like that. But the fewer decisions we are forced to make the more comfortable we are and that is what sales people want us to do. If they get us to choose their product, we don't have to decide again every time. Because of the number of choices we make everyday we have learned to make decisions and evaluate choices almost unconsciously. We seemingly make decisions based on a gut feel but it more likely experience from the past that has crept into our unconsciousness compelling us in a certain direction. Or we consciously choose based on some criteria such as price, but really it's the packaging we just don't know it.

No one is immune from making bad decision, with the possible exception of Warren Buffet.. We've all made bad decisions and it doesn't make us feel very good. The worse our decisions the more we tend to feel like a clown. We don't get up in the morning wishing to make a bad decision in our personal lives and we certainly don't get up hoping to make a bad decision at the work place. We've learned to accept bad decisions in our personal lives but the ones we make in the work place are the ones that can cost us our livelihood so we tend to stay awake and consider more options.

If you happen to work in the government you could be making decisions that affect many more people and although most people have nothing but the best interest of their countrymen in mind, poor decisions are still made. If we give everyone the benefit of the doubt, that they are competent decisions makers, and most likely highly intelligent, why are we awash in seemingly incompetent decisions? This is a very good question and one this short commentary will attempt to answer in time. It's not about the best analytic approach for decision making because there are many. It's not about intelligent people working the problem because there are plenty of competent, well intentioned, and well-educated people thinking about things and placed in positions to make big decisions. It's about being committed to a search for the truth. The title of the entire blog in fact...Truth in Our Profession and how to get there.

This is not some search for higher value or the meaning of life type of truth. It's not a search for a moral correctness in our decision making either. It's about piecing together a puzzle the picture of which we cannot yet see. But wondering enough and being committed enough – for whatever reason – to reveal what is truly in the picture. Commitment to the truth will unlock doors of curiosity and it is through these doors that discovery will occur. Sometimes it will not and you may have to work harder, or you will run out of time and have to do the best you can with what you've got, but you will have done your best and perhaps caught a glimpse of the truth. So then his short paper is about the journey to discovering truth. Striving for and equipped with only a few extra rules in order to make a competent try, elevating the profession for all of us. You may not make it, at the end you may still feel like clown, but it is worth the try if for no other reason then you don't have to sit next to the PM with the big floppy shoes on at the retirement seminar. That clown is already there...let's hope he is not waiting for you.

Based on Actual Events

The account described in our 8 Sept Blog was based on actual events. The analyst is real. He is an intelligent and extremely loyal member of a corporate team. He is an operations expert who had spent significant time in the field, as well as in school to become an analyst. A practical if not recognized expert on both the subject to which he spoke and analysis to which he conducted. Before the study even began the pudgy finger poking PM was asked the simple question...If the results of the analysis are negative towards your program what will you do? The PM was confident and optimistic in his response that he was a company man and a team player. He wanted only the best for the organization and he would support the study to the fullest. It was only a matter of days before the PM started stone walling the analysis team. Despite the barriers placed in their way, the analysis team moved forward, uncovering and discovering many interesting and contrary facts along the way. When the study was complete the team dutifully briefed the PM. The team was immediately descended upon by legions of critics attempting to pull apart the analysis. Briefing after briefing was conducted and the analysis and analyst stood firm – yet no one would tell the CEO that a mistake had been made. Lips were sealed until it was apparent that another company was becoming a potential competitor. When the analysis team could no longer sit on the truth it was taken to the CEO. The results are now history. The analyst was promoted early and will return to operations. The PM was last seen attending a retirement seminar. And the Truth shall set you free…

The Program Manager and the Analyst

The division analyst finished his presentation and returned to his seat. The Chief Executive looked around the room and said, “Gentleman, where was this dissention last year when we decided to fund this program?” He looked at the presenter, who was keeping his eyes glued straight ahead, trying not to lock eyes with the others seated a the table, the ones in the room to whom the executive was addressing. “I say again, where was this dissention when we invested a large percentage of our annual revenue in this program?” The table was dead quiet except for the imperceptible sound of building blood pressure in the Program Manager’s head. “Gentleman”, the executive began again, “It’s very rarely that we see the truth in this company. Sometimes we glimpse the truth, but most of the time we are chasing a target that will not sit still. Today that has all changed with the presentation of this analysis. For your sake, for the sake of this company and our stockholders, I hope it is not to late to turn this ship around.” He thanked the division analyst for his candor and swiftly left through the door in the back of the room. No sooner had the door swung shut than the PM stood and raced around the table to confront the analyst. The analyst only had time to spin his chair toward the approaching manager. Producing a pudgy index finger the now fuming man poked the analyst in the chest and spoke rigidly, “What gives you the right to brief the CEO without my permission? I did not give you my permission.” The analyst looked straight into the PM’s eyes and said, “Sir, I work for this company and the stockholders of this company. Whom do you work for?”