The Primitive Pete

I don’t have the greatest recollection of my days in middle school but there is one thing that stands out in my mind, Industrial Art’s class. I don’t remember my instructors but I remember it was my first experience bending metal. We fabricated ash trays which we proudly brought home and presented to our parents. Aside from the interesting social commentary it was the first time I had actually produced a useful product with my hands. My dad, however was not too interested in my profession in the ash tray fabrication business. I remember him telling me on numerous occasions to get into a business where I could use my head... where I could think and get paid at the same time.

There is one more thing I remember.  I was introduced to a character who has lived with me ever since. His name was Primitive Pete. Primitive Pete had a staring role in an educational film produced in collaboration between General Motors and Walt Disney Productions known as the ABC’s of Hand Tools. A quick search on the Internet and you can view some of what remains of the Primitive Pete material. Pete was an interesting character; he of course was a caveman intent on using all the available hand tools he could find, in the wrong way. Using a screwdriver as a crowbar for instance and always, always using a hammer for just about every application. Now my memory of Pete is most likely flawed and having found the material on the Internet I intentionally did not go back and change my fond perception of him after 30 years – so please forgive me if there are some inaccuracies as I report him to you.

The important part of this topic is that there is a right way and a wrong way to use a tool. And there are certainly many tools available to the professional analyst. Tools, models, spreadsheets, simulations, programs, algorithms, heuristics, hierarchies, etc. all must be used correctly in order to have a remote chance of doing the job correctly. Unlike Primitive Pete however, if you use the wrong hand tool you will do a lousy job and even worse someone could loose an eye. For us, the use of a wrong tool will probably go unnoticed by most, and in the main no one will get hurt or killed in the process of doing the analysis. The end result will be the same, however, a bad job.

The bad job will manifest itself at some later time following the equally bad decision that could be made as a consequence of the influence. It’s certainly not inconceivable that someone could get hurt or die as a result of a bad decision – although this essay should not be so bleak as too suggest that product safety or sound engineering principles in the fabrication of a product wouldn’t be assumed – the other guys always to their job correctly. So we are professionals in are trade at the office, not shade tree mechanics or Norm Abram’s carpenter want-a-bees as we might be at home. Professional means we know what we are doing and we will do the job correctly with the right tools at our disposal.

So why do we fail? Typically it is because we use the tools that are in our toolbox just like Primitive Pete. If the tool is in there we will probably find a use for it. It saves us a trip to the hardware store. Also, our tools tend to be very expensive software applications – and it is difficult to retool the factory without a major investment. Another reason is training – we know how to use the tool that’s in the box – if there is another one, we may not know about it or may not know how to use it. In addition, even if the tool in the box is the correct tool for the application, very frequently we don’t use it correctly or we don’t bother to sharpen it, to draw yet another analogy with hand tools.

I’m fairly certain that an unsharpened tool is about the most dangerous one of them all. That’s the wood chisel that when sharpened will carve wood like soft clay but when dull you will invariably drive deep into the palm of your hand. So here are a few examples from our profession. Using a linear program to optimize a non-linear behavior. Using old data or data that has been prepared by another group that is not fully reviewed and understood in the context of your analysis. Adding too much complexity to a spreadsheet thereby loosing either the cause/effect traceability or fixating on a portion of the problem that is not the main concern. Or conversely, over simplifying a problem that requires more detail.

The list goes on and some of the right ways to approach problems will be addressed in further installments of this blog...please stay tuned.

Lying with Statistics...and PowerPoint

It is well known and obvious to most of us in our profession that the phrase "lying with statistics" does not mean to be intentionally misleading. In fact, knowledge of the ease at which someone can mislead with the manipulation of numbers is common enough to prompt at least a cursory review of most analysis products to head off the inadvertent case of manipulation that could be construed as intentional deception. That is of course a cover your butt response to head off any low hanging criticism. But it is an important one. It is not the altruistic goal we seek of conveying only the truth. Nevertheless, the early discovery of that which may be misleading does have a certain first order effectiveness. It helps by eliminating early distractions as we should always strive for our products to be clear and as absent of controversy as possible.

This doesn’t correct the problem, however, if the result being reported is, itself, incorrect because the analysis behind it was faulty. Still yet we must be reminded that it is extremely important in our search for the truth that we understand how it is very easy to lie with statistics and that we must always be on guard to keep deceptive or misleading results completely out of our activities. Further, we must avoid the unintentional pitfalls of the of graphical products such as PowerPoint that obscure our message, change our message, or become the message...because they will.

For today this essay is not about the mechanics of lying with statistics, the well known book, “How to Lie with Statistics” by Darell Huff covers this topic quite well. On the issue of Power Point, the books and publications about graphical presentations by Edward Tufte, such as “The Graphical Display of Quantitative Information”, pretty much cover the water front on that topic as well. Since in both cases, graphical presentation of statistics and PowerPoint are primary tools in our tool kit and will remain at our immediate disposal for sometime to come, what more can be done to extract truth?

The best way to avoid the pitfalls is to keep your perspective that you are the analyst. Tools and presentation products do not have the brains – you do. Long before you commit a number to a spreadsheet you should know exactly what trend you are trying to show or display. If you find yourself manipulating the data in the spreadsheet because you are looking for meaning in the data, you probably are beginning to walk on thin ice and should be wary. If the trend doesn’t just jump right off the page it’s possible that it’s not really there. This should be your first signal to start questioning what came before the spreadsheet, not what’s in the spreadsheet.

With regard to briefing slides, here again, you should have a pretty good picture in your mind with regard to the story you want to tell. If you let the story emerge while constrained by the software application, you are probably letting something creep into your message that you didn’t intend. When Marshall McLuhan said, "The Medium is the Message", he was serious about the underlying facts and content not being the message. The message is contained in the medium. Over and over again we find this to be the case. A well presented diagram of the facts trumps the facts in most cases. What this means if if you present a beautiful depiction of the relationship between two sets of data, it is the picture of this relationship that will be remembered in the mind of the decision maker. You better hope it was a correct relationship and not a misleading relationship. Time and time again we see presentations that show a wide gap between two alternatives. If the scale was changed, and the observer is not aware of the change in scale, they come away with a vision of separation that may or may not actually exist...the medium became the message. And the message was not necessarily true.

The best way to combat this is to decide on the truthfulness of your message first. This means do the thinking first and then use the tools at your disposal, not visa versa. In this way, the truth, as you will learn to discover, will not be obscured, influenced, or manipulated –either intentionally or unintentionally, by the external limitations some of which you were not even aware. Then your final product will be clear and logical and stand-up to the first wave of PowerPoint critics who enter the room...and those who leave the room will not be deceived.

Essence of "Decision Points"

Since I read the book I thought it was appropriate to post my Amazon Review here. A book called "Decision Points" has got to have a connection to truth in analysis. So here you go.

"Decision Points", by George W. Bush, 43rd President of the United States, presents history with somewhat of a conundrum. On the one hand it is written by the 43rd President of the United States and should be required reading by all. At least anyone alive during the first decade of 21st century who has ever wondered about the actions the United States took in the wake of the terror attacks on 9/11 and how the most important leader in the free world made the necessary decisions to keep us safe. On the other hand, the book is not a biography or a complete memoir in the normal sense. President Bush chooses instead to provide a personal narrative behind several major decisions he made during his two terms in office. He does this by highlighting the top level issues he considered. His approach works because his writing style is easy to understand. His approach does not work because his account lacks sufficient detail to study his actions in depth. It is impossible to independently determine whether his decisions were correctly formed based on the information he had available. His narrative therefore becomes a simple apologetic for his decisions. Along the way he takes full credit, perhaps too much credit, for the good things that happened and responsibility, perhaps not enough responsibility, for the things that didn't go as well.

Whole books have and will be written to address various aspects of the past ten years and the decisions that were made. Topics such as the intelligence surrounding the presence of WMD in Iraq, taking down the Taliban in Afghanistan, democracy building, stem cell research, efforts to eradicate Aids and Malaria in Africa, the aftermath of Hurricane Katrina, and the financial crisis of our country. Thirty pages from the President on each of these topics provide a necessary view from the top but in no way provides the complete picture. Some context is formed but then it's on to the next topic. This is not a study on the complexity of Presidential decision making ala "Essence of Decision" on the Cuban Missile Crisis by Allison and Zelikow although given the gravity of many of this decades decisions, along with the accompanying loss of life and limb, perhaps it should be. Nevertheless it is a useful book.

Regardless of your opinion of George W. Bush a portrait of how our former President's mind works is clearly painted. Supporters of his policies will like his reductionist approach to the complexities of his office. Critics will yell out loud with almost every turn of the page. Since political awareness peeked in this country during his Presidency all will benefit from understanding what the President was thinking at the same time when we were all scared and confused and wondering what was going on in the mind of the man at the top. Now we know. Good or bad, right or wrong, generations will now have access to these thoughts. I'm glad I read this book.

Also, make sure you visit my actual review here and give it a "Helpful" vote.

Jim's Review of "Decision Points" on Amazon

What We are Taught

We are all problem solvers. Some of have been to school, and some of us have learned by being on the job for many years. All of us have learned something about the right way to make decisions. Unfortunately, like leadership or management techniques, we have also learned a lot about the wrong way to make decisions. It is the factors contributing to the wrong way to make decisions that we must always remember if we are to change our bad habits.

We have learned, for instance, that we must be team players or we must protect and defend our territory. These can be instinctual but they should not be automatic if we are attempting to ascertain the truth. Defending our program is natural and many times the right thing to do, however, if our program is detracting from the greater good of the larger enterprise the truth must be surfaced. Many times it can be heard in the back rooms of programs that have started to run thin, “If we only had the analysis to show our worth”, and then they go looking for something to save them.

Perhaps they will find what they are looking for, don’t let it be you. Those of us who have been professionally trained are not immune to the pleas of a program in need of assistance. If we aid a dying program in need, we fall into a category that will be explored in greater depth later. The contrary side of looking for analysis to defend a dying program is have analysis to support the elimination or divestiture of a dying program. Now, this might very well likely be the truth, but what we have learned is the folks that spread bad news like this are the ones who are not team players and when bad news is known, it tends to be swept under the table, and woe be to the individual who blows the whistle.

Still there is more to be learned in school. In a professional analysis educational program such as Operations Research many formal disciplines have been developed. We have been taught to apply the techniques to problems and we have been trained to look for problems that fit the mold of a given technique. We call this formulaic analysis. A good analyst will know exactly what problems can be solved with their particular application and what application cannot. And then they can apply their formula. Sometimes a particular application will be pushed on a problem for reasons that can only be described as self interest – these reasons and the persons behind them will be treated in later.

Hopefully the self respecting and quality analyst will not attempt to use an application where it is not intended, yet still mistakes are made and the wrong tool for the job is used. In most cases this can be attributed to a misunderstanding of a much greater problem. And this gets back to problem definition, the hardest part of analysis. There is only one solution to this dilemma and unfortunately there are not many courses being taught on the lost discipline of thinking and in particular, critical thinking. Yes we are talking about over used buzzwords such as thinking “out of the box”. That is about thinking and being creative in general. Critical thinking, on the other hand is about using discipline and things such as the Scientific Method to fully explore a certain domain. Then tightly constructed logical arguments can be made to make inferences, create testable hypothesis, and generally wander around in side of decision space. All of these things can and should be taught in a professional analysis curriculum. Some do and some don’t.

When interviewing analysts its important to understand if they were either formally instructed in critical thinking or they came to it via experience. Either way, those who have learned or established critical thinking behavior can be invaluable to an organization and should be identified. Others, who have mastered formulaic analysis do have their role, but one cannot assume that just based on the title of analyst alone, one carries the moniker of someone who can think critically as well. Yet, perhaps the worst thing we have learned is that because analysis was done, or because this decision was based on analysis, that we have discovered the truth.

Too many decisions have been based on bad analysis in the past, that perhaps too much trust has been lost. Unfortunately, bad analysis is understood all to well and leads directly to an assumed fix to the problem. If we do not trust a small team of analyst to do a good job, a large team is probably the answer. It is true that by getting a larger set of eyes on a problem will typically lead to the discovery of errors in a complex calculation. In no way do we suggest that more people can’t be brought in to review or look at a problem. What we have learned, however, is that collaboration must naturally lead to better answers. This, however, is practically never the case. Collaboration, in most cases, leads to extending the length of time it takes to get an answer and, unfortunately it leads to compromise. Too many cooks in the kitchen spoil the broth. If some people like salt and other people like hot sauce, you cannot compromise and put both into the soup and expect anyone to like it. Unfortunately, because collaboration has occurred, we have been taught that it must have lead to a better solution.

Less people may get in trouble if it was a group decision to do something stupid, but nothing has been gained. Allowing the analysis to occur in small teams should be the first consideration. The analysis can be brought forward for the review of other people, but never to influence or participate, only to review for errors, but we are now ahead in our discussion. Here we are merely reporting what we have learned to do. Soon we must break these habits. But first there are still a few more bad habits we have learned that should be fully understood before we suggest methods to help.

The Ethics of Analysis

The profession of professional analysis does not, in and of itself, have a written code of ethics. It is not taught to us directly in graduate school as one in an MBA program might take the requisite class in business ethics. Additionally, our field is so wide open and can be applied to every aspect of human endeavor that most would feel the addition of ethics to our curriculum is unnecessary. Yet who are we to judge what is at it’s heart the simple application of a mathematical technique to a solve problem. It is not the applied math that is evil but rather it’s application with disregard for the dangers or with nefarious intent that creates the problem. Exactly. It is not the handgun that is evil; it is the human behind the handgun. The untrained hand that causes an accidental death, or perhaps lacking in the moral understanding of the sanctity of human life. The result is the same. A handgun is an extreme analogy.

A car would be a better analogy. Clearly a car cannot be evil in and of itself (discounting the massive negative impact it’s life-cycle has on the environment). A criminal, as a new example, needs a getaway car, the driver of which may or may not be a witting participant. The car itself is but a tool to make a hasty escape. It is an inanimate and completely neutral object. The driver however, can choose to drive, or can question the passengers who hastily jumped on board with bulging satchels and what appears to be members of law enforcement in hot pursuit. If we as analysts did our job correctly, we would have obtained a fast car, we would have checked the traffic reports, we would have a good map with a planned route. We also would have a GPS for backup, a spare tire, and jack onboard. Wouldn’t want to be the get away driver feeling foolish asking for help on the freeway if a tire goes flat. In fact, we could do the very best planning and have taken great pride in our efforts only to find that we are simply in the midst of a serious crime.

How do we avoid this? Most bank robbers (sorry for the sterotype) want employees who don’t ask questions – coincidentaly most organizations want employees who are loyal and salute smartly – we call them team players. But here we find ourselves in one of the few professions where, in order for us to do our job to the best of our ability, we must ask many probing questions – questions that the boss might rather not answer. If you want the fastest travel time between the building on 17th street and the expressway, I recommend we drive the route before or after rush hour. But the bank is not open during rush hour – what a stupid question. Oh, I didn’t know you were going to be at the bank, I was just working on the fastest travel time. How long will you be in the bank, is there parking on the street just out front? During rush hour, a lot of people take the bus, it’s really hell being down town during those hours, can you take the bus? No. How about the subway? No. Why not? Because we will be in a Hurry? Why such a hurry, do you have to catch a plane? I wasn’t planning to drive you to the airport – but if that’s the case, perhaps a helicopter would be a better vehicle? I could pick you up in the park adjacent to the bank and pop you straight over to the airport.

Well that option is too expensive. I can’t afford a helicopter. OK, how much can you afford? Taxi drivers know the best routes and are the best drivers in the city at that time? Well I hired you to drive so I don’t want to hire a taxi cab? But clearly a taxi would be your best choice and since I want only the best for you I would advise you to fire me and plan on grabbing a cab when you come out. Well I need a driver who will take me any where I want to go? Yes, most cab drivers will. But I need the driver to be discrete? Ok, that is a new requirement; I was unaware of the discrete nature of your transportation needs. I can be very discrete – but I suspect so can a few cab drivers. But since you will not have control of exactly which cabbie you might hire, I think we can agree that I should drive. Yes I have already decided that you should drive – and I have already picked out a car and picked out the route for you to travel.

Oh, you just want someone to drive the car, not to help solve the problem, I’m sorry you just want a mindless, faceless, driver who doesn’t think. You’ve offered me a lot of money to drive this car for you, I think you could save a lot of money and hire a dumb driver for a lot less. Well I don’t want some incompetent ape driving the car – that would be worse than hiring a cabbie. Well that’s why I was trying to help, because I do think about the problems and typically can make the journey more efficient. What is it that you really are trying to do? We, I am robbing a bank and I need a get-away car.

Oh, why are you robbing a bank? If you need money and want to obtain it illegally, why not rob a convenience store, the get away is much cleaner. Not enough money in it for the risk. Ok, how about a grocery store? They have a lot of cash on hand. Well, I’m a bank robber, I’ve never robbed a grocery store. I wouldn’t know how to do it. Oh, well I’ve never robbed one either but I can definitely drive the getaway car. It would be much easier for me if the getaway is really independent of your action. But how about Internet fraud. If you really want money that is supposed to be a good way to get some – and it doesn’t require a get away car. I told you I am a bank robber – I would have to go back to school to learn how to use a computer – and you can’t teach an old dog new tricks.

Ok, looks like the bank is your only option. Have you considered robbing the bank at night when the streets are clear. Yes, but that requires that I hire a safe cracker – and have you ever had to work with a safe cracker. They are some of the hardest people to work with. They think the whole job is about them. Without them there is no bank robbery. I vowed to stop working with them a long time ago. What about tunneling into the bank and blowing up the safe. You don’t need a safe cracker for that. No you are right. Tried that before too. I need a bigger team – and the payroll on this job is tight. Can’t get the big boss to give me a bigger budget. Have you demonstrated that the rate of return on a safe job will be much higher than that of the daylight teller pull? Yes, he knows that, but he is expecting a few lower paying less risky jobs than a high risk, capital-intensive job. We can use you and your car on a couple of jobs the same day. Oh, multiple get-aways on the same day, this is a traveling salesman problem. I studied this one in school. Now I think I understand the problem completely and will optimize our route through town to hit all of the banks and then escape through the tunnel. Let me get started.

OK, hopefully the point is made. Analysts must ask questions, they must understand the total problem, and they actually have to remove themselves from the problem in order to get to the truth, if there is something keeping them from that truth – like an illegal activity. Getting to the truth is like pulling teeth, particularly if the truth does not want to be revealed. Now, I have mentioned that this essay is not about the underlying ethics of the problem – it is about getting to the truth. To get to the truth, and our forth principal, the analyst must understand the entire problem, not just what was fed to them by someone who thought they understood the problem. Understanding the problem is the hardest part of the job, but the most important part. Truth can be discovered if the problem is understood. If the problem is never understood, how can there be a right answer? This does not, however, free them from the underlying ethics of the problem. If they are lost in the problem perhaps they can forget about the ethics until the problem is solved, but they cannot absolve themselves from the greater truth, if in the end, the solution is still illegal. Later I’ll devote an essay to helping you with principle number five. You cannot divorce the truth from it’s ethical meaning, no matter how good a solution you found.

For now, however, I would now like to concentrate on the profession of analysis itself – in pursuit of the truth, because although you might believe you are on path of the truth because you sought the right problem and asked the right questions, if you blew the analysis, the truth you hold in your hand might be a nicely written pack of lies.

The Gauntlet

There is a great scene from the Clint Eastwood movie, “The Gauntlet”. The Clint Eastwood character is asked, “Why do you think they gave you this job?” and Eastwood responds, “Because I get the job done”. His boss retorts, “No, it’s because you don’t get the job done, you were never supposed to get this job done”. And that, in a nutshell is what we call the Gauntlet syndrome.

When an analyst is brought in on a hopeless task for the sake of either showing progress, or for documenting that everything conceivable has been done and we are still failing. This is the nefarious twist. The same result occurs when the analyst is brought in on a project with a known outcome. You are there to prove that the right solution has already been discovered. Woe be to the analyst that thinks they can make a difference or perhaps worse yet, finds an actual solution to the problem. What’s left is nothing short of having to run a bureaucratic gauntlet that will leave you bloody and near dying if you have the guts to run through.

Most don’t have the guts to run through it, or having started will back down when their health begins to fail. Let’s say on some odd occasion the analyst makes it through. Perhaps, they might exclaim, “I stuck to it and I won, and the system works”. This is the height of nonsense. The system did not work if it takes months or years for the truth the emerge and for people to put everything on the line in it’s pursuit. Also, should you win, the ramifications could be the stigma of someone who was noble in their efforts but could be labeled as not a team player or someone who was a zealot in their own right.

We have seen this many times before. The best analyst is the one who can maintain a bit of independence from a problem. The best advocate is someone with a true interest in correcting the problem. If you are close to the issue you might be willing to run the gauntlet. If you are removed you will probably be unwilling to get bloody over the issue even if you know you have the truth in the palm of your hand.

So by definition the system does not work. And we, all of us, are perhaps no greater than the Eastwood character. We were brought in believing we can get the job done. But no one actually believed that we could, or we were really just supposed to go through the actions to support an a priori answer, or worse a decision that had already been made. This just makes us process bureaucrats and you don’t need a whole lot of education in our disciplines to be one of those.

So you have to decide each time – as you peer down through the gauntlet. Do you have enough protection to make it through? Is this one worth getting bloody over? This will probably be the most frustrating dilemma you will have to face. We can only offer that at times like these, if you haven’t done your homework, if you haven’t done the analysis to the best of your ability, if you haven’t followed the prescription for the truth that we hope we have captured as you move forward through these pages, this might not be the time to run down the rabbit hole. But if you have, if you are in possession of the facts, and have the answers to the majority of the questions that will come from both sides of the gauntlet, can there be anything short of life and death combat that could bring heighten your sense of awareness and make you feel alive. Don’t let it consume you; don’t let the sleepless nights ruin your health. But do occasionally run the gauntlet. That is what you are getting paid to do.

Rest in Peace Benoît Mandelbrot

Benoît Mandelbrot has passed (20 November 1924 – 14 October 2010). Let us pause to honor him briefly on our blog and reflect upon his work. 

Click here to link to his Wikipedia Bio

Click here for his obit in the NY Times

The video and song below contains strong language so it's not for the kids...but it honors him in certain ways.   It's more fun than anything else.

The Charlatan and Other Weasels

The professional analyst better known as the cost analyst, the systems analyst, the operations research analyst, grew it’s own profession during World War II and directly thereafter. Some would credit the work done to locate German U-Boats as the first attributable activities of our formal profession. Whereas there is plenty of documentation to suggest the mathematical formulations that were developed during this timeframe constitute what we now refer to as operations research, analysts have been around much longer than 1940.

Frederick Taylor published his work; "The Principles of Scientific" Management in 1911 giving him an auspicious start in the field but one, which is typically credited with the Modern MBA, vise operations research. There is no doubt that his time and motion studies constitute, for the analyst, a very well documented way to collect the first data.

But way before Taylor arrived on the seen anyone in business strived for efficiency although they might not have been certain how to achieve it. They learned by making trades and by making mistakes. Almost 100 years previous, however, Eli Whitney labored to produce machinery and interchangeability to improve efficiency in his factories. Whitney, however, has both supporters and critics. The individual typically credited with the invention of interchangeable parts is Samuel Colt who took before Congress 15 guns. After disassembling them, he mixed up the parts and reassembled them. The approving Congress immediately placed orders for 10,000 of the weapon never realizing that the demonstration was a shame and Colt had marked all the parts so he could reassemble them correctly.

Regardless of what took place on that day, Colt, enlisted the help of Eli Whitney and together they eventually got it right and the first assembly line took root – this well before Henry Ford apparently invented one. This paints a darker side of analysis, a side that allows the initiated to obscure the truth with promises of greater efficiency and cheaper products. Perhaps they will, perhaps they will not, but many well-documented cases of fraud occurred throughout the 1800s culminating in Congress passing a False Claims act in 1863.

The Charlatan is the guy who convinces his boss that by adding sawdust to the gunpowder they were selling the Government that the company could save money. Perhaps it wasn’t the analyst who told his boss to do this, but if the analyst was telling his boss not to do this he never blew the whistle. This creates a problem for our profession, and therefore becomes the heart and soul of our ethical dilemma. This is a typical problem, which is called the whistle blower scenario. Blowing the whistle is a no win scenario and cheating inside the no-win scenario makes matters worst. Even with protections in place blowing the whistle must be considered carefully with both facts and figures.

As with profit, the primary motivator of malfeasance in the commercial sector, crooks may actually desire the "wrong" answer. Let's hope you are not the one with the time managment study showing how to manipulate the machine to make an extra buck. Let's also hope you an not employed by a crook.

The Knee Jerk Reaction of an Anti-Fluids Bigot

The following entry is in response to a debate I have been having in a closed blogosphere at work. It is based on a comment I posted to this article.

http://www.popsci.com/cars/article/2010-09/giving-traffic-lights-mind-their-own-can-reduce-congestion-study-says

I will not replicate the blog comments here…you can pick up the gist of the commentary from my narrative below.

First just to be clear I entered into this debate with Bryan on my own free will although he started the argument by baiting me with his tactics. I tend to be an arrogant troll myself on occasion so I am not at all intimidated by his discussion tactics. The evidence of his particular bullying is his use of the term “knee-jerk reaction” which he directed at me for my very quick, and somewhat negative, response to the article posted above by Sphinx. In that comment I stated that modeling the traffic in a city is not like a fluid so they are doing it wrong (Bryan is clearly in “ Stage 3” as given by King and Logan in the book “Tribal Leadership”) It’s not clear that he was rising in defense of Sphinx and that somehow by posting a negative comment to her blog I was attacking her personally (which is certainly not the case) or he is defending the integrity of the authors of the study upon which the article was based (which was most definitely the case) and in both cases, his and mine, we are not in full possession of the facts within the actual study (the actual study can be found here [1]). Nevertheless the debate ensued and I can either stand down or continue to fire back. Obviously I have chosen to fire back.

As an additional note, I also made in my original comment the statement, “If they were saying they model the traffic as a fluid as an analogy” …for laymen to get the picture…that’s Ok. But if they are using fluid dynamics in their actual calculations, as I stated above, they are using a flawed technique.

I followed up my original comment with slightly more detail on my meaning which included the necessity of scaling various traffic modeling problems from intersections with single car/driver interactions through free flowing highways. I said that in the second case fluid dynamics might find some application but they have no place in the former. Although to be even clearer, classic fluid dynamics may seem like they apply in the second case, and are in fact the source of many studies over the past 50 years, in the main they are no longer used by the industry. Occasionally you will see a fluid dynamics PhD student write a paper on some new application of fluid dynamics to traffic flow. However the problem as defined in the original statement, the requirement is to take a systems wide view of a network of urban intersection in order to synchronize the signal lights…but synchronization is the system response…the idea is for each light to work autonomously while considering the bigger picture. This will not work if modeling traffic as a fluid (the algorithm to drive the timing of the lights needs traffic input and output, it will work as a surrogate for traffic but my argument is that it is not a good surrogate and most likely means the algorithm to drive the timing of the lights is also suspect) That of course is my opinion but if you check the core disciplines aligned under fluid dynamics you will not find traffic…because why? It is not a fluid.

Traffic flow on a road is nothing like fluid flow, either compressible or non-compressible fluid flow in a pipe. As you approach free flowing traffic on a highway with smooth curving bends and flow that is uninterrupted it is easy to believe that fluid equations might apply and simplify down to a basic case. This has been the silly mistake of many individuals who have tried. Yes with a lot of trouble you can show flow with the equations, but then you have modeled a system that is not grounded in reality and most certainly does not represent real traffic. One might read [2] to see how all the math might predict the motion of cars modeled as a compressible fluid in a closed circuit for instance, as a soliton (wave) …but it’s not reality and it’s far too complex to be practical.

Starting from first principles all of fluid dynamics arises from the Ideal Gas Law which states:

pV = nRT

Pressure x Volume = Molecular Mass x Gas Constant x Temperature

Pressure doesn’t apply to traffic. Volume doesn’t apply to traffic. Molecular Mass doesn’t apply. The Gas Constant doesn’t apply. Temperature doesn’t apply.

This is where Bryan tries to convert cars to molecules, traffic speed to temperature, and traffic density to pressure x volume. Go for it. Obviously I can’t deny that folks have done just that…just as folks have tried to drive a nail with a set of pliers as well. I contend there are better and much more accurate methods.

But that’s just the start with fluids…How about viscosity? How about friction? How about the length of the pipe and the inlet/outlet pressures? The list goes on. You can find all the equations for pipe flow “Q” here in this reference below [3].  But here is a look at it.

Now if you throw out everything but traffic density and speed you are reduced to this equation:

Q = D x V

And who would argue that traffic volume/flow in not a function of the number of cars on the highway and their speed…which is traffic but it doesn’t behave like a fluid in any way other than it moves from point A to point B, sometimes. Sometimes it turns, sometimes it crashes. But one “fluid like” equation, is far too simplistic to model or predict the behavior of cars on a road, highway, or at an intersection. A great deal of work has gone into describing traffic mathematically in its many forms. All of that work is better than fluids but none of it better than discrete event simulation which is the standard for the industry and what has been used in thousands of transportation and traffic studies worldwide. A survey of the tools available is given here [4].

But it doesn’t end there. As we use discrete event simulation tools to model the interactions of the individual cars in a network in order to effectively model what is happening more and more autonomy has to be given to the individual elements. Not only do the cars have to behave independently, speed, direction, etc., they have to think for themselves interact with what happening around them. They have to receive feedback as well (try that with a fluid). A better way to do it is by treating the traffic network as a complex adaptive system with the use of agent based modeling to simulate the traffic in the network. Here is a traffic application and references [5]. That is the full sum of what I was trying to say with my comment.

Before I started my simulation company in 1994, I first described agent based modeling in discrete event simulation for traffic flow research I was conducting for Dr. E.B White at George Mason University during my masters program. I have an obvious bias against fluids which I studied in several courses during my undergraduate work in Mechanical Engineering at Texas A&M University. My traffic research was entitled the Strategic Transportation Operational Planning Simulator (STOPS) [6]. Alas the computing power for complex adaptive simulations and agent based modeling was not sufficient in 1992. It is now. But now the field is abounding with these techniques. Couple these techniques with the power of cloud computing and the power of netted information coming from an urban traffic grid and the solution to decentralization of traffic control systems is upon us. Again, all I was saying in my original comment was that fluids are bad and complex adaptive systems are good.

As I have now read the original study it is obvious that the cars are indeed treated as discrete events and were modeled using a simulation called PTV Vissim [7]. My original thought that the use of the word “Fluid” was simply an analogy is true. However it would be interesting to treat this system with a complex adaptive simulation as I have suggested. If my reaction was “knee-jerk” I apologize to those I offended. I am an “Anti-Fluid Bigot” when it comes to traffic modeling.

For those of you who do not like to start with first principles and do the thinking for yourselves, a criticism of using fluids to model traffic, that is not my own, they can be found under Daganzo’s Requiem [8]. Coincidently I observed in several studies that Helbing [9], a coauthor of the subject study, has been a vocal critic of Daganzo’s criticism of fluid dynamics for traffic modeling. This is truly odd since Bryan and I have inadvertently entered into a microcosm of this same debate. Since Helbing choose a direction other than fluid modeling in his most recent work can we assume he is now on-board? Since the word “Fluid” does not appear a single time in the recent report, I have to wonder why it was used at all in the subject article.  To venture a guess would be to suggest the use of the word fluid is an echo from a once lively debate between Daganzo and Helbing.

In the end I must now thank Bryan for forcing me to expand upon my comment so everyone has a better understanding upon which I based my “knee-jerk” reaction.

References:

[1] Lammer & Helbing, Self-Stabilizing Decentralized Signal Control of Realistic Saturated Network Traffic, 2010

[2] Saavedra & Velasco, Solitons in a macroscopic traffic model, 2010

[3] Schroder, A Tutorial of Pipe Flow Equations, 2001

[4] Jones, et al. Traffic Simulation Software Comparison Study

[5] Khalesian & Delavar, A Multi-agent Based Traffic Network Micro-simulation using Spatio-temporal GIS.

[6] Muccio, Strategic Transportation Operational Planning Simulator, 1992

[7] http://www.ptvag.com/software/transportation-planning-traffic-engineering/software-system-solutions/vissim/technical-details/

[8] Daganzo, Requiem for second-order fluid approximations of traffic flow, 1995

[9] Helbing & Johansson, On the controversy around Daganzo’s Requiem for and Aw-Rascle’s resurrection of second-order traffic flow models, 2008

The Truth About Profit

As you drive over to the client site your mind begins to consider Empires again.  My company is an empire, or they are striving to be.  I have always considered my boss to be a little Napoleon, but he has always been straight with me.  He has never asked me to do anything, “unethical” but yet I know he has been consumed lately with the organization chart and our spot in the bigger “business unit”.  Whatever that means.  His annual bonus is tied directly to his position in the “business unit” and how we finish the year in revenue compared to the rest of the business units.  Fortunately for me I am on a fixed salary.  I can be free to say the things that need to be said.  If I can’t help this client, there will be another one, and although we didn’t land this contract, I’ll still get paid.  My boss will just have to wait until next year to take his wife to the Bahamas. 

At this point I should clarify that in any empire, ah hum, organization there are two classes of clients.  Those who wish to profit whom I will refer to as the commercial sector client or CSC, and those who do not wish to profit or that unfortunate class of entities that I will refer to as the not-for-profit client, the NFP for short.  The NFP’s can include non-profit organizations but I am primarily referring to the ubiquitous and all powerful government entities that dot our landscape and blot out the sun, particularly if you live in your nations capital. 

If you happen to be a CSC the truth is as follows – there are many of you competing for a share of the marketplace.  In any particular market sector, unless you are a monopoly (but keep that dirty little secret to yourself), there are many of you who can fulfill the same need in your sector.  What distinguishes you?  Is it fresher ingredients?  Is it an altruistic need to donate 5 cents of the purchase price of your product to a favorite charity?  Is it better marketing and packaging?  Or is it, hopefully, an actual product that is superior perhaps in both quality and service?  These would be rare, but if they could be found wouldn’t they dominate the marketplace?  Yes, in fact they would, or should, if it wasn’t for all the other charlatans running around claiming the same superior quality and service.  How can they do that?  Clearly someone has to be better.  Maybe we should hire a professional analyst to make that determination – or maybe we should just leave well enough alone.  To the CSC the truth might seem like the right and reasonable thing to discover – but it could also be very dangerous.  So for the analyst, employment to seek the truth with the CSC will have to be reserved for perhaps improving the efficiency of internal methods and procedures not so much in marketing a product – that’s where the real money is – but unfortunately, that money goes to the analysts that can help them sell a product.  And those analysts work on Madison Avenue.  We refer to them as marketing and advertising firms or MAFs.  And we all know they always tell the truth.  

Don’t worry we still have the NFP client so you are not out of a job yet – at least one in which you can tell the truth.  Since the NFPs are not for profit they must have room for the truth.  Not so fast.  Profit might be a measure of success for the CSC but it can also be a motivating force in the running of  a NFP, maybe not so much in the way of monetary gain, but typically in the form of prestige and power.  NFPs and those who run them, love to make the right decisions.  Right decisions are good and make you look good and feel good.  Wrong decisions are bad and help you loose your job or get voted out of office.  The problem with NFP is that they typically are heavily bureaucratic organizations that have been built up over many years on a combination of perhaps good decisions but also many bad decisions.  When bad decisions have been made they tend to hang around for a while.  Dirty laundry is everywhere and hey it smells like a locker room in here.  But I digress.  The trouble is, no matter how many good decisions are currently being made, if an analyst comes in and starts asking questions, sooner or later the bad decisions will surface.  Maybe something can be done, maybe something can’t be done, but the dirty secrets make everyone feel uncomfortable.

So even if there is a clear problem with several courses of action, the best course of action – or the course of action determined to be the best – might require airing out the decision that is causing the smell.  A recommendation of this nature, while satisfying to the analyst as the truth, might not be the path the NFP client wants to take.  Sometimes it’s simple if the particular client was not the client when the bad decision was made.  But many times, it was the client themselves who made the bad decision in the first place.  At these moments for the analyst, thinking about opening up that bar down in the Bahama’s doesn’t seem like such a wild idea after all.  Yes, organizations can profit from the truth, and they should.  However, the third principle for seeking the truth is being aware that the truth might not be sought.

The Zealot and His Disciples

Two years prior to the fateful board meeting, the analyst and the PM were in the same meeting foreshadowing the showdown yet to come. We both had the opportunity to be in that very same room. Let’s listen in. “I’m a believer and a zealot and I will tell you right now that I am here to influence you.” These were the words from the PM as they sat at the conference table, trying to shake off the early morning with coffee. Within 5 minutes they were both feeling like believers as well.

Were they drinking coffee or were they drinking grape Kool-Aid? His words were powerful and resonated with the sound of a corporate hero, someone who had been with the company many years, had fought many battles, and was promoted to his position on the basis of the trust he had gained from the corporate structure. “My job is to produce things of great value for the corporation.” For those of us languishing behind stacks of our paper on our desks we have heard the tales of the great ones, those in the field or the lab either landing the big contract or patenting the next invention that will earn the corporation it’s next billion dollars. And here, this morning standing before them was a legend – or so he said.

The charisma of such a hero – those who brave the depths of the commercial marketplace to take the fight to our enemies – gives us another glimpse of the excitement that is capitalism. Instead of being face to face with the empire builder in the form of someone creating and growing an organization as mentioned previously, we were now being addressed by the PM also known as a Product Manager for a corporate product line. They to are empire builders but typically they have a reason that transcends growth for the sake of the seats on an organizational chart – and they will tell you this fact. They transcend the organization because they bring value with a product – the holy grail of the organization in their minds.

Something of such value that everyone else should bow down before them, as they, guardians for the chalice, the Knights Templar by another name, The Zealot as was his nickname out side of work, enters the room. As the Zealot spoke he gave credit to those above him. “Our CEO is a visionary,” was a powerful sound bite. “The Chief has a profound vision of the future,” was another. It seemed to me, however, that those we label as visionary are the leaders who we believe are preaching closest to our own view of the world. We believe this particular PM indeed has a vision, however after a 50 minute sermon I’m not sure he knew that his vision is really about keeping investment dollars flowing into his program and not about winning a war in the corporate market place.

But this blog is not about a zealot trying to keep his program funded. If we had a nickel for every PM we’ve had to listen to ramble on about their product… rather, this essay is about truth. Maybe the PM is right; maybe he is wrong, the question is can we find out? This essay is about how analysis in an uncertain world can be used to shape decisions – and influence decisions the correct way. The promise of analysis has always been about removing bias, removing uncertainty, and delivering the right answer to a decision maker. What we are calling discovering the truth.

In reality our analysis is used to influence friends, bolster intuition, and defend decisions that were made for reasons other than what was revealed our analysis. Let’s get right down to it. Human beings are trained and equipped from preschool to make simple decisions. They can compare two things. Round pegs and square holes. Right choices and wrong choices. Good people, bad people. It’s easy to compare two things. It’s not easy to compare multiple and complex competing alternatives in an uncertain environment. It is easy to change analytic methods as we continually move between mathematical models. It’s not easy to use the right model or even use it correctly in a such a way as to produce analysis that a decision maker can use in a credible manner that can width stand significant scrutiny.

It’s easier to win friends with political rhetoric then it is to change minds with logical arguments, even with the backing of sound analysis. So friends we tell you today that we are also here to influence you – and run the risk of being exposed for our very own brand of zealotry – perhaps we are indeed worshiping at our own holy grail. But, we have a vision related to the future of analysis and its about discovering the truth. It’s about how we might go about producing useful analysis – analysis that helps a decision maker reach a conclusion for the right reason, not for the wrong reason. It’s about helping the decision maker reach the right conclusion by making them a part of the analysis – not a recipient of the analysis product. It’s about helping the human understand complex alternatives in order to make an informed decision. It’s about discovering the real truth, not the truth handed down to us by some dime store philosopher preaching his form of the Gospel with legions of Cool-aide drinking disciples handing out samples.

Good analysis will discover the truth and make it available to not only the initiated but the casual observer as well. It will be presented in clear and intuitive terms – not cloaked in rhetoric and dressed up by the sales man telling you he knows the truth and has done your thinking for you.

The next principal is simple. The Zealot always believes they must do our thinking for us – “You can’t handle the truth” as Jack Nicholson exclaimed at the end of the movie, A Few Good Men. But you can handle the truth – when the truth is revealed it doesn’t require lipstick to make it look good. When revealed the truth is self evident –it is our job as analysts to strip away the hype and find what truly matters. What are the meaningful measures that must be taken into consideration. Not what are the good things that people want to hear, taste, and smell. Tell them the truth, they can handle it, and they will appreciate it.

The Empire Builder

Most people with problems to solve are in the middle of building something. Sometimes it’s a small empire, sometimes it’s a large one. But it is, in almost every case, an empire. It is, in our country, the capitalist way. The bigger, the better. If you are not growing, you are dying. There are many clichés. Many will confess that they are building an empire, some will deny it vehemently, some will not realize they are even doing it. With very few exceptions, they all are in the process of building. Just to note, there is nothing inherently wrong with empire building and there is nothing inherently wrong with falling into one of the three categories of empire builder, depending on what you do with your empire might have ethical implications but that is not the subject of this blog, that comes later.

This blog is about pursing the truth in order to solve problems. If the empire is the problem it is quite conceivable that the builder will not be able to handle the truth, which might be the case where there is, in fact, something inherently wrong with the empire, but hopefully that is a remote case. More than likely, it is the pursuit of something other than the truth in the blind desire to build the empire that will result in something that is ethically questionable. There will be much more on that case later as well. For now, however, you have just received a phone call because there is a real or perceived problem to be solved and you are the solver of problems, although—professional analyst sounds and looks better on your resume. As you put down the phone you realize it’s almost time to go to work. But first you want to read a little more about empires.

We know the Rise and Fall of the Roman Empire, not because we have all read it, but because we know time and time again that history repeats itself. It’s not just that way with Dynasty’s. It is the ubiquitous the cycle of life (sorry Lion King). Everything around us is either growing or dying. Look around you – are you trapped in urban sprawl as your city planners struggle to decide if they should approve yet another tract of housing or strip mall? Is your social group attempting to attract new members and/or seek donations? Is your company trying to land new contracts by pursing business with new clients? Is your boss trying to increase the size of your division? You haven’t been called necessarily to solve any of these problems; you have been called to help a very specific client with a very specific problem.

But your boss did mention that it was an important client whose business was very important to the company. Should you know anything about that client? Should you ask your boss more about the client before you respond? Perhaps you should ask your boss just how important and how long they have been a client? Perhaps you should ask if your boss knows about the problem to be solved. Chances are, you now know everything your boss knows. It is important, as you move forward, however to know from where you came and to where you are going. Why? Because the problem is about to become your problem. And if you are to attempt a solution that allows you to pursue the truth and maintain your integrity you will need to know your level of independence from the issues.

Understanding your level of independence is our second principal for pursuing the truth. And it is why you must know your place in the empire. Almost everyone is inside the empire or nested empires – it is practically impossible to not be connected in some way. Fortunately however, for now it is sufficient to simply understand you’re level of independence as you begin to assess the problem. You must reassess, however, throughout your work, this level of independence. If you don’t know where the lines are, you will no doubt become part of the problem itself and hence part of the solution. The truth will have been lost before you even leave your office.

The Analyst: Bold--Gusting to Arrogant

A colleague once told me that an analyst must be bold--gusting to arrogant. Those are some of the wisest words I’ve ever heard with regard to this profession –that of the professional analyst. Typically, or stereotypically, we are meek, akin to the computer geek or CPA. We can be seen perhaps lunching with the engineers, all of us sporting pocket protectors with outdated neckties and short sleeve shirts. A great professional analyst, on the other hand, must by definition be bold and not meek. It is their job to question everything, and then with the important knowledge gained, help those who have hired them to either solve their problems or answer their questions.

Unfortunately, to do this correctly, we must gain considerable knowledge in the pursuit of an answer. Often we become so knowledgeable that our understanding exceeds that of our employers. If we show our cards too early, we can be labeled as arrogant. By definition though, if we indeed have a solution to our employer’s problem, we are smarter, at least on the subject under consideration. Therefore we must be bold to be heard.

If, however, there is something missing, and we are not smarter than our employers and our solution is either inadequate or is rejected, we have either failed to understand the problem completely or we were hired to help with a problem that was not intended to be solved (Let’s call this the Gauntlet for you Clint Eastwood fans). If either happens we work harder, quit, or slip into analyst purgatory. Analyst Purgatory or AP is that waiting room on the way to Hell where we might get a second chance. When we find ourselves, as we inevitably do, staring at the bleak walls of AP, we still have options we can take.

We can join with the devil thereby completely sealing our fate by becoming the chambermaids of our master to assist him with his evil plans. We can become prostitutes by selling our services to the highest bidder, remaining neutral on ethics, but affirming the remainder of our lives will continue to exist in the AP of our creation. We can leave the profession completely. Or, as I would like to be the option you have chosen exhibited by your desire to participate and contribute to this blog, you can attempt to elevate our profession, with me, above the charlatans and whores that practice all around us.

We must pursue truth in everything we do. We must leave no stone unturned in the pursuit of the truth; to strive for less would be to strive for something incomplete, and most likely something to label as incompetent. For if our recommendations are followed, and they are not the truth, what is left is blind luck. We might as well have rolled the dice, kicked up our feet, took a nap, and waited for a paycheck. But to pursue the truth, to really find the answers to the questions we seek, we must be bold, very bold. To do so requires rock solid faith, abundant energy, and complete competence. When you have faith, energy, and competence you are bold – perhaps not yet gusting to arrogant, but you will gust soon, perhaps alienating those around you, perhaps winning the respect you deserve – there will be time for humility after you’ve solved world hunger.

In the mean time there is no room for the incompetent in our profession. Why would those with problems to solve choose to hire the incompetent? That would be the height of irony. It unfortunately, is not just a good question, it is a reality that happens everyday, and there are reasons it happens, some, like the sinister Gauntlet will be addressed in the days and weeks to come. Other reasons will remain a sacred mystery, but most, fortunately are simply the landscape of a profession that has forgotten how to discover the truth.

We are all ignorant when we first come upon a new problem to solve; as we start we know very little about the problem. There are other essays and professional works that can help you discover the problem and how to define things, then later how to solve things. This blog is not about the analysis process proper. This blog is about seeking the truth. So as we begin each problem anew, ignorant but hopefully with motivation, it is our next step that will dictate competence or incompetent behavior. We cannot predict which. Nonetheless, our first principle should be clearly stated here--

Discovering the truth is not easy, but if you strive for it, pursuit of the truth keeps you closer to path of competence rather than the one of incompetence. However, there are no guarantees.

So let’s begin boldly and gust arrogant as we go along.

Frameworks for Understanding

Orge Here --

Thank you Mooch for the opportunity to learn and share. Though I have long been an amateur thinker, the job title I have had for the past two years, "management analyst" asserts that I am now a professional thinker. Consequently, it is in my best professional interest to learn from my peers and avail myself to them, to the extent I am able. For now, I will not detail my background, because my current thought is that I want the ideas that I present to stand on their own. As you will see, they do require critique, maturing, and possibly expansion.

So, friends, without further introduction, here is one of my thoughts on "analysis." Analysis being defined as: seeking the truth of a particular matter with intent to understand in order to make useful decisions. To think productively, we need to understand the frameworks we use (one will not do, for reasons that appear intuitively obvious to me – let me know if you disagree; and to omit any one puts daylight between our analysis and reality/truth). Please let me know what other frameworks I have over looked. I will grant that they overlap, but I consider each a primary driver for how things work.

Frameworks:
Deterministic (because some things happen because another event caused them)
Random (because some things happen on a randomly distributed basis)
Chaotic (because some things happen on a non-random, non-periodic, unpredictable basis) Deliberate (because of free will)
Bias based (because we are human)

My goal is to increase in knowledge, understanding, and (some day) wisdom. Ogre

Lies, damn lies, and ...

“There is always a well-known solution to every human problem—neat, plausible, and wrong.” H.L. Mencken

On the subject of analytic and scientific truth…I haven’t entirely sussed out what this means to analysis, to analysts, and to me, but it leaves me questioning an awful lot of the things I’ve seen, read, and done. If nothing else, it leaves me with an even greater skepticism than I had when I woke up this morning.

I've been reading a little book titled Wrong: Why Experts Keep Failing Us--and How to Know When Not to Trust Them, by David H. Freeman. In this case, experts refers to “scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, high-powered consultants, health officials, and more”—pretty much everyone who offers advice or conclusions in other words—and the book is all about the many and varied ways they (and we) get it…well…wrong most of the time. According to Freeman, we live in a world of “punctuated wrongness,” a world where, according to one expert (the irony here is intentional on my part and acknowledged on Freeman’s), “The facts suggest that for many, if not the majority, of fields, the majority of published studies are likely to be wrong…[probably] the vast majority.” This is a pretty stunning claim. In fact, if I think about this issue as a mathematician—the area of emphasis for most of my formal training and publication—I’m simply staggered by the claim. But my field is a little special I suppose, since “truth” (within the axioms) is pretty easy to spot. We may be the only discipline wherein one can actually lay legitimate claim to prove anything since ours is probably the only completely deductive intellectual endeavor. (That still doesn't mean we have any greater access to Truth, though.) In other fields of inquiry, the fundamental process is inductive—observe, hypothesize, observe, adjust, observe, adjust, etc.—and claims to proof are problematic in the extreme—which doesn’t stop anyone and everyone from using the phrase “studies show” as if they’re quoting from the Book of Heaven. But I also have a fair bit of training in statistics—both on the theory side and in applications—and one of Freeman’s explorations of “wrongness” really hit home.

Why do we use statistical methods in our research? Basically, we want to account for the fact that the world—as we observe it—is stochastic (although whether it is fundamentally stochastic might be an interesting debate) and ensure the measurements we make and the inferences derived from those observations are not (likely to be) statistical flukes. So, when we make a claim that some observation is “statistically significant” (not to be confused with a claim that something is “true”—a mistake we see far too often, even in our professional crowd) we mean there is some known probability—the level of significance—that we'll make a (Type I) mistake in our conclusion based on observing a statistical fluke. So, for example, a level of significance of .05 indicates (kinda sorta) a 5% chance that the results observed are the result of chance—and that our inferences/conclusions/recommendations are “wrong.” 1 in 20? Not so bad. How do we make the leap from there to “the majority of published studies are…wrong?”

As an exercise for the student, suppose 20 teams of researchers are all studying some novel hypothesis/theory and that this theory is “actually” false. Well, (very roughly speaking) we can expect 19 of these teams will come up with the correct ("true negative") conclusion and the 20th will experience a “data fluke” and conclude the mistaken theory is correct (a "false positive"). With me so far? Good. The problem is that this makes for a wonderful theoretical construct and ignores the confounding effects of reality—real researchers with real staff doing real research at real universities/companies/laboratories and submitting results to real journals for actual publication. Freeman has estimates from another set of experts (again with the irony!) indicating that “positive studies” confirming a theory are (one the order of) 10 times more likely to be submitted and accepted for publication than negative studies. So, we don’t get 19 published studies claiming “NO!” and one study crying “YES!” We see 2 negative studies and 1 positive study (using “squint at the blackboard” math)...and 2 out of three ain’t bad. (Isn’t that a line from a song by Meatloaf? I think it’s right before “Paradise by the Dashboard Light” on Bat Out of Hell. Anyway…) The other 17 studies go in a drawer, go in the trash, or are simply rejected. Cool, huh? Still…we don’t have anything like a majority of published studies coming out in the category of “wrong.” In the immortal words of Ron Popeil, “Wait! There’s more!”

Statistical flukes and “publication bias” aren’t the only pernicious little worms of wrongness working their way into the heart of science. “Significance” doesn’t tell us anything about study design, measurement methods, data or meta/proxy-data used, the biases of the researchers, and a brazillion other factors that bear on the outcome of an experiment, and ALL of these affect the results of a study. Each of these are a long discussion in themselves, but it suffices to say “exerts agree” (irony alert) that these are all alive and well in most research fields. So, suppose some proportion of studies have their results “pushed” in the direction of positive results—after all, positive studies are more likely to get published and result in renewed grants and professional accolades and adoring looks from doe-eyed freshman girls (because chicks dig statistics)—and suppose that proportion is in the neighborhood of an additional 20%. Accepting all these (not entirely made up) numbers, we now have 5 false positives from the original 20 studies. If all five of the “positive” studies and the expected proportion (one tenth) of the “negative” studies get published, we expect to see 7 total studies published, of which 5 come to the wrong conclusion. 5 of 7! Holy Crappy Conclusions, Batman! (Don't go reaching for that bottle of Vioxx to treat the sudden pain in your head, now.)

Freeman, following all of this, goes on to warn we should not hold science as a method or scientists themselves in low regard because of these issues. They are, in fact, our most trustworthy experts (as opposed to diet gurus, self-help goobers, television investment wankers, and other such random wieners.) They're the very best we have. Scientists are at the top of the heap, but “that doesn’t mean we shouldn’t have a good understanding of how modest compliment it may be to say so.”
CUMBAYA! It’s no wonder we poor humans muddle through life and screw up on such a grand scale so often! I need a drink, and recent studies show that drinking one glass of red wine each day may have certain health benefits…

Did Boyd Get Entropy Wrong?

Orge here -- I suspect that Boyd, as bright as he was, and as much as I respect what he accomplished, is not so good on thermodynamics. While I don't intend to discuss entropy (the part of a thermodynamic equation that shows irrevesability and how far the process is from the ideal adiabatic process) at length, I want to ask if anyone else remembers fom their thermo days the concept of entropy and agrees that Boyd got it wrong. I figure that if this discussion can teach me in this, perhaps it will be a good forum to learn other, more important things.

Send in the Clowns

Potentially everyone is a problem solver. We all have to make decisions about things in our life be they major decisions like the purchase of a car or house, life changing decisions like whether to get married or not, change jobs, or have children, or very simple decisions like should you get out of bed today, OK sometimes that's not a simple decision. But at the end of the day, we really hope that the decisions that we make are the right ones. Unfortunately the Program Manager introduced in this blog on Sept 8th might come across as overly bad or someone with nefarious intentions. That could be true but most likely his behavior could simply be the result of bad decision-making.

It's painful to know we make bad decisions, sometimes very bad decisions, and we tell ourselves if we only knew this, or we only knew that, we would have decided differently. It hurts even more if we are exposed for our incompetence. The point is, whether we know it or not, we have been solving problems and making decisions all our lives. Marketers know that humans in general would rather move through their day making as few choices as possible – for some reason our primordial genetic make-up wishes for us to form habits. It probably has something to do with safety. If I always eat the berries from this tree I will not die, or something like that. But the fewer decisions we are forced to make the more comfortable we are and that is what sales people want us to do. If they get us to choose their product, we don't have to decide again every time. Because of the number of choices we make everyday we have learned to make decisions and evaluate choices almost unconsciously. We seemingly make decisions based on a gut feel but it more likely experience from the past that has crept into our unconsciousness compelling us in a certain direction. Or we consciously choose based on some criteria such as price, but really it's the packaging we just don't know it.

No one is immune from making bad decision, with the possible exception of Warren Buffet.. We've all made bad decisions and it doesn't make us feel very good. The worse our decisions the more we tend to feel like a clown. We don't get up in the morning wishing to make a bad decision in our personal lives and we certainly don't get up hoping to make a bad decision at the work place. We've learned to accept bad decisions in our personal lives but the ones we make in the work place are the ones that can cost us our livelihood so we tend to stay awake and consider more options.

If you happen to work in the government you could be making decisions that affect many more people and although most people have nothing but the best interest of their countrymen in mind, poor decisions are still made. If we give everyone the benefit of the doubt, that they are competent decisions makers, and most likely highly intelligent, why are we awash in seemingly incompetent decisions? This is a very good question and one this short commentary will attempt to answer in time. It's not about the best analytic approach for decision making because there are many. It's not about intelligent people working the problem because there are plenty of competent, well intentioned, and well-educated people thinking about things and placed in positions to make big decisions. It's about being committed to a search for the truth. The title of the entire blog in fact...Truth in Our Profession and how to get there.

This is not some search for higher value or the meaning of life type of truth. It's not a search for a moral correctness in our decision making either. It's about piecing together a puzzle the picture of which we cannot yet see. But wondering enough and being committed enough – for whatever reason – to reveal what is truly in the picture. Commitment to the truth will unlock doors of curiosity and it is through these doors that discovery will occur. Sometimes it will not and you may have to work harder, or you will run out of time and have to do the best you can with what you've got, but you will have done your best and perhaps caught a glimpse of the truth. So then his short paper is about the journey to discovering truth. Striving for and equipped with only a few extra rules in order to make a competent try, elevating the profession for all of us. You may not make it, at the end you may still feel like clown, but it is worth the try if for no other reason then you don't have to sit next to the PM with the big floppy shoes on at the retirement seminar. That clown is already there...let's hope he is not waiting for you.

Based on Actual Events

The account described in our 8 Sept Blog was based on actual events. The analyst is real. He is an intelligent and extremely loyal member of a corporate team. He is an operations expert who had spent significant time in the field, as well as in school to become an analyst. A practical if not recognized expert on both the subject to which he spoke and analysis to which he conducted. Before the study even began the pudgy finger poking PM was asked the simple question...If the results of the analysis are negative towards your program what will you do? The PM was confident and optimistic in his response that he was a company man and a team player. He wanted only the best for the organization and he would support the study to the fullest. It was only a matter of days before the PM started stone walling the analysis team. Despite the barriers placed in their way, the analysis team moved forward, uncovering and discovering many interesting and contrary facts along the way. When the study was complete the team dutifully briefed the PM. The team was immediately descended upon by legions of critics attempting to pull apart the analysis. Briefing after briefing was conducted and the analysis and analyst stood firm – yet no one would tell the CEO that a mistake had been made. Lips were sealed until it was apparent that another company was becoming a potential competitor. When the analysis team could no longer sit on the truth it was taken to the CEO. The results are now history. The analyst was promoted early and will return to operations. The PM was last seen attending a retirement seminar. And the Truth shall set you free…

The Program Manager and the Analyst

The division analyst finished his presentation and returned to his seat. The Chief Executive looked around the room and said, “Gentleman, where was this dissention last year when we decided to fund this program?” He looked at the presenter, who was keeping his eyes glued straight ahead, trying not to lock eyes with the others seated a the table, the ones in the room to whom the executive was addressing. “I say again, where was this dissention when we invested a large percentage of our annual revenue in this program?” The table was dead quiet except for the imperceptible sound of building blood pressure in the Program Manager’s head. “Gentleman”, the executive began again, “It’s very rarely that we see the truth in this company. Sometimes we glimpse the truth, but most of the time we are chasing a target that will not sit still. Today that has all changed with the presentation of this analysis. For your sake, for the sake of this company and our stockholders, I hope it is not to late to turn this ship around.” He thanked the division analyst for his candor and swiftly left through the door in the back of the room. No sooner had the door swung shut than the PM stood and raced around the table to confront the analyst. The analyst only had time to spin his chair toward the approaching manager. Producing a pudgy index finger the now fuming man poked the analyst in the chest and spoke rigidly, “What gives you the right to brief the CEO without my permission? I did not give you my permission.” The analyst looked straight into the PM’s eyes and said, “Sir, I work for this company and the stockholders of this company. Whom do you work for?”