Decision Analysis

I have taught a business statistics course for about two years at Southern New Hampshire University, a small private school in Manchester. The course covers confidence intervals, hypothesis testing, regression analysis, forecasting, etc. All of these techniques have been applied poorly, even disastrously, at times. I try to stress to my students that a quantitative technique pointing to a nonsensical decision may be wrong, and they should check their assumptions and their arithmetic again. (They should be very skeptical of a quantitative analysis that seems to justify an immoral decision.) However, the lecture which always makes me feel dirtiest is when I explain decision analysis.

One aspect of decision analysis lays out the choices which face you on the left side of a table and the possible futures you envision on the top of the table. For every entry, corresponding to a particular decision and a particular future, you determine a cost or a payoff. There are a number of methods for deciding which choice will most likely lead to the best outcome. These methods can take into account how much risk you are willing to accept or how probable you view a particular outcome. There are alternate methods for laying out more complex decision paths, but at its heart, decision analysis quantifies the payoff or cost of your decisions.

It is very hard to argue that a careful ordering and consideration of your choices and their likely outcomes is a bad way to approach decision making. In fact, when Israel was readying for negotiations after the 1973 War, they used decision analysis to prepare the negotiating team. The participants didn't view the payoffs listed in their tables as particularly useful; however, the process of preparing such a table also forced careful consideration of all options and what their global and regional implications were. The decision analysis was constantly updated during negotiations, and the participants also discovered a lot of value in seeing how their estimations of payoffs changed as decisions were made and events moved forward.

So why don't I like decision analysis?

I find it objectionable to reduce a complex process to a single quantity. There are many reasons as a mathematician I find this incorrect and unjustifiable. The claim/caveat is that you shouldn't pay too much attention to the exact numbers you get when performing decision analysis and that the process is the important part. It is even provable that you will reach the optimal decision when your payoffs are accurate. However, accuracy is very hard to guarantee when you are estimating the impact of an advertising campaign on potential market share. If people spend significant time on an analysis leading to a particular number, then it is hard to argue that number should be ignored. It also appears very easy to structure the analysis so that the favored decision is the one with the highest score.

I explain to my students that they should always document their analyses thoroughly, so that others can look at their assumptions and see whether they agree with the inputs, and therefore the outcome. For decision analysis, this documentation seems most useful as a way to show why you made an incorrect decision when things go wrong. Just as it seemed wrong for judges to choose nits in the statutory language to justify a ruling, it seems wrong for an analyst to justify their decisions with an easily gamed process whose documentation serves mostly as cover if your decision is wrong.

I try to find examples of improper use of all the techniques I teach, so that I can make students wary of non-intuitive decisions justified solely by a single numerical method. I have yet to find a documented instance of badly used Decision Analysis. There are a huge number of websites, organizations, and consultants extolling the virtues of Decision Analysis. This also bothers me, since I don't believe any technique is infallible.

We should also keep in mind the comment of an Israeli friend. He said the government spends a lot of effort on Game Theory and Decision Analysis to determine how to deal with the other players in the region. However, it is not at all clear that their adversaries are playing the same game.