Errors in multicriteria decision making

I talked about my approach for multicriteria decision-making, and mentioned that it was fundamentally flawed. Here’s why.

The charts above compared two industries. The bigger the area, the more favourable the industry. The underlying assumptions being:

• The criteria are comparable. (Points at the same level are of comparable importance. Twice as large is twice as important.)
• All (and only) relevant criteria have been included.

In this particular example, I know for a fact that both these assumptions are invalid. And in every case I used this methodology, the assumptions fail.

You won’t draw the criteria to scale. We used revenues and growth as two parameters, and marked each industry as high, medium or low. The scale for revenue was Rs 100 cr, Rs 500 cr and over Rs 500 cr. The scale for growth was <5%, 5-10%, >10%. We picked this scale in order to fit the range well on these graphs. Not because Rs 100 cr of revenue was worth about the same as 5% of growth. And yet, that’s the implicit trade-off this graph is asking us to make.

We also had very qualitative criteria, like “Capability” (KSF), and they were compared head-on with growth and revenue. Using qualitiative criteria is not a bad thing. But when the visual makes you trade-off capability against Rs 500 cr of revenue, I feel queasy.

You will miss important criteria. Usually, the process for identifying criteria is bad. “Think of every criteria you can” was our standard approach. In this instance, in our first iteration, we had a dozen parameters. We showed it to the client. They said, “Look, our Chairman likes these industries a lot. He doesn’t like that bunch. We’re much more likely to focus on the ones he likes.” And that’s absolutely important! We ended up adding a “Passion / vision” based on the fit with the company’s existing businesses, and that proved the deciding factor.

Another time, I built an entire model on which project to outsource based on 10 parameters. (It was everything I could think of at the time.) The one that I missed was, “When is the project starting?”. It turned out that this was the most important criteria. In fact, it was the only important criterion. If I’d simply said, all projects starting after 1-June-2006 can be outsourced, I would’ve been 90% right.

You’ll keep the irrelevant parameters. This is the worst of all. Even after we learnt which the important criteria were, we didn’t throw away the useless parameters. We never throw away hours of work, even if it’s useless. So the model keeps bloating, and the irrelevant criteria influenced the shape of the graph more than the relevant ones.

Another problem is that this methodology cannot answer questions concisely.

“Why did you knock off Industry X?”

“Oh, because on a cumulative score against revenue, growth, lifecycle, capability, passion and 10 others, it scores less than 45 points.”

A good answer should be short. For GE, it would be “You’ll never be number one.” For HP, once, it would’ve been “It’s not where we can excel technically.” For Warren Buffet, it may be “I don’t understand the business.”

After these experiences, and based on hindsight, I’ve come to believe the following about MCDM (multi-criteria decision making):

1. Remove irrelevant criteria. Usually a few criteria make the decision. The rest don’t matter.
2. Filtering works. You don’t optimise in MCDM. You’re trying to do well against criteria that are often not comparable. You’re better off if you filter out unacceptable levels of criteria, and treat what’s left as acceptable.
3. Use a decision tree. Don’t compare options. This is the basis of fast and frugal heuristics. It keeps focus on the important criteria, and makes the process easy to implement / explain.

I’ll let you read up on fast and frugal heuristics. I’m convinced it’s the best way to make decisions based on multiple criteria in the scenarios I’ve worked on.