I am selecting a CRM package for a bank. I asked my colleagues how they’d gone about it, and got 8 responses. Every single one of them had the same weighting approach: Take a huge list of criteria, assign weights, score each package, calculate a weighted-average score, pick the highest one.
As I mentioned earlier, I think weighting is a lousy method. (See Errors in multicriteria decision making.) You can’t say “I picked this package because it has X, Y and Z features, which the others don’t.” You can only say, “Oh, overall, it has the highest score…”
The scores and the weights are subjective. You spend ages arguing between a 3 and a 4. You can manipulate them very easily. And you end up having to revise the scores many times to get to the answer you want.
Since I now had an opinion, I put my foot down, and said, “Here’s what we’ll do. Let’s make a list of essential criteria. They will all be YES / NO questions. Any package that doesn’t meet any criteria is knocked off. That’s it.”
This may appear simplistic, but it isn’t. You see, at the end of the day, only a few criteria really matter. Ideally, you just pick these, and compare packages against these. Since you don’t know which these are, you make a bigger list, evaluate them all, and then realise the truth.
Sometimes, you have too many criteria. Then none of the packages make it, and you have to sacrifice some of your criteria.
Sometimes, all of them make it. Then you can choose to enforce more criteria. Or maybe not. If all of them meet your criteria, just pick the cheapest one.
Internally, we were convinced of this approach, and took it to the client. Things were fine for a week. Then, complaints started trickling in.
- Fear: “I’ve already used weighting in earlier package evaluations. Now, if you use filtering, it’ll make my earlier evaluations look bad…”
- Uncertainty: “I don’t know… what if there’s no YES / NO answer? What if we need shades of grey?”
- Doubt: What if it lets everything through? What if it rejects everything?
Uncertainty is the most popular objection.
“What if we need shades of gray?”
I always ask: “Any example?”
“Well, you know… it can come up.”
So I give them an example, and explain how it can be broken in to sub-questions.
“Well, yeah… but just to be on the safe side, could we have a score?”
The exercise is still going on. I haven’t seen a valid concern yet. What’s interesting is, everyone is hesitant about filtering, but no one can defend their objection.
Anand, while weighting is just a tool, you can’t make decision making a science. Well, i know decision making is called a science by the academia, but if it could be resolved by metrics, then every decision taken could potentially be right, maybe at that point in time. But i don’t agree with that as the results of actions taken are known only much later. I don’t know if i am conveying this correctly. If you take filtering, then sure, you will come up with must haves and you will be able to come down to a few choices. Where i am going with this is irrespective of the tool weighting/filtering, the most you can do is come down to a top 3 or 2. For example most CRM systems today fullfill the campaign to lead to opportunity to quote to order to support process. You will then break it down either to industry based sub-processes or ability to handle volume transactions/enterprise/smb ..so on. My point being you can do this by either method. Finally it’s your gut. Maybe it’s just the sales guy in me talking.
Sai, if you’re saying the methods are equivalent in their outcome, I’d say MAYBE. But with filtering, I can say WHY I’ve knocked off a package. I can’t with weighting. Both eventually try and reflect the subjective process, but filtering does it by trying to identify the EXACT reason you want to reject something, while weighting tries to quantify the feel for it. I see the value of trying to pinpoint the reason. It may be unquantifiable, but filtering forces you to try, and weighting leads down the easy path. But unless we get into specifics, we’d just be talking hypothetically. Let’s have a chat…
I guess this will be a two way approach. You start with elimination process using set of questionnaires and Yes/No responses. This will help you to narrow down to tools which meets mandatory features. After that you can use weighted approach to zero in on the tool.