Here’s an example of what I mean, posts on forums where claims like the following are made.

Eliminate the number of potential risks threatening database development and deployment by 60% and reduce deployment costs by 95%.[sic]

What does this even mean? Reduce what deployment costs? Against what baseline? Is this an average observable saving, if so, where’s your supporting data?

Which development and deployment risks are you talking about? Which of these risks does your tool mitigate? How much effort is required to mitigate those risks? Is it really worth it?

In this particular case a link was provided to a press release. Still no sign of any substantiating data. What about their web site. Nope. Not a thing.

Or how about this one,

AccuRev will eliminate up to 90% of normal merge activity.

What is ‘normal merge activity’? On what basis does this person (who was a representative of the company) make this claim? 90%? At first blush this is a remarkably specific claim (read it carefully and they’ve included those marketing weasel words ‘up to’), it must be based on some pretty sound data. Or maybe not. Maybe it’s just one of those made up statistics again. Yeah. That’s what it was. I contacted the person involved and they continued to be vague and push out marketing speak, but failed to come up with anything more substantial than ‘Our customers have typically seen between 70-90% reduction in merging activity.’ Another unsubstantiated claim, and even if it’s true is still seems to be based on perception and hearsay rather than hard numbers. (Oh, and 70-90% is a heck of a wide range and boldly claiming 90% in the original statement is misleading even if you accept the more qualified range.)

Ah, you say, but it claims ‘up to 90%’. Well that’s BS too. I can save up to 100% by not doing any parallel development at all. No merging, no merging costs. Alternatively, if your merges are normally done by a bunch of alcoholic monkeys, then I guess saving 90% is possible by using properly trained software engineers and following good branch and merge practices.

And if they save you 0%, well, that’s up to 90% too. 90% is just a made up number plucked out of some marketing guy’s butt.

Now, I understand that marketing people like to make these claims, they look impressive on paper, but they’re absolutely useless to anyone with an IQ greater than their shoe size who is trying to evaluate tools. Frankly, if you’re going to make ridiculous, unsubstantiated claims then I for one don’t want to be doing business with you. I mean, if you can’t be up-front and straighforward at the beginning of our relationship, how trustworthy are you going to be when it matters? (I’m tired of hearing people say things like, ‘but the sales guys said this tool would solve this problem’. Really? The sales guys said that did he? And you didn’t think to verify that before you bought the damned tool?)

The stock reply of course is, ‘but everyone does it and if we don’t make these claims no one will pay any attention.’ Really? You’re falling back on the ‘everyone else does it’ defence? If that’s the best you have then, once again, I’m not sure I want to do business with you. (As my old Mum says, ‘if everyone else stuck their head in an oven, would you?’)

Another justification goes along the lines, ‘but there is no good way to measure X’. Now that may be true, but that doesn’t give you license to just make shit up. If there’s no good way to measure something then claiming to reduce that something by a specific quantity is… well, bullshit. ‘Oh yes, I can reduce your unspecified cost by 53%.’ How the hell can you say that with a straight face?

These sort of unfounded claims really make my blood boil. So I’ve decided that whenever I see unsubstantiated claims I’m going to call them on their BS. It’s simple enough, just reply to their post asking, ‘I am interested in your claim that X will say me Y%. I wonder if you could provide us with the data on which this claim is based, I’m sure the community will be interested to see the methodology and metrics you used to gather this data, as well as the data itself. In an industry where such benchmark studies are rare, or at least seldom published, your data will be most useful.’

I bet not one will reply with anything remotely like an independent data set that substantiates the marketing claim. And if they do, all the better because I for one would love to see this sort of data collected and published.

Remember, marketing statistics and claims are generally worth exactly what you paid for them. Zero. And I have data to support that.

Sometimes the questioner will provide some clues about their specific circumstances, but inevitably the answers will reiterate, “it depends”.

The correct approach to this problem is not to solicit ad hoc advice from internet fora, but to consider your own situation. The process is simple enough but requires that you put some thought and effort into finding the right tool. It boils down to the following steps.

1. Draw up a set of criteria to be met by the tools you intend to evaluate.
2. Draw up an evaluation matrix.
3. Create a shortlist of possible tools.
4. Evaluate the shortlisted tools against your criteria.
5. Get vendor demonstrations.
6. Get evaluation license installations.
7. Finally select, in as objective a manner as you can, the best solution for your predefined criteria.

An alternative strategy is to employ an expert in the ALM tools field, but they will follow this process anyway. Any other approach is utterly pointless. Let’s take a more detailed look at each step.

Generate a Requirements Document

You must have a list of requirements the tool is to satisfy (otherwise you would not know you wanted a tool). Formalise these requirements into a document. Be specific.

If the tool is to support version control then consider whether you need it to support branching, baselining, labelling, parallel development and so on. Do you need it to support locking to prevent more than one user changing a file revision at a time? If so, must this locking be provided at a revision, branch or file level?

Are there specific integrations you require? Must the tool integrate with developer IDEs? Do you have existing build systems that the system must integrate with, or do you perhaps need a system with an integrated build system? Do you have special tool integration requirements?

Does the tool need to track changes (workflow control of change requests)? Or perhaps you have an existing change request system that needs to work with the new tool?

Do you need to support specific platforms? It’s no use looking at tools that run on Windows platforms if your development uses Unix. Similarly you may (as is common nowadays) have many platforms to support. If so, make sure you specify those that must be supported (and perhaps some that would be ‘nice to have’ — there’s no use turning down a perfect fit for 99% of your environments just because one developer uses an odd platform).

Mario Moreira’s excellent  Software Configuration Management Implementation Roadmap (Amazon uk,us) is an excellent place to start in establishing your requirements.

Don’t forget the future! You may have a specific set of requirements now, but talk to your strategy team and find out whether they have plans to move to other platforms in the near to mid-term future. It may take many weeks to find and procure the tools, and once you have them integrated into your development process it can be tricky to change them, so make sure you are buying a tool set that will serve your needs for the next five years or more. (Sure, things may change in that time, but it’s stupid in the extreme to ignore what you know now. So plan ahead!)

The future is, they say, a foreign land and no amount of planning can anticipate every future requirement. Planning ahead is essential, but not guaranteed foolproof. Plan to transition to new tools in the future by including requirements for extracting data from the tool set you are investing in. This is a common mistake, tying yourself in to one system and finding it impractical to move to a new system later. Again, think ahead.

Formalise all your requirements into a document and review it with the stakeholders. Get sign-off from all stakeholders so that you know exactly what you need to find.

Draw up an evaluation matrix

Once you have the core requirements agreed on, draw up a simple evaluation matrix.

An evaluation matrix is a simple grid down one side of which are all your requirements. Next to each requirement put a weighting (a simple multiplier) indicating how important that feature is to your organisation. For example, you could use a scale of 1 through 5 (1 being optional and non-essential, and 5 being absolutely mandatory). Going across the top of the grid will be a series of columns where you specify the tools you intend to evaluate. This matrix can then be completed for each tool as you evaluate it when creating the shortlist (next step).

For each tool, you will enter a value indicating how satisfactorily it meets each requirement (0 for not at all, through to 5 for completely, for example). By multiplying the original factor with the satisfaction measure, you get a more-or-less objective value for how much each tool satisfies each requirement, and thus the degree to which each tool in its entirety meets the needs of your organisation or project. So, an optional requirement will always contribute less while a mandatory requirement will contribute more; and a tool that does not satisfy the criteria contributes nothing while a tool that satisfies the requirement completely contributes more.

There are many ways to adjust the way you calculate the value of each tool’s satisfaction of requirements (by adjusting the weighting factors or the way you calculate the final score), but whatever you decide to do, decide before you start evaluating tools and do not be tempted to fudge the results. We are aiming for as objective an assessment as we can get and it is tempting to see an appealing tool and try to adjust the weightings so that tool scores well. Doing this defeats the purpose of the evaluation; you may as well just ‘buy the pretty one’.

Draw up a shortlist

Okay, now you are ready to start researching tools. First, scan the internet for likely candidates. By all means visit forums and ask questions, but keep them both focussed and general. Ask people to suggest tools in general categories, for example: “I am looking for a change management tool to run on a Unix server that can be accessed from both Windows and Unix clients” or “I am looking for a version control tool that supports branching and merging and allows locking at the revision level”. Once you get these suggestions, go look at the vendor’s website and see how the tool matches your general requirements.

During this research phase you don’t need to be too specific, you’re just trying to get a shortlist of tools to add to your evaluation matrix and examine more closely.

Once you have your shortlist, you’re ready to do some more detailed assessments.

Evaluate shortlisted tools

A quick way to start your detailed evaluation is to send your requirements document (not the matrix, it won’t have any detailed description of your requirements) to the vendors of each tool. Make sure you contact the vendor and that you are dealing with a specific individual (you want to have someone accountable for making sure you get responses).

Only once you have responses should you move on to real evaluations. Oh, and any vendor that fails to respond should be dropped, after all if they can’t answer (or if they refuse to answer) questions then you have good reason to suspect their after-sales support may not be too hot either.

Okay, so you have your responses. Now add these to your evaluation matrix and add in your evaluations for how satisfactorily each tool meets your requirements. Contact the vendor (the person you dealt with when submitting your requirements) and get clarification as required.

Eliminate tools that score too badly from further investigation. (Of course, ‘too badly’ will depend on what you are looking for.)

The final part of your initial investigation is to check the vendor company. Are they well established? Financially stable? What do forums say about their support facilities? How often do they release updates? And, of course, how much will the product cost (based on list price at this point, your procurement department will deal with negotiating the final price of licenses)? Add these details to your evaluation matrix.

Now, and only now, should you approach vendors for demonstrations.

Get vendor demonstrators

Why leave it so late to see the tool in action? Two reasons; firstly, until you have requirements, and a statement from the vendor on how ther tool satisfies the requirements, you are not in a position to evaluate any demonstration. Secondly, it is very easy to be distracted by a generic shiny demonstration (I know only too well that generic vendor demonstrations have the ‘wow’ appeal, especially to non-technical managers, and it’s easy to be seduced). Generic pre-sales demonstrations are designed to look good. The vendor is showing off their tool. Of course it looks good. Of course it runs smoothly (although I’ve seen some howlers in my time). This is fine for showrooms, but terrible for purchasing decisions. Remember, we are looking for the right tool for your organisation, not the tool the vendor wants you to buy.

Once you have good evaluation criteria you can ask vendors to come and demonstrate how their product satisfies your requirements. Make sure you are clear when requesting the demonstration that you expect to see evidence of how their tool meets the requirements document that they have already seen and provided feedback upon. Do not accept a generic demonstration at this point. If they cannot, or are unwilling to, provide a suitable demonstration, strike them from the list — you’ve put in a lot of effort and are likely to be giving them a substantial amount of money in the near future, so they should be willing to show you what you want.

During the demonstration, make sure you see their tool perform to the requirements that they have claimed it fulfils. Get clarification on how marginally met requirements may be worked around (those that the vendor identified as only partially satisfied) and be sure that you see an unambiguous demonstration of all requirements that the vendor declared they satisfied completely.

During demonstrations, be fair. It is unreasonable to expect the vendor to show you everything you need. There will also be requirements that they state they only partially satisfy; give them a break and don’t harangue them for this. Accept the limitation. Discuss options. Make a note and move on.

Once all the vendors have demonstrated their products, re-evaluate them on your evaluation matrix. Add notes on where the demonstration failed to meet expectations, and where the demonstration exceeded expectations.

Eliminate any that fail to meet expectations satisfactorily.

Get evaluation license installations

You should now have a very short list of potential products (two or three if you are lucky). Obtain evaluation licenses for each product.

While no evaluation will be 100% true to life, this evaluation phase should be as realistic as possible.

In particular, make sure you can check out all the integrations you need and, if possible, make sure the product supports your real processes. Also, investigate how easy the system is to administer, backup and use. Run through common scenarios and make sure the tools perform to expectations. Use real projects, real code and real users for the final evaluation.

Finally, make a decision

Phew! It’s been a lot of work to get this far, but you should now have a pretty comprehensive idea of what the finally shortlisted tools are capable of.

Using all the information gathered so far you can make an excellent business case for the tools that best meet your needs. You can include details of the problems it will solve, the costs of the tools and their support, details about the vendors’ business, and the savings and contributions you anticipate the tool making.

In this part I will look at the most important audience, the people on whom the decision maker relies for advice.

Make no mistake, in most organisations, the decision maker will rely on a group of trusted people to make recommendations about whether a business case should be acted upon or not. So you must answer the questions that these people are likely to ask in your summary (not the executive summary, the summary that you prepare to the business case itself).

These key stakeholders will want the same basic questions answering as the decision maker, the only difference being that they will probably need a little more detail. So, the summary at the front of the business case should be direct, concise and just a little more detailed that the executive summary.

Delivery statement

The very first thing you should present is a clear statement of what the business case will deliver.

Starting off by telling people about how you are going to deliver configuration management, blah, blah, blah, is a great way to lose your audience. Most of the people who read this summary will know precisely nothing about configuration management and those that know something about it probably know vaguely what it is from an ITIL course they once attended. This is not to denigrate these people. On the contrary, why should they know anything about CM, or be interested in it? Remember back in the first post of this series I said marketers talk about selling the sizzle, not the sausage? Well, here’s where we do that.

I am not suggesting that you start setting out your summary like some gaudy marketing flier, nor should you write it as a piece of marketing copy. What I am saying is that you must talk about the benefits that your business case will deliver.

Your summary must then state when these benefits will be delivered. Here you should aim to show some immediate benefits to be delivered close to the start of the project, then some medium-term benefits, and then the final delivered benefits. Don’t just say, “this project will be delivered in six months”. Say, “in the first month we will deliver…, in month three …” and so on.

Finally, tell them how much all this goodness is going to cost them. When you do this make sure that you also point out how much it will cost them if they don’t do this project.

Next page

Hang on! Hold the phone! Didn’t I just say that the business case was not going to swing the deal? Sure I did, but I also said you have to have a good business case because sooner or later you’re going to need it. Besides, if you can’t make a business case why are you so convinced CM is a good idea?

That’s the point isn’t it? You’re all gung-ho for CM, but why? If you’re waving the flag you better know why and you’d best be able to justify your reasons.

People often get caught up in the emotion of a new idea. “Configuration management is an essential part of effective IT development”, we are told, and so we start telling everyone that CM is a good thing and sooner or later someone says, “why?”

“What?”

“Why is it a good thing?”

“Urm. Because it is part of ITIL. And CoBit. And, erm, CMMI. It MUST be a good thing for all of these to have it.”

“But why are any of those good for out situation?”

“Erm”

Believing that configuration management is a good thing means you know nothing. Knowing why configuration management is a good thing is better (hopefully you already know why CM is a good idea, if not then we need to talk some more – maybe another series of posts ). Knowing why configuration management is a good thing for your situation is better still. Being able to justify all of this with a solid rationale backed by defensible data, is a business case.

I should note at this point that, in the course of preparing a business case you might discover that there is no saving or profit to be made in taking the proposed action. Your research may even demonstrate that the action would result in a loss. Business cases that reveal loses don’t make it past the research stage. That is the value of preparing a proper business case.

What goes into a business case?

Each organisation will have their own layout for a business case, but they all amount to the same thing:

• What do you want to do?
• When do you want to do it?
• How much is it going to cost?
• How risky is it?
• What happens if it fails?
• How are you going to mitigate those risks?
• How much will we save or profit if it works?

That’s basically it.

Who is going to read your masterpiece when it’s done?

Again, the specifics are down to the individual organisation but in general you know that the business case will be reviewed by the stakeholders. Stakeholders will certainly include:

• The decision maker. The person who is ultimately responsible for giving the go ahead.
• A business sponsor who will take responsibility for the action on behalf of the business.
• The implementers – all the people who will actually have to make the proposed action work.
• The users – who will have to live with the consequences of the action.

All, or some, of these roles may be the same person or group.

As with any document it is vital that you understand your audience if you want them to make a decision in your favour. Your business case is the last piece in your sales pitch, it can also be a deal breaker if it is not what your audience wants (but we will have prepared them to want it, so hopefully writing the business case will be the easy part).

If you want to know why I keep emphasising the need to prepare your audience, consider the following example.

In what order should you present the information?

Assuming you are not forced to follow a specific template, here is my suggested order for presenting the information in a business case.

Firstly, consider your audience. The main audience for a business case is the decision maker. The decision maker will be a busy person (usually quite senior and your business case may be one of dozens they are asked to consider), so you must present the best case possible as quickly as possible.

Provide an executive summary. This should be no more than one page, and should preferably consist of only one or two paragraphs. The executive summary must summarise the business case for configuration management in language accessible to the decision maker. If the decision maker is a business person you must be careful to use business language when presenting your business case, and most especially when writing the executive summary. The executive summary should be the last thing that you write and should be put on the front of the business case, before any title page or content. (I often simply attache the executive summary on the front like a covering letter. That way the person reviewing the business case does not even have to open the document.)

The main body of the business case is similarly focussed. Present the essential detail first, then the supporting information, and relegate the raw data to the appendices.

• Summary
  • Benefits
  • Costs, including risks
  • Deliveries
• Supporting case
  • Benefits detail
  • Cost and risk details
  • Plan
• Appendices
  • Raw data

The precise details and content of each major section will vary, but the broad outline should present the information in the order shown above.

The reason you present the business case like this is simple. You are asking someone to make a decision. A busy person wants to know four things; what am I getting, how much is it going to cost, how long will it take, what are the risks? The summary must contain all of this information. The entire summary should be presented in bullet points and take nor more than one, perhaps two, pages (but try hard for one!). If the decision maker feels that this summary does not provide sufficient detail to make the decision then they will look into the main body of the business case.

Experience shows that even fairly major decisions are often taken on the basis of the executive summary or the business case summary. Why? Because the decision maker will have canvassed opinion from trusted subordinates beforehand. They will quickly assess the business case finally presented based on the summary, but the decision will already have been substantially made based on discussions with subordinates.

What about these ‘trusted subordinates’? These are the people at whom the main supporting case is aimed. These are the people who will need to be convinced of the business case and will need to pass on their confidence to the decision maker. It is to these people we shall return in the next article in this series.

Before I get stuck in to the topic I would like to make clear that much of what I am about to discuss applies to making a business case for anything, not just Configuration Management. So, why not have the title “Making a Business Case”? Well, two reasons spring to mind; firstly, this series was originally provoked by a direct question posed on the CM Crossroads forum about convincing an organisation to implement configuration management (here we are in 2009 and people still need convincing! Go figure) and secondly, having a specific business case in mind allows me to illustrate points in a more concrete way.

General Observations on Business Cases

As a freelance consultant I get to see my fair share of business cases. I am still surprised at what passes for a business case in many organisations. They range from barely coherent requests to poorly presented financial statements. The information presented in them is often barely defensible and frequently looks like the work of an accountant who really wants to be the next Stephen King – creative works of fantasy with a serious horror twist. I have seen some excellent business case, but these are a significant minority. (My impression is that perhaps one out of every thirty business cases I have read in the past twenty years where actually useful in making a decision.)

Why are they so bad? Part of the problem is that the organisation’s processes demand that a business case be presented, but the people asked to present them are seldom trained in how to create a business case. Added to which most people reading a business case have become used to seeing whatever passes for a business case in their organisation, they do not demand more. Consequently, over time it becomes accepted that a business case is just a hurdle that must be overcome. Fill in the template with anything that looks like a justification for us to do the project and get on with it.

This leads me to my first point. Most business cases are nothing more than a rationalisation for a decision that has already been made.

In many organisations the process goes something like this.

If the presentations had gone badly Bill’s job would have been more difficult. It Frank and John were not emotionally engaged by the sales presentations then the business case would never have been requested and Bill would need to consider other strategies to get the Whizz-bang application he so desperately wants.

Suppose the events had played differently. What if Bill had put together an unsolicited business case. John might have read it, and he might have been convinced about the economic case (assuming one were even made). He may even have prompted Frank to read the business case. However, in my experience, an unsolicited business case faces several serious problems.

The main problem is that preparing a business case before you have emotional buy-in exposes your project to objections very early on. If, for example, Bill had written into the business case that new licenses for the Whizz-bang application would cost £1M, followed by annual renewal costs of 15%. This is compared to the current license renewal charges for applications A and B at £175K. The renewals are attractive at £25K less per year but to recover the original license cost of £1M will take (assuming no other benefits accrue) 33 years. This makes rejecting the business case very easy. Even if Bill manages to make a better case by included other accrued benefits such as increased productivity, additional functionality and so on, making the financial case for these benefits is more difficult and more easily challenged.

We all want to believe we make good, rational business decisions. Sadly the evidence does not support this position at all. Business decisions are made on ‘gut feel’, intuition, ‘business sense’, experience, and a plethora of other intangible, difficult to account for methods.

In truth rational, hard-nosed business justifications like business cases are used, not to make the decision, but to justify it after the fact. At best, a business case is used to verify that the decision is not going to generate a loss (on paper at least).

In part 2 we start looking at how we approach writing the business case.

More from me at Principia.