One thing that’s always bugged me is unfounded, or at least unsupported, claims made by tool vendors (actually, by anyone, but in this context particularly tool vendors). I’m starting to call them out whenever I can.
Here’s an example of what I mean, posts on forums where claims like the following are made.
Eliminate the number of potential risks threatening database development and deployment by 60% and reduce deployment costs by 95%.[sic]
What does this even mean? Reduce what deployment costs? Against what baseline? Is this an average observable saving, if so, where’s your supporting data?
Which development and deployment risks are you talking about? Which of these risks does your tool mitigate? How much effort is required to mitigate those risks? Is it really worth it?
In this particular case a link was provided to a press release. Still no sign of any substantiating data. What about their web site. Nope. Not a thing.
Or how about this one,
AccuRev will eliminate up to 90% of normal merge activity.
What is ‘normal merge activity’? On what basis does this person (who was a representative of the company) make this claim? 90%? At first blush this is a remarkably specific claim (read it carefully and they’ve included those marketing weasel words ‘up to’), it must be based on some pretty sound data. Or maybe not. Maybe it’s just one of those made up statistics again. Yeah. That’s what it was. I contacted the person involved and they continued to be vague and push out marketing speak, but failed to come up with anything more substantial than ‘Our customers have typically seen between 70-90% reduction in merging activity.’ Another unsubstantiated claim, and even if it’s true is still seems to be based on perception and hearsay rather than hard numbers. (Oh, and 70-90% is a heck of a wide range and boldly claiming 90% in the original statement is misleading even if you accept the more qualified range.)
Ah, you say, but it claims ‘up to 90%’. Well that’s BS too. I can save up to 100% by not doing any parallel development at all. No merging, no merging costs. Alternatively, if your merges are normally done by a bunch of alcoholic monkeys, then I guess saving 90% is possible by using properly trained software engineers and following good branch and merge practices.
And if they save you 0%, well, that’s up to 90% too. 90% is just a made up number plucked out of some marketing guy’s butt.
Now, I understand that marketing people like to make these claims, they look impressive on paper, but they’re absolutely useless to anyone with an IQ greater than their shoe size who is trying to evaluate tools. Frankly, if you’re going to make ridiculous, unsubstantiated claims then I for one don’t want to be doing business with you. I mean, if you can’t be up-front and straighforward at the beginning of our relationship, how trustworthy are you going to be when it matters? (I’m tired of hearing people say things like, ‘but the sales guys said this tool would solve this problem’. Really? The sales guys said that did he? And you didn’t think to verify that before you bought the damned tool?)
The stock reply of course is, ‘but everyone does it and if we don’t make these claims no one will pay any attention.’ Really? You’re falling back on the ‘everyone else does it’ defence? If that’s the best you have then, once again, I’m not sure I want to do business with you. (As my old Mum says, ‘if everyone else stuck their head in an oven, would you?’)
Another justification goes along the lines, ‘but there is no good way to measure X’. Now that may be true, but that doesn’t give you license to just make shit up. If there’s no good way to measure something then claiming to reduce that something by a specific quantity is… well, bullshit. ‘Oh yes, I can reduce your unspecified cost by 53%.’ How the hell can you say that with a straight face?
These sort of unfounded claims really make my blood boil. So I’ve decided that whenever I see unsubstantiated claims I’m going to call them on their BS. It’s simple enough, just reply to their post asking, ‘I am interested in your claim that X will say me Y%. I wonder if you could provide us with the data on which this claim is based, I’m sure the community will be interested to see the methodology and metrics you used to gather this data, as well as the data itself. In an industry where such benchmark studies are rare, or at least seldom published, your data will be most useful.’
I bet not one will reply with anything remotely like an independent data set that substantiates the marketing claim. And if they do, all the better because I for one would love to see this sort of data collected and published.
Remember, marketing statistics and claims are generally worth exactly what you paid for them. Zero. And I have data to support that.
