Gargh!

Okay. Don’t get me wrong, I have nothing against you telling me how your tool addresses the problem at hand. In fact I really want to know how your tool address the specific problem being posed. If it takes too long for a short post, link to a white paper or an article or post on your website/blog. But please, please, please, make sure that whatever you point to really does address the problem being asked about. Don’t just link to some random marketing bullshit about your product. The person asking the question wants to know how to solve a specific problem, at least do them the curtesy of answering with something more enlightening than ‘buy our product’.

What if the question is more general? Well, again, add to the conversation. Answering with ‘take a look at my tool’ just makes you look like one. The only question that should be answered ‘take a look at my tool’ (unless you’re in a dodgy ’70s porno) is ‘can you please list tools in the X category’, and only then if your tool is firmly in the X category (not in the ‘could be used in the X category, if you have a really good imagination or are willing to invest an additional one billion dollars in customising our tool’).

Vendor input on forums is, at its best, very useful and helpful. Sadly it is too often lazy, irritating, and unhelpful.

Here’s an example of what I mean, posts on forums where claims like the following are made.

Eliminate the number of potential risks threatening database development and deployment by 60% and reduce deployment costs by 95%.[sic]

What does this even mean? Reduce what deployment costs? Against what baseline? Is this an average observable saving, if so, where’s your supporting data?

Which development and deployment risks are you talking about? Which of these risks does your tool mitigate? How much effort is required to mitigate those risks? Is it really worth it?

In this particular case a link was provided to a press release. Still no sign of any substantiating data. What about their web site. Nope. Not a thing.

Or how about this one,

AccuRev will eliminate up to 90% of normal merge activity.

What is ‘normal merge activity’? On what basis does this person (who was a representative of the company) make this claim? 90%? At first blush this is a remarkably specific claim (read it carefully and they’ve included those marketing weasel words ‘up to’), it must be based on some pretty sound data. Or maybe not. Maybe it’s just one of those made up statistics again. Yeah. That’s what it was. I contacted the person involved and they continued to be vague and push out marketing speak, but failed to come up with anything more substantial than ‘Our customers have typically seen between 70-90% reduction in merging activity.’ Another unsubstantiated claim, and even if it’s true is still seems to be based on perception and hearsay rather than hard numbers. (Oh, and 70-90% is a heck of a wide range and boldly claiming 90% in the original statement is misleading even if you accept the more qualified range.)

Ah, you say, but it claims ‘up to 90%’. Well that’s BS too. I can save up to 100% by not doing any parallel development at all. No merging, no merging costs. Alternatively, if your merges are normally done by a bunch of alcoholic monkeys, then I guess saving 90% is possible by using properly trained software engineers and following good branch and merge practices.

And if they save you 0%, well, that’s up to 90% too. 90% is just a made up number plucked out of some marketing guy’s butt.

Now, I understand that marketing people like to make these claims, they look impressive on paper, but they’re absolutely useless to anyone with an IQ greater than their shoe size who is trying to evaluate tools. Frankly, if you’re going to make ridiculous, unsubstantiated claims then I for one don’t want to be doing business with you. I mean, if you can’t be up-front and straighforward at the beginning of our relationship, how trustworthy are you going to be when it matters? (I’m tired of hearing people say things like, ‘but the sales guys said this tool would solve this problem’. Really? The sales guys said that did he? And you didn’t think to verify that before you bought the damned tool?)

The stock reply of course is, ‘but everyone does it and if we don’t make these claims no one will pay any attention.’ Really? You’re falling back on the ‘everyone else does it’ defence? If that’s the best you have then, once again, I’m not sure I want to do business with you. (As my old Mum says, ‘if everyone else stuck their head in an oven, would you?’)

Another justification goes along the lines, ‘but there is no good way to measure X’. Now that may be true, but that doesn’t give you license to just make shit up. If there’s no good way to measure something then claiming to reduce that something by a specific quantity is… well, bullshit. ‘Oh yes, I can reduce your unspecified cost by 53%.’ How the hell can you say that with a straight face?

These sort of unfounded claims really make my blood boil. So I’ve decided that whenever I see unsubstantiated claims I’m going to call them on their BS. It’s simple enough, just reply to their post asking, ‘I am interested in your claim that X will say me Y%. I wonder if you could provide us with the data on which this claim is based, I’m sure the community will be interested to see the methodology and metrics you used to gather this data, as well as the data itself. In an industry where such benchmark studies are rare, or at least seldom published, your data will be most useful.’

I bet not one will reply with anything remotely like an independent data set that substantiates the marketing claim. And if they do, all the better because I for one would love to see this sort of data collected and published.

Remember, marketing statistics and claims are generally worth exactly what you paid for them. Zero. And I have data to support that.

You have set up a continuous integration build using CruiseControl. The build loop initiates your build using an ANT script mainBuild.xml. This mainBuild.xml needs to invoke several other build files, but (and here’s the catch) you want the sub-builds to use a different JDK.

Using the ant task will invoke the sub-build using the same context as the calling script. Changing JAVA_HOME will have no effect on the Java used by the sub-build. How then, can you invoke the sub-builds with the correct Java context?

Solutions

One obvious solution is to create wrapper scripts (bat or sh, for example) that set the context for you and then invoke the ‘real’ ANT script. This will work, but lacks elegance and means adding in the wrapper scripts — more maintenance work.

A better, more elegant, solution is to invoke ANT directly from the mainBuild.xml using the java task.

The listing below shows a macro that wraps the direct invocation of ANT to provide simplify calling ANT this way.

Note: This macro requires ant-contrib.

<macrodef name="runAntWithJVM">
    <attribute name="buildFile"/>
    <attribute name="buildDir"/>
    <attribute name="target"/>
    <attribute name="cmdLineArgs" default=""/>
    <attribute name="javaHome" default="${JAVA_HOME}"/>
    <attribute name="timeout" default="4000000"/>
    <attribute name="maxMemory" default="512m"/>
    <sequential>
        <var name="l.javaresult" unset="true"/>
        <var name="l.javaoutput" unset="true"/>
        <var name="l.level" value="info"/>
        <java classname="org.apache.tools.ant.launch.Launcher"
                  jvm="@{javaHome}/bin/java"
                  fork="true"
                  failonerror="false"
                  resultproperty="l.javaresult"
                  outputproperty="l.javaoutput"
                  dir="@{buildDir}"
                  timeout="@{timeout}"
                  maxmemory="@{maxMemory}"
                  taskname="startAnt">
            <classpath>
                <pathelement location="${ant.home}/lib/ant-launcher.jar"/>
            </classpath>
            <arg value="-buildfile"/>
            <arg file="@{buildDir}/@{buildFile}"/>
            <arg value="-DJAVA_HOME=@{javaHome}"/>
            <arg line="@{cmdLineArgs}"/>
            <arg value="@{target}"/>
        </java>
        <if>
            <not>
                 <equals arg1="${l.javaresult}" arg2="0"/>
            </not>
            <then>
                <var name="l.level" value="error"/>
            </then>
        </if>
        <echo level="${l.level}">System build output:
${l.javaoutput}

Returned status:${l.javaresult}</echo>
        <fail>
            <condition>
                <not>
                    <equals arg1="${l.javaresult}" arg2="0"/>
                </not>
            </condition>
        </fail>
    </sequential>
</macrodef>

Line 01 declares the macro and lines 02-08 setup the macro’s attributes.

Lines 09-12 set up some local properties that are used to capture output from the sub-build ANT file so that it can be reported back as part of the mainBuild.xml output.

Lines 13-31 are the real meat of this macro. Lines 13-22 set up the java task, lines 23-25 simply put the ant-launcher.jar onto the classpath (it is the ant-launcher that is used to run ANT in the new context).

Lines 26-30 are the command line arguments to be passed to the new JVM initiated by the java task. Notice that the JAVA_HOME is set explicitly here and will match the JVM invoked for the sub-build.

Returning to lines 13-22.

Line 13 provides the entry point class into ant-launcher.

Line 14 is the one that does the magic, it tells the java task to use a specific JVM to run.

Line 15 ensures that a new JVM is used (fork is not true then the jvm setting is ignored and the sub-build will run in the same JVM as mainBuild.xml).

Line 16 ensures that any failure of the sub-build will not bring down the mainBuild.xml. We need this so that output from the sub-build can be properly reported before mainBuild.xml fails gracefully.

Line 17 collects the return status of the sub-build and is used to decide if the mainBuild.xml should fail after reporting the output of the sub-build.

Line 18 collects all the output from the sub-build into a property.

Line 19 sets a default working directory. In this case it is set to the sub-build directory provided by the caller.

Line 20 sets a timeout (in milliseconds) that we are willing to have the java task run for. This is a backstop that prevents our build hanging indefinitely.

Line 21 sets a maxMemory for the forked JVM (you may not need to set it this high, or you may need to make it higher, all depends on your sub-build requirements).

Line 22 changes the task name reported in logging information (it’s just a bit clearer than seeing [Java] all the way through the log.

The remainder of the macro just marshals the output of the sub-build, reports it and then decides whether to throw a fatal error if the sub-build failed.

Using the macro

Just invoke it with a command like the following.

<runAntWithJVM buildDir="${basedir}/subBuild"
               buildFile="bulid.xml"
               target="all"
               javaHome="C:/Program Files/java/jdk1.6.15_05">

Whenever you are looking at a new methodology or technology. Whenever you are designing a new process or procedure. Ask yourself, ‘what problem does it solve?’. If you can’t come up with a clear well defined answer then the chances are you should not proceed with an implementation.

If you cannot answer the question ‘what problem does it solve?’ then why is the tool, method, process, or whatever needed? If you don’t know why it’s needed, why are you spending time implementing it?

I suspect that one of the reasons for a lack of contributions to the body of knowledge is people feel rather vulnerable offering opinions in writing on a public forum. Committing your untested view to writing can be a daunting prospect, especially when you lack confidence in your ability to clearly express yourself in writing and your audience is well informed and apparently well established on the forum/site. I know I’d benefit greatly from such a forum, even though I contribute to blogs and forum debates now.

While it is satisfying to exchange ideas in forums or through blogs, it is not quite the same as a more real-time interactive session. There is simply no substitute for sitting down in a relaxed informal environment and talking over a subject that interests you. Things like the ALM Expo help but I think they are still too formal and often little more than a series of marketing pitches by vendors, which is not what I’m looking for. I want a sort of ‘watercooler’ or ‘drinks in the bar’ exchange of ideas.

I want to move away from a stultifying formal environment, where we feel our every word is being judged and consequently we feel the need to be ‘professional’ at all times, into a more relaxed environment where we can have a proper conversation.

To this end I have looked at a number of possible mechanisms for sharing and interacting in real-time.

One obvious method is the webinar. Most webinars I have seen get the format horribly wrong (or rather, they corrupt the format into something so far removed from a webinar that it’s unrecognisable). A webinar should be like a face-to-face seminar (a host makes a short presentation on the topic at hand to stimulate participation, followed by an extended period of dialogue between participants). Most ‘webinars’ that I have attended are little more than a marketing pitch in which the host presents their product (sometimes thinly veiled behind some more general question) followed by one or two selected questions. This format is more like a lecture but would be more accurately described as ‘ad-inars’. Completely useless for what I want, which is a dialogue, an exchange of ideas—in other words, what I would expect from a seminar/webinar.

Then there are broadcast forums such as BlogTV. Again, while this is primarily a broadcast medium, the interactive chat available to all viewers does provide the audience with a means to direct the ‘show’ by asking questions and challenging the presenter. So, perhaps a little more of the dialogue I am looking for; a chance for people to ask questions and challenge the host’s views. BlogTV also has the opportunity to co-host, allowing for the interesting possibility of debates, interviews and the like. BlogTV’s non-commercial nature also, in my opinion, makes it more relaxed and less formal than some other formats.

Another interesting possibility is the chat room style interaction offered by services like Stickam or YouCams. These allow several people to interact via audio/video links while others contribute in chat format. People can “step up to the mic” when a slot is released, making for a fairly dynamic interaction.

These five modes (blogs, forums, webinars, broadcast, and video chat-room) each has merits and shortcomings. I already participate in most of these formats but only the first two in a professional capacity. I find myself dissatisfied with the interactions provided by the forum and blog formats (it’s like having a conversation in slow motion and it takes too long to write the post and then spend time clarifying and resolving misunderstandings that could be cleared up in seconds face to face). So, what about the other formats?

Well, the most obvious problem with each of the other three is precisely their real-time nature. At what time should I offer to take part in them? I have good conversations with people in India, Europe, and the US and these cover a wide range of time zones. To cover all of them would require several sessions. Then there is the matter of when people can take part. Some of you can access audio or video at work, others cannot. Some are willing to take part in these sorts of conversations in your spare time, others are not. To reach as wide an audience as possible, to discuss the topic with as many people as possible and benefit from different viewpoints means running multiple sessions but obviously I want to make each session as relevant and accessible as I can.

How many people want to spend their spare time talking about their profession? When I first thought about this my reaction was ‘not many’, but then I thought, ‘but you want to, there must be others like you’. But maybe I’m just strange? We shall see.

All of this musing lead me to the conclusion that the only way to proceed was to ‘give it a go’. So, over the coming months I will be setting up some open webinars, doing some BlogTV, and coordinating some video chat-room sessions. Some will start with a specific topic, others will be more open-ended. I will try to run sessions at different times in the hope you can join in at least one of them.

If you are interesting in making your own presentation at a webinar (I want people willing to present a 10-15 minute introduction, a well framed question would make a good introduction, to stimulate dialog—no vendor style ‘ad-inars’), or you want to co-host a blogTV session, then contact me and we’ll sort something out. If you want to participate but cannot find a suitably timed session then contact me and let me know your time zone and when would be a better time for you (no promises, I do have to work and sleep too ).

Oh, and if you have any suggestions for topics for these sessions, let me know and I’ll see what I can do.

I will blog, tweet, post to the CM Forum, and announce on the ITSLM Body of Knowledge web site all of the sessions.

Whether I continue with these will depend upon whether people are interested in participating (there’s no point in them if people don’t take part).

I hope to speak with many of you in the near future.

One often sees, in ANT scripts, something like this at the head of the build.xml file.

<project name="myproj">
    <property name="buildroot.dir" location="${basedir}/.."/>
    <property name="source.dir" location="${buildroot.dir}/src"/>
    <property name="bin.dir" location="${buildroot.dir}/bin"/>

    <property name="bld.package" value="true">

So far, so good. The author of this ANT script has thought to define properties that control the script’s behaviour at the start of the script. This means that, should circumstances change, it is relatively easy to change the script’s behaviour. So, should it be decided that the bin directory should be moved to another location, we simply change the definition of the bin.dir property and no other changes are necessary in the script.

Another, not so obvious, advantage of this approach is that these properties can be easily changed by defining the properties outside the script (passed on the command line, e.g. ant -Dbin.dir=special/bin or by some client script, e.g. a call from CruiseControl), either way, the property defined will not be overridden by the script because <property> never redefines a property once it is set. Neat.

The problem with putting properties in the build.xml file itself makes them more difficult to access from other ANT scripts.

Suppose I have another ANT script that I want to interact with the same build system. I want to ensure that the script can pick up the configuration information (things like the directories the build system is using or settings that dictate build behaviour like the bld.package property). If these are defined in the build.xml then accessing them is tricky. If, however, they are defined in their own file I can simply load that ‘configuration’ file separate to the build.xml but be confident that the settings are the same as were used by the build itself.

There are two approaches to isolating configuration from the scripts that use them; use another ANT file and <include> or <import> it, or I can use a property file, e.g. build.properies and <property file="build.properties"> or <loadproperties srcfile="build.properties"> it into a script. The approach you take will depend upon the sort of information you want in your configuration, where you are likely to use it, and how much flexibility you need.

Personally I prefer <loadproperties srcfile="build.properties"> for most purposes and I use <include> as a wrapper when I need special configuration that is not accessible through the more direct build.properies format.

So, I might have the following in my main build.xml.

<project name="myproj">
    <include file="config.xml"/>

And then, in the config.xml I might have the following.

<project name="myproj.config">
    <loadproperties srcfile="build.properties"/>

    <defaultexcludes add="**/*.bak">
</project>

The defaultexcludes would me more difficult to set using a properties file, so I use the config.xml here. I am using <loadproperties> here, but could have used <properties file="">. <loadproperties> provides some extra flexibility because it supports filterchains, but that’s a topic for another post. I just feel that <loadproperties> describes better what the script is doing, but as I say, in this case, it’s a matter of personal preference.

And finally, in the build.properties I might have the following.

buildroot.dir=${basedir}/..
source.dir=${buildroot.dir}/src
bin.dir=${buildroot.dir}/bin

bld.package=true

Now the build.properties are available in a simple form to be loaded into any script that needs this project’s main configuration. The config.xml provides additional configuration that I may want to load but is not available in the property file format, and is often not relevant to basic configuration information such as locations. I can also use the config.xml to perform more advanced scripting of configuration if needed.

I find that this approach of breaking out configuration into separate files according to both its function and how I think it will be accessed has saved me a lot of work over the years. It also disciplines me into thinking about how my builds should be configured and encourages me to keep all the configuration information in one place (if I find myself tempted to use a <property> to define a location or simple string in my main build.xml I ask myself, would this be better in the build.properties? It generally is.)

Keeping configuration separate also makes the main build.xml much easier to reuse. Just copy the build system into the new project, change the build.properties and, assuming you’ve written your build.xml well, everything should work without further change.

The obvious, and probably correct, answer is ‘probably not essential‘ but if you work in software configuration management it is well worth considering.

Perl is fairly ubiquitous, there are few platforms that do not have a Perl implementation available and it is a standard part of many Unix distributions. Perl code is (with a little caution needed to avoid explicit platform dependencies) portable too, the Tk packages even provide a reasonably portable GUI system should you need it. Perl is a fairly simple language to learn and has a legacy of library code freely available from CPAN that offers a multitude of useful utilities to the SCM practitioner.

While a similar case can be made for Java or Python, I contend that, excellent skills to have though they may be, both languages place more demands on the casual programmer, requiring more time to learn and more discipline to write code. Perl benefits from a much more relaxed approach to scripting. Being a semi-compiled scripting language it is easy to write quick scripts and easy to develop these in a more-or-less ad hoc way.

It is undoubtedly true that Perl’s relaxed approach can bite the unwary, and it is very easy to end up with unmaintainable systems if you let them get out of control. But with a little discipline you can, so to speak, have your cake and eat it — benefiting from simple, rapid scripting facilities, leveraging libraries and ease of developing small scripts into greater things.

If you get serious about your Perl then you will need an IDE. I use ActiveState’s Kermodo and it serves well with its integrated code assistance, online help, coverage analysis, and debugger being major benefits. Activstate’s Perl developer package also comes with a nice packaging tool that wraps a Perl interpreter with your code to provide a single executable package that does not require Perl to be installed on the user’s environment.

If your tastes run more to the Eclipse IDE then there’s an extension for Perl, which also supports the Perl debugger, called EPIC. Having used both, I favour the ActiveState environment for all the additional features their package supports, but that’s a personal choice.

Over the past fifteen years I have found my Perl skills have enabled me to quickly develop and reuse Perl scripts in many different environments, on many different projects, for many different clients. So, for me, Perl has been one of the skills I have been very glad to have in my toolbox.

Sometimes the questioner will provide some clues about their specific circumstances, but inevitably the answers will reiterate, “it depends”.

The correct approach to this problem is not to solicit ad hoc advice from internet fora, but to consider your own situation. The process is simple enough but requires that you put some thought and effort into finding the right tool. It boils down to the following steps.

1. Draw up a set of criteria to be met by the tools you intend to evaluate.
2. Draw up an evaluation matrix.
3. Create a shortlist of possible tools.
4. Evaluate the shortlisted tools against your criteria.
5. Get vendor demonstrations.
6. Get evaluation license installations.
7. Finally select, in as objective a manner as you can, the best solution for your predefined criteria.

An alternative strategy is to employ an expert in the ALM tools field, but they will follow this process anyway. Any other approach is utterly pointless. Let’s take a more detailed look at each step.

Generate a Requirements Document

You must have a list of requirements the tool is to satisfy (otherwise you would not know you wanted a tool). Formalise these requirements into a document. Be specific.

If the tool is to support version control then consider whether you need it to support branching, baselining, labelling, parallel development and so on. Do you need it to support locking to prevent more than one user changing a file revision at a time? If so, must this locking be provided at a revision, branch or file level?

Are there specific integrations you require? Must the tool integrate with developer IDEs? Do you have existing build systems that the system must integrate with, or do you perhaps need a system with an integrated build system? Do you have special tool integration requirements?

Does the tool need to track changes (workflow control of change requests)? Or perhaps you have an existing change request system that needs to work with the new tool?

Do you need to support specific platforms? It’s no use looking at tools that run on Windows platforms if your development uses Unix. Similarly you may (as is common nowadays) have many platforms to support. If so, make sure you specify those that must be supported (and perhaps some that would be ‘nice to have’ — there’s no use turning down a perfect fit for 99% of your environments just because one developer uses an odd platform).

Mario Moreira’s excellent  Software Configuration Management Implementation Roadmap (Amazon uk,us) is an excellent place to start in establishing your requirements.

Don’t forget the future! You may have a specific set of requirements now, but talk to your strategy team and find out whether they have plans to move to other platforms in the near to mid-term future. It may take many weeks to find and procure the tools, and once you have them integrated into your development process it can be tricky to change them, so make sure you are buying a tool set that will serve your needs for the next five years or more. (Sure, things may change in that time, but it’s stupid in the extreme to ignore what you know now. So plan ahead!)

The future is, they say, a foreign land and no amount of planning can anticipate every future requirement. Planning ahead is essential, but not guaranteed foolproof. Plan to transition to new tools in the future by including requirements for extracting data from the tool set you are investing in. This is a common mistake, tying yourself in to one system and finding it impractical to move to a new system later. Again, think ahead.

Formalise all your requirements into a document and review it with the stakeholders. Get sign-off from all stakeholders so that you know exactly what you need to find.

Draw up an evaluation matrix

Once you have the core requirements agreed on, draw up a simple evaluation matrix.

An evaluation matrix is a simple grid down one side of which are all your requirements. Next to each requirement put a weighting (a simple multiplier) indicating how important that feature is to your organisation. For example, you could use a scale of 1 through 5 (1 being optional and non-essential, and 5 being absolutely mandatory). Going across the top of the grid will be a series of columns where you specify the tools you intend to evaluate. This matrix can then be completed for each tool as you evaluate it when creating the shortlist (next step).

For each tool, you will enter a value indicating how satisfactorily it meets each requirement (0 for not at all, through to 5 for completely, for example). By multiplying the original factor with the satisfaction measure, you get a more-or-less objective value for how much each tool satisfies each requirement, and thus the degree to which each tool in its entirety meets the needs of your organisation or project. So, an optional requirement will always contribute less while a mandatory requirement will contribute more; and a tool that does not satisfy the criteria contributes nothing while a tool that satisfies the requirement completely contributes more.

There are many ways to adjust the way you calculate the value of each tool’s satisfaction of requirements (by adjusting the weighting factors or the way you calculate the final score), but whatever you decide to do, decide before you start evaluating tools and do not be tempted to fudge the results. We are aiming for as objective an assessment as we can get and it is tempting to see an appealing tool and try to adjust the weightings so that tool scores well. Doing this defeats the purpose of the evaluation; you may as well just ‘buy the pretty one’.

Draw up a shortlist

Okay, now you are ready to start researching tools. First, scan the internet for likely candidates. By all means visit forums and ask questions, but keep them both focussed and general. Ask people to suggest tools in general categories, for example: “I am looking for a change management tool to run on a Unix server that can be accessed from both Windows and Unix clients” or “I am looking for a version control tool that supports branching and merging and allows locking at the revision level”. Once you get these suggestions, go look at the vendor’s website and see how the tool matches your general requirements.

During this research phase you don’t need to be too specific, you’re just trying to get a shortlist of tools to add to your evaluation matrix and examine more closely.

Once you have your shortlist, you’re ready to do some more detailed assessments.

Evaluate shortlisted tools

A quick way to start your detailed evaluation is to send your requirements document (not the matrix, it won’t have any detailed description of your requirements) to the vendors of each tool. Make sure you contact the vendor and that you are dealing with a specific individual (you want to have someone accountable for making sure you get responses).

Only once you have responses should you move on to real evaluations. Oh, and any vendor that fails to respond should be dropped, after all if they can’t answer (or if they refuse to answer) questions then you have good reason to suspect their after-sales support may not be too hot either.

Okay, so you have your responses. Now add these to your evaluation matrix and add in your evaluations for how satisfactorily each tool meets your requirements. Contact the vendor (the person you dealt with when submitting your requirements) and get clarification as required.

Eliminate tools that score too badly from further investigation. (Of course, ‘too badly’ will depend on what you are looking for.)

The final part of your initial investigation is to check the vendor company. Are they well established? Financially stable? What do forums say about their support facilities? How often do they release updates? And, of course, how much will the product cost (based on list price at this point, your procurement department will deal with negotiating the final price of licenses)? Add these details to your evaluation matrix.

Now, and only now, should you approach vendors for demonstrations.

Get vendor demonstrators

Why leave it so late to see the tool in action? Two reasons; firstly, until you have requirements, and a statement from the vendor on how ther tool satisfies the requirements, you are not in a position to evaluate any demonstration. Secondly, it is very easy to be distracted by a generic shiny demonstration (I know only too well that generic vendor demonstrations have the ‘wow’ appeal, especially to non-technical managers, and it’s easy to be seduced). Generic pre-sales demonstrations are designed to look good. The vendor is showing off their tool. Of course it looks good. Of course it runs smoothly (although I’ve seen some howlers in my time). This is fine for showrooms, but terrible for purchasing decisions. Remember, we are looking for the right tool for your organisation, not the tool the vendor wants you to buy.

Once you have good evaluation criteria you can ask vendors to come and demonstrate how their product satisfies your requirements. Make sure you are clear when requesting the demonstration that you expect to see evidence of how their tool meets the requirements document that they have already seen and provided feedback upon. Do not accept a generic demonstration at this point. If they cannot, or are unwilling to, provide a suitable demonstration, strike them from the list — you’ve put in a lot of effort and are likely to be giving them a substantial amount of money in the near future, so they should be willing to show you what you want.

During the demonstration, make sure you see their tool perform to the requirements that they have claimed it fulfils. Get clarification on how marginally met requirements may be worked around (those that the vendor identified as only partially satisfied) and be sure that you see an unambiguous demonstration of all requirements that the vendor declared they satisfied completely.

During demonstrations, be fair. It is unreasonable to expect the vendor to show you everything you need. There will also be requirements that they state they only partially satisfy; give them a break and don’t harangue them for this. Accept the limitation. Discuss options. Make a note and move on.

Once all the vendors have demonstrated their products, re-evaluate them on your evaluation matrix. Add notes on where the demonstration failed to meet expectations, and where the demonstration exceeded expectations.

Eliminate any that fail to meet expectations satisfactorily.

Get evaluation license installations

You should now have a very short list of potential products (two or three if you are lucky). Obtain evaluation licenses for each product.

While no evaluation will be 100% true to life, this evaluation phase should be as realistic as possible.

In particular, make sure you can check out all the integrations you need and, if possible, make sure the product supports your real processes. Also, investigate how easy the system is to administer, backup and use. Run through common scenarios and make sure the tools perform to expectations. Use real projects, real code and real users for the final evaluation.

Finally, make a decision

Phew! It’s been a lot of work to get this far, but you should now have a pretty comprehensive idea of what the finally shortlisted tools are capable of.

Using all the information gathered so far you can make an excellent business case for the tools that best meet your needs. You can include details of the problems it will solve, the costs of the tools and their support, details about the vendors’ business, and the savings and contributions you anticipate the tool making.

Many ticketing systems, supporting processes such as incident management, are prone to ticket ping-pong where tickets are bounced from queue to queue as people keep forwarding the problem to get it off their queue. This is simple to control once you see the problem.

Processes that suffer with this problem all have one thing in common; they all allow one user to simply pass a ticket from their queue to another unchallenged. For example, an incident ticketing system shows up a ticket on Person A’s queue. They cannot deal with it (or do not want to deal with it), so they look for someone they believe should deal with it and assign it to their queue (Person B). Person B looks at the ticket, decides they should not be dealing with it and looks around for someone else to deal with it (Person C). And so it goes, the ticket to bounced around from queue to queue never being handled but always being sent on, possibly even returning to Person A who forwards it on again and the cycle continues.

The problem is that the ticket owner in such systems is defined as the person who has the ticket on their queue and anyone can pass a ticket to another queue. Since no one wants to be lumbered with the ticket it just keeps being passed on. It becomes an SEP (someone else’s problem), doomed to forever wander the queues, never being resolved.

Fixing this process is actually very simple. When a ticket is initially raised by a user onto someone’s queue (let’s say Person A) the queue owner becomes responsible for progressing the ticket. If the Person A believes the ticket belongs to someone else (say, Person B), they propose it into Person B’s queue. But, and this is the important change to the process, ownership of the ticket remains with Person A. Person A remains responsible for ensuring that the ticket is progressed (not simply passed from queue to queue) until Person B accepts the ticket into their queue. This puts pressure on the Person A (as the ticket owner) to actually manage the process, whereas the original process did not pressure anyone because they could simply pass the ticket on to someone else.

It may seem unfair on Person A. It could be that they really cannot deal with the ticket. It could be that Person B really should take ownership of the ticket but they may be trying to avoid doing so by not accepting it into their queue. This is what escalation is for. If Person A has a legitimate cause they can escalate the problem and effectively force Person B to acknowledge their responsibility. In practice this is seldom an issue as Person B, if they recognise their responsibility will simply accept the ticket into their queue and progress it to completion.

I Person B genuinely cannot, or should not, deal with the ticket they simply inform Person A of their reason for not accepting it into their queue. Person A must now seek the correct person to resolve the ticket. Because Person A is responsible for the ticket until someone else takes charge of it they have a vested interest in finding the correct person to deal with it. If they propose the ticket to someone’s queue and do not get a quick response they have a vested interest in following up to find out why the ticket has not been accepted. Person A is motivated to find the correct ticket owner since no one but the correct ticket owner will take the responsibility for progressing the ticket away from Person A and until some takes it from Person A it remains theirs to progress.

Ensuring that people cannot simply pass tickets around the system ensures that they are progressed because it puts pressure on owners to either resolve the ticket or to find the person who really should resolve the ticket.