Bob Aiello lit the blue touch paper (again) on the debate about ‘what is configuration management?’ and, once again, he seems to be trying to redefine configuration management to fit the role of Configuration Manager identified (incorrectly) in many organisations. This is absolutely the worst way to define configuration management.

Let’s now look at Bob’s list of things he want to dump on CM’s lawn.

I often describe that there is a big CM and a little CM.

Big CM is the field of Configuration Management which I suggest includes:
* Source Code Management
* Build Engineering
* Environment Configuration
* Change Control (seven different types)
* Release Management (coordination and engineering)
* Deployment

while little CM refers to Configuration Control (of interfaces).

From Bob’s original post to this thread

This is simplifying CM?  In this article I hope to show why this redefinition is unnecessary, flawed, and ultimately damaging to CM.

If we were to accept that a discipline is correctly defined by looking at the roles identified in organisations then we would never get anywhere (much like this debate). I for example often have the role named ‘Configuration Manager’ and yes  I also do builds, releases, etc as identified in Bob’s list. But then I also fetch coffee for the team and I do the odd bit of development or testing, I write technical documentation, perform product evaluations, project management, risk assessment, and I assist in problem solving for production systems—and yes, sometimes these things (with the exception of fetching the coffee) are part of my official role (although given the current economic climate, who knows). Does that mean these too are defining features of configuration management? No. This is a silly way to define a discipline. Even if you observe that ‘many’ organisations lump the things together it still would not justify redefining configuration management because there’s absolutely nothing to gain, and much to lose, from doing so.

Configuration management (the discipline) has a perfectly sound definition (all together now); identification, change control, status accounting, and auditing. To change this definition is to define something other than configuration management and I have no objection to defining an umbrella term to cover all the disciplines identified by Bob; in fact I do, I call them Development Support Services.

Yes, we could do more to clearly communicate what CM is. Although, I cannot recall a time when I was ever confused by what the the four core activities were, the difficulty always stemmed from more subtle issues like ‘how do I select good configuration items’ and Bob’s redefinition does precisely zero to address these problems. Adding more complexity by mixing in a load more disciplines is hardly likely to aid clarity.

If the idea is that by lumping all of these disciplines under the CM heading we can make CM appear more useful or make it an easier sell to management, then I think that’s horribly misguided too.

You: We want to introduce CM.
Manager: What’s that when it’s at home?
You: Oooh. Loads of good stuff. Build, release, change….
Manager: Hang on. We do most of that stuff already.
You: Ah. But we’ll pull it all together under one heading. Make it more coherent. Make it better.
Manager: Create a bottleneck you mean.
You: No. No. No. We’d be specialists.
Manager: So CM is just all this stuff under one team?
You: Oh no. We also do identification, change control, status…. *noticing manager slip into comma* Oh, wait. Remember the ITIL course you went on?
Manager: Oh yeah. That was a great couple of week. Great evenings…. Oh. I mean, yes, very valuable.
You: And you remember them saying you needed a good CMDB?
Manager: Yeees?
You: Well that’s what we’ll do.
Manager: As well as all this other stuff?
You: Sure.
Manager: And what about all the people already doing the other stuff?
You: They’ll be part of my team.
Manager: Hang on. Your team?
You: Of course. They’ll be under the CM team.
…. and so on…

Obviously I’m being slightly facetious, but hopefully you see my point; why complicate things? The manager is now more confused about what CM is than he was at the start. Sure, you’ve made a pitch to create a service team supporting development, but you’re not closer to furthering the CM cause than you would have been had you just reminded the manager about his ITIL course in the first place. You’ve made no real progress in explaining CM, you’ve just hidden it amongst a load of other stuff, possibly to the point of obscuring it completely.

‘But we should move with the times’ is a common enough response. I agree. Let’s draw a parallel to show how it’s unnecessary to abandon the existing definition of CM but still move with the times.

Science is decomposed into disciplines. Physicist will often joke (and in some cases only half-jokingly) that all other sciences are ultimately reducible to physics and therefore physics is the only science we need. This seems to be what those who want to lump all these other disciplines into configuration management are set on doing, lumping all the other disciplines into one. And, just as with the science example, it’s a bad idea.

Continuing the parallel between CM and science. Just because scientists resist the temptation to mangle all their disciplines together hardly means they’re not moving with the times. Biological science is unrecognisable from fifty years ago, but it’s still Biology. Similarly, configuration management’s scope has increased as we have more powerful tools we can realistically monitor and control to a finer and finer degree of detail. Where in the past a configuration item’s scope would be limited by the capacity of a paper system to realistically track it, we can now declare almost every file in a system to be a CI if we so choose.

What Bob (and I’ll use Bob as a proxy for all those who support the view that CM should absorb all these other disciplines) seems to be proposing is something like the following (I’ve reduced the list to just build and release management to keep the diagrams simple, my point is still made I think).

Or, heaven forbid, the following.

In the first case build and release management become intrinsic parts (aggregate into) configuration management. In the second (shudder) they become inner classes, only accessible through the configuration management wrapper.

Both of these system architectures are… well. They’re terrible design. If a software engineer came to me with this design I’d fire his ass. (If people are sufficiently interested in me expanding this claim, I will do so in another post.)

One observation, made by Joe Townsend, was that the four activities that constitute configuration management are still to be performed within many of the sub-discplines identified by Bob. If this is so, then why not keep the CM discipline separate and have these disciplines use it rather than mangling them all together? What possible benefit is accrued by redefining CM in this way? ‘Oh well, CM is still a part of some of the disciplines we want to put under the CM banner. Erm. No. Wait. The ‘four activities formerly known as CM’ as now part of some, but not all, of the disciplines that now make up CM’. It just makes no sense. It’s adding insult to injury. It’s muddying the waters. You get the idea; you’re doing nothing at all to help clarify CM and your piling in a load of additional stuff that does not belong in there.

Let me be clear. If you want to be called ‘Configuration Manager’ and cover all the disciplines Bob identifies, good luck to ya. You won’t be the first, or the last, to do that. Heck, you can take the title ‘Lord of all he surveys’ if you like. Makes no odds to me. My gripe is about redefining the discipline of Configuration Management in these terms.

The following diagram shows how build and release management simply use configuration management. A much more elegant structure and one that benefits from modularity, making it much simpler to explain each discipline—also making it simpler to ‘debug’ the processes involved. (The benefits of such modularity should be obvious, but if people are keen for me to expand I will do so at another time.)

It’s simple. It’s elegant. It’s modular. It’s scalable. And IT’S WHAT WE HAVE NOW! (Well, those who have not polluted the well.)

Another objection is that this combination of disciplines under the CM rubric is only done for Software Configuration Management (SCM). What! Why? Why do people insist on making this asinine distinction? SCM is ‘Configuration Management of Software’, which, according to the correct definition of CM is precisely the same discipline as, for example, ‘Configuration Management of automotive engineering’, ‘Configuration Management of hardware’, or ‘Configuration Management of X’ for whatever ‘X’ you care to put in there.

If you accept the universality of CM principles then the multi-discipline agglomeration proposed by Bob makes even less sense. Most engineering CM systems, for example, have no build discipline within them; the CM discipline is used by fabrication plants to provide reference material (engineering specification, computer controlled tool programs and the like) from which the fabrication plant build the items. The closest these CM teams get to performing a build is assembling a baseline to be passed to the fabrication teams. It is simply coincidental that people labelled ‘configuration manager’ in the software business tend to be qualified to do build management. This is not sufficient justification for putting build management under the CM (or even SCM) banner.

Even if you claim justification for putting these other disciplines under the CM banner by saying something along the lines, ‘oh, well CM does not ‘do’ the builds necessarily, they’re just responsible for ensuring they are done’ I say that road leads to perdition. You could justify the inclusion of almost anything you want under the CM discipline using that argument, and you’d be wrong every time.

In the course of helping a client look for tools to support their development process I recalled that a couple of years ago I looked briefly at JIRA when building a change control system for another client. So, I suggested we look at JIRA as a possible tool for this new system.

Having looked around the JIRA site I noticed that Atlassian had also taken FishEye (a tool I had used before for providing viewing and analysis tools on repositories) into their stable. And they also support a whole suite of integrated tools that can be used individually or together.

I suggested we take a closer look. So, we set about creating a very simple demonstration of the tools.

I have to say, ‘I’m impressed’.

These are great looking, well integrated, easy to use, easy to set up and very competitively priced tools. Based on a couple of months playing around with them, I’d have absolutely no difficulty recommending them to organisations of almost any size—certainly they should be on your list of tools to look at.

Not only are the tools themselves very flexible and easy to configure but the company seems very keen to hear from their customers. They have open beta programmes and actively encourage users to take part in product development (they really do want to know what their customers want). They also provide really low cost licenses; up to 90 days evaluation licenses and then for small teams (or individuals) they offer 10 user licenses for $10. Seriously! This is a brilliant move. It means freelance consultants (such as your’s truly) can actually use their tools and develop for them, without needing to remortgage the house to obtain licenses. (The tools also don’t require a massive hardware commitment—in fact I’m running a system on a Linux virtual machine on my Mac Pro  and it works great for development and creating documentation/training material.)

Oh, and if you’re a non-profit or open source organisation they will let you have unlimited licenses for free! Genius.

Can’t be bothered with all that setup? Atlassian offer a hosted solution for many of their tools too. (Although I can’t comment on these as I have not tried them, but if they’re anything like as good as what I have seen so far, they’re probably worth looking at if you want a hosted solution.)

The price apart, the tools really are very good. And if they don’t do exactly what you want you can always write your own extensions. Atlassian publish comprehensive developer information and the tools all communicate through open web interfaces and all support a plugin architecture (which I understand Atlassian are working hard to make simpler to use).

Another promising sign is that Atlassian ‘eat their own dog food’. They user all their own tools internally and pride themselves on being the guinea pigs for all their product development. Their own website and online documentation is all run through their own products.

Even better! Atlassian let their product speak for itself. No annoying salespeople calling, no pressure to come along and ‘do a demo’. They just provide you with a really simple way to download their product, get an evaluation license, and then let the product do the selling (no kidding I had downloaded, installed and licensed JIRA in about ten minutes—it took about a day to install, license and integrate the entire suite). Brilliant! [Note to all vendors: if you're product is good then let it speak for you. If you're product needs to be explained, then make it better, provide good online documentation, and provide help when it's requested. I understand that some customer like a demo, but many just want to try out your product for themselves. After all, if it can't be setup and used by their own staff, what hope is there for the future? They just end up being highly dependent on your consulting services. Oh. I think I get it now.]

If, as seems likely, the client goes ahead with this solution I’ll be sure to report back on how these products perform under real life conditions, but for now, go take a look.

The problem with many of the other terms is their absolutism; ‘this is the right way’, a sort of ‘here is the solution, not what was the problem’ attitude. I say, ‘no’. I say that the correct approach is always ‘it depends’.

Contingent CM emphasises that there is no one solution, there is no ‘best’ way to do CM and the core CM discipline is independent of the method, process, approach or dogma to which it is being applied. You should learn the underlying principles of CM, then you can apply it in any arena.

Given the modest success of that experimental session, I have planned a series of similar events (details can be found here).

Currently planned are:

• “What is a configuration Item?” – Friday 4th March 2011
• “Version control for beginners” – Friday 18th March 2011
• “Branching. What, when, and how” – Thursday 31st March 2011
• “Parallel development. What, when and how” – Friday 15th April 2011
• “Selling configuration management” – Friday 29th April 2011

All meetings are free to attend and are currently planned for 20:00GMT (There really is no good time that covers all time-zones. If enough people ask I may repeat meetings at other times to cover other timezones—better yet, why not organise one yourself .)

I’ll certainly be chatting for half an hour at least (if no one else steps up to contribute I can certainly talk about any of these topics for at least that long ) but I’ve allowed two hours for each session and am happy for sessions roll on beyond this if there is enough interest.

These are informal discussion groups intended for any level of expertise (although obviously some are aimed primarily at beginners I would welcome input from experienced users too). I will probably make some opening remarks to get things rolling, but I am hoping enough people are interested to make for some real discussion.

You can contribute through the text ‘chat’ on the site or better yet if you have a mic (and video camera if you like) you can step up and present your ideas that way.

Come along if you have questions, are just interested in the topic, or have something to say on the topic being discussed.

Perhaps my greatest personal challenge is a need to respond in full to forum posts. Lengthy responses are often unwelcome (of even impossible on some forums). People so often want a ‘simple’ answer to a quite complex question. Seldom are there such simple answers and even when there is a simple answer I feel the need to discuss options, to point out fundamentals, and to get the questioner to think about the problem rather than simply adopt some rote implementation based on a simplified reply. I feel strongly that people must understand why an answer is appropriate, otherwise nothing has been learned beyond the most superficial ‘getting it done’ answer. If I don’t feel able to spend time giving a full answer that would satisfy myself, I tend not to offer one at all.

Of course, some technical questions and answers demand such direct and simple handling but again, without understanding, the questioner is simply following a script and when subsequent complications occur they have no idea how to deal with them. Inevitably they return to the forum with a stream of follow-up questions (a sure sign that someone has not understood the answer but is simply following it rote).

Then there is the question of how one keeps up with all the sites that offer community facilities. It’s simple enough to monitor most blogs and forums (well, those wise enough to offer RSS feeds) even though it takes time to read and comprehend what the author is trying to communicate. Other facilities, such as groups on CM Crossroads or web sites without RSS feeds (or some sort of notification system like mailing list), require me to visit them regularly to see whether anything interesting has happened (great for the site’s hit count, sucks for taking up my time). Frankly, it’s hard enough to to keep up anyway without needing to keep cycling through checking for updates. The result of this demand is that I just don’t do it; both I and the community lose out.

Then there are more dynamic community resources like IRC or Twitter. I tend to have an IRC and Twitter client open on one screen just to keep an eye on various resources I find useful or I feel I can contribute to. Mercifully, these tend to be, by their very nature, places where people point to more comprehensive resources or ask very straightforward technical questions. I have occasionally been drawn into protracted exchanges, but generally these resources provide a great way of finding new information without committing too much time. (Of course, the follow up of reading the indicated information can be time consuming ).

This all on top of reading books, journals, and keeping up to date on published papers.

And I still see people comment that ‘there is not much information available’. Where are these people looking?

I suspect that one of the reasons for a lack of contributions to the body of knowledge is people feel rather vulnerable offering opinions in writing on a public forum. Committing your untested view to writing can be a daunting prospect, especially when you lack confidence in your ability to clearly express yourself in writing and your audience is well informed and apparently well established on the forum/site. I know I’d benefit greatly from such a forum, even though I contribute to blogs and forum debates now.

While it is satisfying to exchange ideas in forums or through blogs, it is not quite the same as a more real-time interactive session. There is simply no substitute for sitting down in a relaxed informal environment and talking over a subject that interests you. Things like the ALM Expo help but I think they are still too formal and often little more than a series of marketing pitches by vendors, which is not what I’m looking for. I want a sort of ‘watercooler’ or ‘drinks in the bar’ exchange of ideas.

I want to move away from a stultifying formal environment, where we feel our every word is being judged and consequently we feel the need to be ‘professional’ at all times, into a more relaxed environment where we can have a proper conversation.

To this end I have looked at a number of possible mechanisms for sharing and interacting in real-time.

One obvious method is the webinar. Most webinars I have seen get the format horribly wrong (or rather, they corrupt the format into something so far removed from a webinar that it’s unrecognisable). A webinar should be like a face-to-face seminar (a host makes a short presentation on the topic at hand to stimulate participation, followed by an extended period of dialogue between participants). Most ‘webinars’ that I have attended are little more than a marketing pitch in which the host presents their product (sometimes thinly veiled behind some more general question) followed by one or two selected questions. This format is more like a lecture but would be more accurately described as ‘ad-inars’. Completely useless for what I want, which is a dialogue, an exchange of ideas—in other words, what I would expect from a seminar/webinar.

Then there are broadcast forums such as BlogTV. Again, while this is primarily a broadcast medium, the interactive chat available to all viewers does provide the audience with a means to direct the ‘show’ by asking questions and challenging the presenter. So, perhaps a little more of the dialogue I am looking for; a chance for people to ask questions and challenge the host’s views. BlogTV also has the opportunity to co-host, allowing for the interesting possibility of debates, interviews and the like. BlogTV’s non-commercial nature also, in my opinion, makes it more relaxed and less formal than some other formats.

Another interesting possibility is the chat room style interaction offered by services like Stickam or YouCams. These allow several people to interact via audio/video links while others contribute in chat format. People can “step up to the mic” when a slot is released, making for a fairly dynamic interaction.

These five modes (blogs, forums, webinars, broadcast, and video chat-room) each has merits and shortcomings. I already participate in most of these formats but only the first two in a professional capacity. I find myself dissatisfied with the interactions provided by the forum and blog formats (it’s like having a conversation in slow motion and it takes too long to write the post and then spend time clarifying and resolving misunderstandings that could be cleared up in seconds face to face). So, what about the other formats?

Well, the most obvious problem with each of the other three is precisely their real-time nature. At what time should I offer to take part in them? I have good conversations with people in India, Europe, and the US and these cover a wide range of time zones. To cover all of them would require several sessions. Then there is the matter of when people can take part. Some of you can access audio or video at work, others cannot. Some are willing to take part in these sorts of conversations in your spare time, others are not. To reach as wide an audience as possible, to discuss the topic with as many people as possible and benefit from different viewpoints means running multiple sessions but obviously I want to make each session as relevant and accessible as I can.

How many people want to spend their spare time talking about their profession? When I first thought about this my reaction was ‘not many’, but then I thought, ‘but you want to, there must be others like you’. But maybe I’m just strange? We shall see.

All of this musing lead me to the conclusion that the only way to proceed was to ‘give it a go’. So, over the coming months I will be setting up some open webinars, doing some BlogTV, and coordinating some video chat-room sessions. Some will start with a specific topic, others will be more open-ended. I will try to run sessions at different times in the hope you can join in at least one of them.

If you are interesting in making your own presentation at a webinar (I want people willing to present a 10-15 minute introduction, a well framed question would make a good introduction, to stimulate dialog—no vendor style ‘ad-inars’), or you want to co-host a blogTV session, then contact me and we’ll sort something out. If you want to participate but cannot find a suitably timed session then contact me and let me know your time zone and when would be a better time for you (no promises, I do have to work and sleep too ).

Oh, and if you have any suggestions for topics for these sessions, let me know and I’ll see what I can do.

I will blog, tweet, post to the CM Forum, and announce on the ITSLM Body of Knowledge web site all of the sessions.

Whether I continue with these will depend upon whether people are interested in participating (there’s no point in them if people don’t take part).

I hope to speak with many of you in the near future.

One often sees, in ANT scripts, something like this at the head of the build.xml file.

<project name="myproj">
    <property name="buildroot.dir" location="${basedir}/.."/>
    <property name="source.dir" location="${buildroot.dir}/src"/>
    <property name="bin.dir" location="${buildroot.dir}/bin"/>

    <property name="bld.package" value="true">

So far, so good. The author of this ANT script has thought to define properties that control the script’s behaviour at the start of the script. This means that, should circumstances change, it is relatively easy to change the script’s behaviour. So, should it be decided that the bin directory should be moved to another location, we simply change the definition of the bin.dir property and no other changes are necessary in the script.

Another, not so obvious, advantage of this approach is that these properties can be easily changed by defining the properties outside the script (passed on the command line, e.g. ant -Dbin.dir=special/bin or by some client script, e.g. a call from CruiseControl), either way, the property defined will not be overridden by the script because <property> never redefines a property once it is set. Neat.

The problem with putting properties in the build.xml file itself makes them more difficult to access from other ANT scripts.

Suppose I have another ANT script that I want to interact with the same build system. I want to ensure that the script can pick up the configuration information (things like the directories the build system is using or settings that dictate build behaviour like the bld.package property). If these are defined in the build.xml then accessing them is tricky. If, however, they are defined in their own file I can simply load that ‘configuration’ file separate to the build.xml but be confident that the settings are the same as were used by the build itself.

There are two approaches to isolating configuration from the scripts that use them; use another ANT file and <include> or <import> it, or I can use a property file, e.g. build.properies and <property file="build.properties"> or <loadproperties srcfile="build.properties"> it into a script. The approach you take will depend upon the sort of information you want in your configuration, where you are likely to use it, and how much flexibility you need.

Personally I prefer <loadproperties srcfile="build.properties"> for most purposes and I use <include> as a wrapper when I need special configuration that is not accessible through the more direct build.properies format.

So, I might have the following in my main build.xml.

<project name="myproj">
    <include file="config.xml"/>

And then, in the config.xml I might have the following.

<project name="myproj.config">
    <loadproperties srcfile="build.properties"/>

    <defaultexcludes add="**/*.bak">
</project>

The defaultexcludes would me more difficult to set using a properties file, so I use the config.xml here. I am using <loadproperties> here, but could have used <properties file="">. <loadproperties> provides some extra flexibility because it supports filterchains, but that’s a topic for another post. I just feel that <loadproperties> describes better what the script is doing, but as I say, in this case, it’s a matter of personal preference.

And finally, in the build.properties I might have the following.

buildroot.dir=${basedir}/..
source.dir=${buildroot.dir}/src
bin.dir=${buildroot.dir}/bin

bld.package=true

Now the build.properties are available in a simple form to be loaded into any script that needs this project’s main configuration. The config.xml provides additional configuration that I may want to load but is not available in the property file format, and is often not relevant to basic configuration information such as locations. I can also use the config.xml to perform more advanced scripting of configuration if needed.

I find that this approach of breaking out configuration into separate files according to both its function and how I think it will be accessed has saved me a lot of work over the years. It also disciplines me into thinking about how my builds should be configured and encourages me to keep all the configuration information in one place (if I find myself tempted to use a <property> to define a location or simple string in my main build.xml I ask myself, would this be better in the build.properties? It generally is.)

Keeping configuration separate also makes the main build.xml much easier to reuse. Just copy the build system into the new project, change the build.properties and, assuming you’ve written your build.xml well, everything should work without further change.

Let’s put this into concrete terms.

Suppose the original change request is something like, “correct the spelling mistakes in the poem” and the poem is held in two files.

file1.txt contains the following text.

Mary hard a litl lamb,
It's fles was wite as snow,

And file2.txt contains the following text.

And everywere that Mary went,
That lam was sure to go.

We correct these two file to the following using a change record CR1.

Mary hard a little lamb.
It's fleece was white as snow

And everywhere that Mary went,
That lamb was sure to go.

Here then is my first build (build1).

Initial build: conceptual and literal change equivalent

So, not bad, but the still have “hard” instead of “had” in the first line of file1.txt. This does not change the integrity of build1, but does make another build (build2) necessary to include a correction to the erroneous work submitted into build1.

There are different ways we might choose to track the correction of the defect arising from build1.

Each approach has its own merits but, from the perspective of build integrity, they share an the important feature; when dealing with the conceptual change “correct the spelling mistakes in the poem”, they are semantically equivalent. For build2 to have integrity it must include both the revisions associated with the original change and all new revisions associated with the correction to the change, regardless of how that correction is controlled.

We correct the defect in build1 with an additional change to file1.txt to produce the following.

Mary had a little lamb,
It's fleece was white as snow,

The following illustration shows build2 (in this case I have re-used CR1 to correct the defect). I should be obvious that missing any revision associated with CR1 has the potential to break the build’s integrity (we may have only part of CR1 in the build).

Fix Build: Conceptual change implemented with reworked literal change

Fix build: Conceptual change now implemented with two literal changes

And this is the point. Yes, the literal changes CR1 and D1 are, in one sense, changes. In practical terms though each on its own fails to achieve the objective of the original change request and consequently they are little more than sets of revisions. The two literal changes controlled by records CR1 and D1, taken together, achieve the original goal of the change request “correct spelling mistakes in the poem”. It is this latter type of change (what I have called conceptual change) that is more interesting to configuration management and build integrity is maintained through these conceptual changes rather than the less significant literal changes.

This distinction between the conceptual change and literal change becomes increasingly important as we consider configuration integrity, a subject I shall discuss in more detail in subsequent posts.

The great advantage of semantic web over efforts such as the now defunct Application Lifecycle Framework (ALF) is that it requires no agreement between vendors (beyond using semantic web technology). The weakness of efforts such as ALF is always that they demand buy-in from the main tool vendors. A substantial number need to agree to develop and support the new standard.

Certainly semantic web is no panacea, but at least if Vendor A chooses one semantic representation of CM information and Vendor B chooses another they can still communicate by creating a correspondence rule set between the two representations (a little like XSLT can transform one XML into another — only a little though, semantic web has much more to offer).

So vendors need to agree to use and provide semantic web representations for CM information? No, not really. Most tools provide APIs that would allow this information to be interpreted from, or added to, any existing tool. Certainly a non-trivial effort, but one that is at least feasible. Better still, if multiple implementations are created for any tool these can again be consolidated using semantic web techniques.

The real power of semantic web technology comes from two sources; the abstraction of information semantics, and the ability to draw inferences from this information. Once you have an ontology, some inference rules and semantic relationships between ontologies, your inference rules will work across ontologies — neat. What does all the gobbledy-gook mean? It means that if Vendor A develops an ontology with a set of inference rules (rules for extracting more information from the underlying information) then Vendor B can map their ontology onto Vendor A’s and use Vendor A’s inference rules too. Actually it’s even better. User X can extend the rules and have them apply to Vendor A and/or Vendor B’s information sets equally, even if the original inference rules were designed for only Vendor A.

Brilliant. Problem solved then? Sadly, no. Although this all offers promise of a way forward there remains a lot of work to establish these semantic descriptions and, as many have discovered before, agreeing on the precise meaning of each semantic element is nontrivial in its own right. Not that this should stop us attempting the task.

Remember a few posts back I mentioned working on a CMS reporting tool? Well, I’ve been working with a client on a tool for managing parallel development, which is closely related to the type of reporting I had in mind for the CMS reporting tool. The client’s focus is on Dimensions, but in the interests of creating a flexible system I’ve been designing with other underlying in mind (although, as is often the way, timescales and pragmatism mean the implementation is sometimes more tool specific than the original design). The result of this has been some extended mental gymnastics that have resulting in some useful and hopefully interesting thoughts that I would like to share with you now.

The parallel development tool needs to both report, analyse and, ultimately, update the host SCM database. There are a number of special considerations that I won’t go into now, but these created much candle burning and banging of my head on desks, walls and other solid objects as I struggled to resolve some apparently simple problems that turned out to be rather tricky. (The sad thing is, as is often the case with these things, once I had an answer it turned out to be deceptively simple.) The upshot of all this cranial damage covers a wide variety of topics, too many for a single post, but we can make a start.

Identification

How do we identify things?

Each tool has its own method of identifying the things it controls. In Dimensions items are ultimately identified by some sort of unique specification; files are identifies by an Object Spec, projects (worksets) are identified by Project Specs and so on. Each of these Specs. provides a unique reference within the context of a specific Dimension base database. Subversion identifies things by their position in it virtual file system; path and revision. Again, Subversion references are qualified by the repository in which they are held. Since Subversion repositories can be referred to by any number of methods there is no one unique ‘name’ for an item in Subversion once you step beyond the bound of the repository (I can, for example, use several host names to refer to the same repository).

With this in mind, how can we uniquely refer to a single controllable entity (I shall use the term ‘Artefact’ to represent a uniquely identifiable thing). For simplicity we’ll focus our attention on a file, after all with software configuration management (SCM) this is the most common sort of Artefact we want to control. As I discussed briefly in a previous post, files exist along two histories; the Revision History (in which new Artefacts are created for each revision of the file), and the Lifecycle History (in which an Artefact may undergo change without changing its revision, without becoming a new Artefact). Dimensions does a nice job of recording these two histories, but our challenge here is to differentiate identities.

Do we assign the identity to the Artefact’s revisions, or do we assign the identity to each Lifecycle historic state?

To illustrate, consider file1.txt. This file will be subject to the simple lifecycle illustrated below.

A very simple lifecycle

Starting in the state NEW we assign the file1.txt Artefact a revision of 1. The Artefacts two histories and its change log are illustrated below.

file1.txt - a simple change log

The developer chooses to change the original file1.txt but while in lifecycle state NEW we do not require a new Artefact, the resulting edited version of file1.txt remains revision 1. The lifecycle state also remains NEW, so although we have travelled the In-place Edit path in the lifecycle, the result is an edit event but no change in either the state or the revision of file1.txt. Using state and revision to identity the Artefacts (the one before the edit and the one after the edit) is therefore not sufficient to account for the two different edited versions of file1.txt.

We could simple state that we are not interested in such edits or we could also simply assign some ‘point’ revision to distinguish these intra-state revisions. Both of these options have been adopted by tools in the past, but when trying to generalise a solution we need to accommodate as many possibilities as we can. Generalising a solution should allow us to sit atop any underlying implementation. To arrive at a generalisation we need to explore the problem domain further.

Derived Identities

Perhaps the answer lies in content-based addressing (CBA). With CBA we identify Artefacts by creating a signature based upon intrinsic attributes of the item itself. This works very well for electronic data (in our case, files). There are many CBAs available and they are widely used in peer-to-peer systems for identifying files. The current darling of the CBA world is the SHA-2 family of hashing functions. These hashing functions process the stream of bytes in a file (or any other source of data), producing a single ‘summary’ string of characters which is much shorter than the original (often 64, 128, or 256 bits long).

CBA has huge appeal for our purpose in uniquely identifying an Artefact. A CBA is almost guaranteed to be unique to the data being encoded (I say almost because any CBA will ultimately collide — two data sets produce the same hash — since a potentially infinite set of data is being mapped onto a finite set of hash codes). A CBA is intimately related to the Artefact’s characteristics and so no assignment authority is required (see Assigned Identities a little later). This also means that given an Artefact we can work out its unique identity, providing we know the hash function and the attributes used for identification (a files data, for example).

CBAs work very well for electronic data. Unfortunately, CBAs do not work well for the general case. How, for example, might we assign a CBA to a physical Artefact?

Assigned Identities

This brings us to assigned identities. An assigned identity is one that, unsurprisingly, is assigned by some agent. Examples of assigned identities are legion. The ISBN you find on any book you buy is an assigned identity, not for that specific book, but for the data and form of that book (hardback version of the first edition of XYZ). Similarly, almost all products have a UPC (Universal Product Code) — the one that you find encoded in bar codes. Again, UPCs are assigned to a ‘type’ rather than the specific item; so the UPC code 50036 90803 refers to JBL Duet speakers, but not to the specific pair of JBL speakers sitting on my desk.

To ensure the uniqueness of assigned identities two conditions must be met: the scope of the identity must be well defined (ISBNs apply only to books, UPCs apply only to products, and not, for example, to ideas or patents), and there must be some central agency who defines and issues codes (or at least issues blocks of codes to be issued by approved agents).

Assigned identities are not derived from the Artefact that they represent, so there is no way, when presented with an Artefact, to establish that Artefact’s identity, other than by reference to some catalogue to see whether it already has an assigned identity. This could be a very long process if we relied on the data as a whole. ISBNs for example, can be looked up using the books format, title, and edition. These three attributes for the key by which the ISBN can be referenced in a catalogue. This presumes the existence and universal availability of a definitive catalogue.

Another form of assigned identity is the Universally Unique Identifier (UUID). UUIDs also use hashing functions to condense a set of attributes to a string of 128 bits, but the source data is not the file’s content but a series of attributes that are themselves (hopefully) unique. There are many implementations of the UUID scheme and they use different attributes to generate the UUID itself. Like CBA, a UUID is almost guaranteed to be unique — certainly sufficiently unique for most practical purposes.

So where does all this leave us?

An identity based on CBA is great for known data sets because the identity is generated from the data itself. We can therefore recreate the identity independent of any central authority. We can take a file and work out its unique identity and then use this unique identity within our tools. This, as we shall see in coming weeks, has significant advantages to resolving configuration data across distributed databases.

The problems with CBAs are that they do not translate well into the physical realm (we need to establish some characteristic of the physical object that can be uniquely selected for encoding into data and subsequently used to generate the CBA), and they are quite costly to generate (we need to read and process the entire data set to create the CBA).

Assigned identities that rely on an assigning authority (like the ISBN) require significant organisation and coordination. Each assigning authority must be organised to ensure that each identity is assigned uniquely. And here we run into another question related to the other form of assigned identity, UUIDs.

What do we mean by ‘uniquely assigned’? Suppose I create a file containing the text ‘The quick brown fox jumped over the lazy dog’ and I assign it a UUID. Some time later, you create a file with identical content and assign it a UUID. The two UUIDs will be different. Are we then dealing with two Artefacts?

In one sense we are. They are two distinct incarnations of the file. But in any useful sense, we are not. They are in fact the same file. Just as if I took my original file and copied it to another location in my file system I would consider it the same file. Or if you copied your file to my file system I would be unable to distinguish your file from mine. They are the same ‘thing’, but we have assigned different identities.

This is the problem with all assigned identities; how to ensure that the ‘thing’ being identified has no identity already assigned.

Is there an answer?

Yes and no. The work currently underway with the semantic web project (quick overview here) offers some hope. There is still a reliance on ‘authority’ to assign identities, but I think there is much to be learned from the general approach and using a combination of derived and assigned identities will allow easy reconciliation of information between CM datasources. I see the problem of configuration management (and particularly identification) as a special case of identification within the semantic web project.

This is a fascinating area (well, it is to me) of research and one that I believe will be increasingly important as CM moves into ‘the cloud’. As the distinction between CM data under your direct control and that controlled by proxy through service providers and third parties is eroded the need to consolidate CM information across disparate tool sets and managed by different agencies will become increasingly important. This is a problem that we must solve.

I spent some time a couple of years back considering Topic Maps and Resource Description Framework (RDF) as possible SCM schema. Although they are functionally equivalent, RDF seems to have won out in the standards battle. If you are interested in seeing where I think information management is going in the 21st century (and where CM must also go), check out the semantic web.