Many ticketing systems, supporting processes such as incident management, are prone to ticket ping-pong where tickets are bounced from queue to queue as people keep forwarding the problem to get it off their queue. This is simple to control once you see the problem.

Processes that suffer with this problem all have one thing in common; they all allow one user to simply pass a ticket from their queue to another unchallenged. For example, an incident ticketing system shows up a ticket on Person A’s queue. They cannot deal with it (or do not want to deal with it), so they look for someone they believe should deal with it and assign it to their queue (Person B). Person B looks at the ticket, decides they should not be dealing with it and looks around for someone else to deal with it (Person C). And so it goes, the ticket to bounced around from queue to queue never being handled but always being sent on, possibly even returning to Person A who forwards it on again and the cycle continues.

The problem is that the ticket owner in such systems is defined as the person who has the ticket on their queue and anyone can pass a ticket to another queue. Since no one wants to be lumbered with the ticket it just keeps being passed on. It becomes an SEP (someone else’s problem), doomed to forever wander the queues, never being resolved.

Fixing this process is actually very simple. When a ticket is initially raised by a user onto someone’s queue (let’s say Person A) the queue owner becomes responsible for progressing the ticket. If the Person A believes the ticket belongs to someone else (say, Person B), they propose it into Person B’s queue. But, and this is the important change to the process, ownership of the ticket remains with Person A. Person A remains responsible for ensuring that the ticket is progressed (not simply passed from queue to queue) until Person B accepts the ticket into their queue. This puts pressure on the Person A (as the ticket owner) to actually manage the process, whereas the original process did not pressure anyone because they could simply pass the ticket on to someone else.

It may seem unfair on Person A. It could be that they really cannot deal with the ticket. It could be that Person B really should take ownership of the ticket but they may be trying to avoid doing so by not accepting it into their queue. This is what escalation is for. If Person A has a legitimate cause they can escalate the problem and effectively force Person B to acknowledge their responsibility. In practice this is seldom an issue as Person B, if they recognise their responsibility will simply accept the ticket into their queue and progress it to completion.

I Person B genuinely cannot, or should not, deal with the ticket they simply inform Person A of their reason for not accepting it into their queue. Person A must now seek the correct person to resolve the ticket. Because Person A is responsible for the ticket until someone else takes charge of it they have a vested interest in finding the correct person to deal with it. If they propose the ticket to someone’s queue and do not get a quick response they have a vested interest in following up to find out why the ticket has not been accepted. Person A is motivated to find the correct ticket owner since no one but the correct ticket owner will take the responsibility for progressing the ticket away from Person A and until some takes it from Person A it remains theirs to progress.

Ensuring that people cannot simply pass tickets around the system ensures that they are progressed because it puts pressure on owners to either resolve the ticket or to find the person who really should resolve the ticket.

In this article I am going to lay out my definitions for some terminology that will become increasingly important as I develop my CMS model.

The terms I will be discussing are as follows.

• Stream
• Branch
• Configuration Item
• Revision
• Configuration
• Component
• Repository
• Configuration Management Database
• Record

At this point I caution the reader that these definitions are deliberately quite loose and informal. Each will be expanded, refined, rewritten and formalised as I work through articles in this blog. For now, my working definitions are as follows.

Project

A coordinated effort usually conducted by several individuals to deliver a Product. Project describes the totality of planning and activity requires to gather requirements and interpret these into Product.

Product

That which is to be delivered by a Project. Products include, but are not limited to:

• Executable software
• Documents — manuals, design documents, requirements, installation guides, administration and maintenance manuals
• Hardware — computers, network components, any other physical components required as part of the Product
• Training materials — exercise version of data or system components, trainer presenation, training the trainer material, sandbox systems for trainees
• Source code — when developing for a 3rd party. Source code may also be a deliverable in interpreted languages or when delivering web content such as HTML.
• Media — video, graphics, audio

Stream

Projects often consist of more than one piece of development. A common strategy is to manage these pieces of development as a sort of sub-project. Timescales of these Streams are overlapped to allow the project to compress its overall timescale.

Branch

An implementation technique used in development to manage simultaneous changes to common items. In software development Branches are common and used to allow two or more developers to work on the same source code simultaneously.

Configuration Item

A configuration item is an item within the configuration management system that is the focus for change management.

Revision

Each time an item is modified and submitted into version control, a new revision is created. In this way an item’s history can be traced by looking back through the sequence of revisions.

Delta

The difference between two revisions.

Configuration

A specific arrangement of item revisions.

Component

An item that is subject to version control, but is not elevated to the status of a configuration item.

Repository

A safe store for item revisions.

Configuration Management Database

A database containing information about each item revision and their relationships to one another and to records.

Record

A set of data that is subject to a process or workflow but not necessarily version control. Records normally carry information used to account for an item’s current disposition or the current state of a process or workflow.

Fences seem most appealing (even if only for humanitarian reasons), but they are expensive to construct and maintain. Furthermore, constructing a high fence will block the beautiful views across the sea and people will stop visiting the site (which is bad). So your fences can be only so high before they become an impediment and stop people using the facilities. Unfortunately, keeping the fences too low means people can still climb over them and (you guessed is) fall to their doom.

Maintaining ambulances at the foot of the cliff is also expensive. There is the possibility that people falling over the cliff will be so harmed that they do not survive the fall and even those that do survive may be out of commission for some time. Ambulances seem like a last resort, a safety net. Too few ambulances means more injured fallers die when they could have been saved, too many ambulances means increased cost without any benefit.

There is a balance to be struck. Fences to guard against the unwary and ambulances to recover the unfortunate (or plain stupid).

This balance between ambulances and fences is the line that process engineers walk. We try to design fences that are cheap to install, efficient to maintain, stop the majority of people falling over the cliff, and allow people to enjoy the facilities along the cliff edge safely. At the same time we know that there will always be those who venture beyond the fences (they cannot be secured without blocking the view and driving people away — possibly to another cliff), so we prepare a small fleet of ambulances to minimise the impact (forgive the pun) of those who fall.

Creating a process that allows facilities to be used efficiently but also provide a means of identifying and safely recovering when things go wrong is the skill of a good process engineer.

A good example of this is the combination of change and incident management. Change management is a fence, a process designed to avoid problems. Incident management is an ambulance, a process designed to minimise problems once they occur. When change management fails because either someone circumvents the process (climbs the fence) or because the process is incorrectly applied (the fence is damaged or too low) then incident management identifies the resulting problem and hopefully resolves it as quickly as possible (ambulances recovering the injured faller and saving their life).

If change management processes are too draconian they will be expensive (delaying change and making your organisation unresponsive) or simply circumvented when users perceive them to be an unacceptable impediment. If incident management is under resourced they will be incapable of recovering from problems when they occur. As change management becomes less effective the cost of incident management rises, as change management becomes more effective the need for incident management declines.

Striking the right balance is a matter of judgement and will vary from organisation to organisation, and from service to service. The more sensitive your service is to problems, the more change control is called for, the less sensitive your service is to problems, the less change control is required.

A problem in, for example, an order processing system for a mail-order company has serious consequences and suggests more change control should be applied. The cost of failure is so high that incident management costs would be unacceptable. Within the same organisation the system that produces the print catalogue is less sensitive, so change management costs would normally be lower and more incident management might be called for.

The key to good process design and implementation is balance between constructing fences and maintaining ambulances.

CM Tool Architecture

As with all multi-layer architectures the important feature is that each layer creates an abstraction then allows layers it serves to use a stable interface. The CM Data layer, for example, will need to deal directly with individual the tools we wish to interrogate. (Recall that I am currently only proposing a tool that analyses and reports the content of the underlying tools, writing a tool to also update the content of those tools adds a whole new set of challenges to the problem. Challenges I have no desire to deal with right now, but which will be served well by this architecture nonetheless.) The CM Data layer presents one, hopefully, stable interface to the CM Model, so the CM Model will not have to be changed is a tool changes its interface (changing its database schema, command line, API, or whatever method we use to access it).

CM Data

This layer presents a common interface to the CM tools that the CMS is fronting. CM Data can be divided into two categories.

• Item library — this holds the items the CM tools store; files, directories, symbolic links, etc.
• Data about items — this is data such as relationships between items, attributes of items (owner, author, date modified, and so on).

The CM Data layer will provide methods for accessing these data. Items will be held in a Repository and the data about these items will be held in a CMDatabase.
All version control tools are a Repository (they store items) and provide at least some data about those items (author, modified date, ancestor/successor relationships, contains relationships), so they provide some functions of a CMDatabase. More advanced tools (like Dimensions and Synergy/CM) provide the same Repository functions but they expand these considerably (adding in records such as change requests) and the CMDatabase functionality is also expanded. These points will be revisited and clarified as we proceed.

CM Model

The CM Model will provide an abstraction of common CM artefacts (items and their relationships, collections, baselines, branches, changes and so on). The point of this abstraction is to, once again, provide a consistent and stable model to the layers that use the CM Model. So, the code in the Business Logic and Presentation layers is written to use the CM Model rather than the various implementations of these artefacts presented by the underlying tools. This means our system can access, say, an Item without needing to know how it is represented in the tool or even which tool it is held in.

The CM Model is perhaps the most important part of the architecture to design. It needs to be sufficiently comprehensive to accommodate the various tools’ concepts but at the same time it needs to be as simple as possible and as stable as possible to preserve a stable interface to the rest of the system.

Business Logic

The Business Logic layer is where all of the rules about how the CM Model is to be interpreted and managed. The most challenging aspect of the Business Layer is the need to allow users to define their own rules without requiring them to program anything into the system (although we will surely make the tool easy to extend programatically). This layer will require a detailed set of user cases and a considerable investment of time in the early design if the final product is to be flexible enough to accommodate a wide variety of users.

The Business Logic layer’s functionality can be divided into two classes; those features of SCM that are fundamental and those that are definable. Establishing fundamental features may be a matter of diktat on my part, but never-the-less I will try to only hardcode rule when absolutely necessary, preferring instead to provide configurable rules. *scratched head* This could take some time.

Presentation

Finally there is the presentation layer. Here all of the various methods of presenting the system’s functionality are gathered. This is perhaps the simplest layer to define functionally but the most difficult to design aesthetically. Having established what the system can do in the other layers the function of the Presentation layer is to provide a method of allowing the user to request that functionality, and providing a means for the system to provide the results to the user. The challenge with user interface (UI) design is how to achieve these two functions in an easy to use and, preferably, intuitive way.

In the first instance I shall probably provide a command line interface as a priority with some form of rudimentary GUI as a secondary development. Once these are done other interfaces may be considered.

1. People are often more afraid of the perceived problems than the practical realities of parallel development.
2. People do not truly appreciate the problems and practical realities of version control and parallel development.
3. There is a need for more theoretical work on the topic and, perhaps more significantly, a need for more formal expression of the process and problems involved.
4. There is a quite substantial book that could be written on the subject.

The first two points are related to the last two insofar as people need a clear explanation of the discipline and, to really acquire a firm understanding of the subject, you need to commit to studying it. There is simply no easy way to explain the more complex material (and by simple I mean one side of A4 summaries demanded by people who think they need to know but who have no practical exposure to the technical nature of software development or version control — there is no ‘managers guide too..’).

As to the third point, I have expressed a concern about this before in various forums. Rather than continue to observe the problem I have decided to start doing something about it. So, over the coming weeks and months I will be expanding some ideas I’ve been mulling over for many years in a series of posts to this blog (special category set up here) and on my website. Be warned though, these articles will probably get very theoretical, often be speculative, and may raise as many new questions as they resolve existing ones. We shall see.

As for the final point, as I build up a sufficiently cogent set of material while writing on the blog (a process unlikely to be well structured), I will assemble more coherent articles on my website and who knows I might get round to pulling them together into a book one day. (Ha! Another of the many books I either intend to write or am in the process of writing but may never finish or publish — *sigh* so many things to do, so little time.) I may even pull material together and add it to the CM wiki or the ITSLM BoK. You never know, stranger things have happened.

Rather than start here, I’ll start an introductory post right now and hopefully have it up in the next day or two. In the meantime, you could check out some past posts on the subject; here and here

Simple Merges

The most basic merge is a very straightforward (although, as we shall see, can have hidden complexities).

Simple Merge

The most obvious limitation of these simple merges is that the file must be ‘mergeable’, which for most tools means the file must be record oriented. Text files are a collection of records, one for each line in the file. Binary files tend to be resistant to merge unless a special tool is provided (MSWord can merge two documents with reasonable accuracy).

Conflicting Merges

Sometimes a merge involves changes to the same records (the same lines in a text file) in the same file. The illustration below shows just such a merge.

Conflicting Merge

Sometimes these conflict resolutions will require more than a simple decision about which line from the two changes should be adopted and the developer will need to rework both changes to make them work together. Fortunately the majority of merge conflicts are relatively straightforward providing integrations are performed frequently.

Renamed and Moved Files

While merging a file’s content is commonly supported by version control tools, tracking renamed and moved files during a merge operation is less well supported but in todays software development world, where refactoring is common, it is increasingly essential.

The problem with renamed and moved files are summarised in the following illustration.

Merge with rename

Rename/move conflicts

Merges involving renames do not necessarily run smooth. Not only are they subject to the same content merge issues as the simple file merge but also the problem when the file is renamed/moved in both branches. This situation is illustrated below.

Rename conflict

Mitigation

Before moving on to some other issues you will encounter when using parallel development, we pause briefly to consider how to avoid, or at least mitigate, some of the issues mentioned so far.

Firstly, despite the impression the previous sections may have conjured, merging is not fraught with perils. After many years of managing parallel development on projects ranging from the simple to very complex, I can assure you that, with a little common sense and some forethought the issues can be minimised (but seldom completely eliminated).

1. Merge as early as you can, and then merge often to keep each merge as small and manageable as possible.
2. When you have input to a project’s design, ensure a sound architecture. This will minimise overlapping changes and consequently merge conflicts.
3. If you identify files that are merged often, try to decompose the file into smaller files to avoid high merge rates. (This is common with files used for configuration or files containing a lot of utility code.)
4. Keep refactoring on one branch. Renaming and moving files can cause tricky merge issues when performed on many branches.

Now, on to some issues that, while not unique to parallel development, can make your life difficult if not managed properly.

As with most people who have worked long enough on the coalface of SCM I have a ragtag collection of routines, scripts and utilities already but now is the time to pull them all together, clean them up, give some thought to a coherent design and build a better mousetrap that I can use whenever I go to a new client.

This is not an effort to develop a be-all-and-end-all CM system, but more a matter of me realising that I develop the same sort of scripts time and again. So I would be better off developing a more robust architecture that will allow me to exploit report generators or integrity checkers written when using one tool when I go to a client who uses another tool but has the same problem to be solved.

A typical example of a script I have written several times is one to check the integrity of a build configuration. Answering questions such as, ‘does this build configuration contain all, and only, the completed change requests assigned to this release?’

It would also be nice to have a tool for federated reporting across tools. Often clients have several version control tools and it would be nice to have one reporting front-end that I could plug into their many VCSs and get information about them all.

There are, of course, many problems to overcome (not least all the different philosophies supported by these tools) but this is not a case of trying to solve all problems in one go, rather the idea is to write a facade that will allow me to access the implementation tools through a common interface. In this way, all I need do for each client is write a new facade that complies with my own interface definitions but interrogates their particular implementation. The hard parts of these scripts, the logic, will remain unchanged and useful in multiple applications, treating the tools as databases.

As with many efforts born of frustration, this one is not completely thought out yet (in fact some of the ideas are positively vague), so expect more to come on this side project…

There are a multitude of technologies that can be used to create glue software, including but by means limited to; SQL, various shell scripts, C, C++, C#, Java, PHP, Python, various web technologies (XSLT, ASP etc.) and so on. Over the years I have used all of these and more but by far and away my favourite glue software development language is PERL. And here’s why…

PERL is ubiquitous. There is almost no regular platform on which PERL cannot be run; Windows, *NIX, Mac and Mainframe are all generally covered and therefore most of the platforms one is likely to be using in a modern CMS are covered.

PERL is simple. Well, more accurately, PERL is easy to teach/learn because it is accommodating, typically providing more than one way to express a programming notion. PERL is a fairly forgiving language, but it has to be said that sometimes this flexibility leads to confusing pieces of code. One soon learns to interpret the more idiomatic usage though and the confusion is lifted leaving only the choice of programming approach afforded by PERL’s flexibility.

PERL has a vast library of freely available libraries. Like many languages that are available across multiple platforms, PERL has developed a wide following from many disciplines and this has meant the development of a large library of routines to do almost anything you want. Check out CPAN and look through all the modules available there. To paraphrase Apple’s App. Store tag line, ‘there’s a PERL module for that’. This facility alone can save hours of tedious programming.

Write it once, use it anywhere. Using a native script language (such as ksh, Bourne shell, or even – shudder – BAT) means you are largely tied to one platform. PERL’s ubiquity, when combined with careful programming, means your carefully crafted and debugged script will work pretty much anywhere.

PERL has a long and illustrious career as a web programming language. Pre-dating Java, PHP, ASP and others as serious dynamic web technologies¹ the Apache mod_perl module has allowed PERL code to be embedded in web pages for many years.

With PERL modules like Tk (or Tkx) you can create graphical user interfaces for your scripts that adapt to the window system on which they are displayed. (The newer Tkx provides a much more raw interface to the underlying TCL/Tk libraries. It also, to my eyes, produces a better UI that matches more closely the host platform. Basically because the TCL/Tk libraries are more up-to-date than the Perl/Tk implementations.)

So, that is in broad strokes my case for PERL. But there’s more.

Perhaps the best supported PERL package is that provided by ActiveState. Although hardcore PERL mongers may consider this limiting I suggest (especially to new PERL users) using ActiveState’s installation and limiting yourself to using the packages provided through there PERL Package Manager from their repository.

Although ActiveState provide only a subset of available packages it is a substantial subset and, more importantly, they work across all the platforms supported by the main ActiveState distribution. This in turn means that the write once, use anywhere philosophy is easier to maintain.

Added to this common support facility the ActiveState PERL Developer Kit (PDK) provides a cross platform packaging facility that means applications developed on one platform can be wrapped into executable packages suitable for multiple target environments. So, if you develop your PERL on Mac OS X you can provide packages targeted for Windows, Solaris, Linux and AIX platforms without ever leaving your OS X environment. And that has to be a good thing.

As if this were not enough, the PDK also provides the means to package your PERL as a Windows service, a dll, a System Tray utility, or a .NET library (making your PERL libraries available to your .NET developers).

As with any broad statement, claims of PERLs utility must be accompanied by a cautionary note. The path to PERL Nirvana is not an easy one. Not all PERL modules are of a high quality. Not all PERL applications will easily package using the PDK. And not all PERL code will work on all platforms. Just as with any programming language you, the programmer, must be aware of platform differences and ensure your script will handle them gracefully.

All this said, with a little forethought and care PERL provides a powerful platform for glue software and more.

A properly designed CM system will be able to handle any IT system environment. Problems only occur when you are married to the idea that centralised control of CM data and centralised change control are necessary for CM, or that a specific development process is integral to CM for it to function correctly.

It is an all too common mistake to design your CM system as a part of another process. For example, adopting a particular software development process and then designing your CM system into this development process will result in a CM system that is inextricably linked to the development process.

At this point some of you may be protesting that this is not a bad thing. Wrong. It is unnecessary, bad practice and lousy system design.

The irony is that many people who design their CM systems as integral parts of their development process are the same people who would be appalled if they saw such poor design in their software systems. They are effectively designing their CM system as a tightly coupled aggregate to their development process when it should be quite separate.

A CM system should be designed with clear interfaces that can be used by any other process. These interfaces should include:

• Change control – although change control is identified as a part of configuration management, it is preferable to maintain change control as an independent process that interfaces to the configuration management system.
• Query and reporting – getting information out or the configuration management system must be performed through a clearly designed interface.
• Configuration item references – a clear and unique method of referencing any CI in the configuration management system must be provided.
• External references – the configuration management system must be capable of unambiguously referring to any entity held outside the CM system itself.

By maintaining a clear interface like this the configuration management processes are insulated from any change to other processes in your IT development and operation.

Maintaining a consistent set of well defined interfaces also allow you to create a configuration management system that can be easily replicated across your organisation. The same configuration management process can be given to each of our organisation’s development projects for example. Providing the configuration management processes have been well designed with clean interfaces they should all freely inter-operate and, from the organisation level, will look like one coherent sytem.

When you need to extend your configuration management system beyond the boundaries of your organisation (for example, when interfacing with a service supplier or third party developer) all you need do is ensure that their configuration system presents your internal organisation with your interface. This can be achieved by either agreeing with the external agency that they will use your process or by creating a facade that will take present your interface internally but communicate with their systems externally.

Although his might appear complex it is not and it pays dividends by ensuring your configuration management system is easily to extend and will interface with all your existing processes.

As a freelancers I see part if my role on an contract as eliminating as much of the purpose for my presence as possible. This means optimizing away my role, generally through process efficiency (ensuring there are no extraneous operations in the process), automating as much as possible, and ensuring that my role is not performing tasks that can be performed by others.

Process efficiency

Examine you product lifecycle and eliminate all unnecessary steps in the process. If a step is not essential, remove it. If a step can be optional, make it optional. If you can reduce the number tasks in a process step, then do so. Bear in mind that a good design is not arrived at by adding features but by removing features until you have the least possible without compromising the objective of the design.

Automation

Automate everything you can. If you repeatedly create certain reports then create a scheduled job to run any scripts required to generate them. Make reports availble on a website if they are for general consumption, or e-mail them to controlled list if they are for a restricted audience.

Automate all builds and make sure you automate all deployment and testing. Any output from testing (including things like test results, coverage data and performance statistics should be collected and published on a website).

Automate baseline promotion if at all possible.

Make baselines that pass each test cycle available for automated deployment into the next test environment. Make this a ‘one button’ demand from the target test environment manager.

Task reassignment

Closely related to process efficiency, task reassignment is ensuring that your role is not performing inappropriate tasks that distract from work that you are needed for. Often a role, especially a role such as configuration, change or build management accumulates tasks by default. Make sure that all tasks in the processes are assigned to the appropriate owner.

For example, these roles are often tasked to produce ad hoc reports. It is better to make it easy for others to create their own reports as they need them. In this way you eliminate the need to burden yourself preparing reports that people can produce themselves. The cost may be a small amount of time training people to prepare their own reports, but this is more than offset by the time gained in not preparing every ad hoc report required by users.