The Software Minefield

This article was originally published in The Testing Planet magazine, as was the titular piece of my first book, The Software Minefield.

Let me take you on a journey...

You are with a party of friends on a beautiful walk in the countryside.  It's a lovely day, and according to your map you are making good time.  Some time soon you should arrive at your destination - a café which has come highly recommended to you.  You can almost smell the coffee and cakes now!

As you open a gate a sheep accidentally comes through, and soon bounds off in front of you.  Imagine the total shock when with a sudden BOOM it explodes before your eyes.  It's at that moment you realize that although you didn't intend to, you've somehow entered a minefield.

 Suddenly your beautiful day has become treacherous and your good progress has lead only into peril.  The only way you'll get out alive is by making good decisions from here on in.

 There is no doubt that software development echoes this parable in many ways, especially when it comes to testing.  Our projects can often seem to make good progress, but little do we realize we're just getting ourselves deeper into a minefield.

As testers we can often feel much like that rambler, we know there's peril out there in the form of defects, but we're not sure where exactly and of what severity.  But often to the rest of our party the wake up call will only come when the first mine has gone off ...

How do we best deal with minefields?  And how can we apply that to software?

One approach would be just to back up and hammer a sign (okay, being very careful where we hammered it) to warn "Caution Minefield". Then we can quite happily state that anyone entering does so at their own risk, and we did warn them.  Some companies have indeed taken this attitude with customers, saying "well our customers will let us know if they find any problems".  The problem with letting customers loose in a minefield is that you could soon find yourself running short of customers...

In a similar vein, we can follow the example of history, and drive a flock of sheep over the field, and find problems that way (helps to have a bottle of mint sauce handy though).  This can be what happens when we let loose 'end users' on our system in a controlled way, and "if it doesn't fall over it must be okay ... but no people will be harmed".  This is rather quick and dirty way to test, and although it should find a lot of problems, it's rather random, and not guaranteed to make it completely safe.  Who knows what bits the sheep might be avoiding ... indeed the sheep may have a mentality towards self-preservation the shepherd isn't aware of.  Maybe the grass above a land mine just isn't juicy and fresh enough to tempt a sheep.

Much better is to bring specialist mine hunters or engineers in to clear a path.  They will need some expensive equipment like metal detectors to probe ahead, and when a mine is found, they need to work with an expert to remove and defuse it.  But this is slow and methodical work, with everything almost grinding to a halt when a big problem is unearthed.  In the story above, as your engineers clear a path, there will be someone who will complain it's too slow and the café will be closed by the time you're across the field.

And though you'll have cleared a path, it may still not be safe.  Your friend needs to go to the toilet behind a bush whilst this is going on, but that involves stepping off the safe path.  When can you declare a field safe and move on?  Do you keep going backwards and forward with your engineers until you've covered every square inch?  Do you cover the main footpaths first?  Or can you claim it safe if you've not found any mines for the last couple of hours?

How can you ever really be sure?  It's easy to know about the presence of mines - they have a way of making their presence known - but how can you be confident of their absence?  How can you be sure that as you're unearthing mines in one part of the field, some scoundrel isn't laying new ones in the other end (ooh, and you thought the minefield was static - that could be a dangerous assumption)?

That question is of course one that applies to testers, managers, developers.  How can you be sure you've de-mined your software of defects?  Of course you never can be sure, but it doesn't hurt to have covered as much ground as possible.  Every mine you find makes the field safer.

In reality we tend to bring in our specialist demolition engineers, our Testers to check methodically as many paths as possible.  Then to cover ourselves, we bring in our users for acceptance, and send out the sheep ...

 As with so much in life there's no "one way" to solve this problem scenario.

There's really no way to one way, no right way to solve this.  You have a drive to get somewhere (your café).  But you can easily spend the rest of your life going backwards and forwards in this field before you can declare it safe.  Of course if you're careless, you will be spending the rest of your life here.

At the end of the day, the most important thing is we and everyone in our party respects that we're in a minefield, and act accordingly.  Not letting our leaders persuade us "you're all being silly, it's just one mine, c'mon follow me, they'll be out of meringues if we dawdle ...".