Just because you can automate everything ... doesn't mean you should

In total I've accumulated almost 20 years in IT - throughout that period I haven't always been a tester, although I've pretty consistently tested.

To me, when I was a developer, "doing testing" was something I was very good at, but something I was assigned at the end of a project to "get it over the line".  But in 2004, someone wanted to put my programming skills to good use, and I was brought in to help out on a project which was using test automation.  It was a huge learning experience, which caused my job title to change from software engineer to automation tester.  Since that day the word "test" has been in my job title ever since.

Currently I'm picking up a kind of "ghost in the wires" murmuring about automation testing replacing manual testers.  I was even asked about directly by a student at Summer Of Tech who'd been advised "don't go into software testing, it'll soon be extinct".

Back in 2004 though I joined a department where all testing was done through automation.  Since then I've always attempted to learn more about tester, and importantly tried to blend automation and manual techniques.

For many today 100% automation is the Holy Grail - why don't I believe in that anymore?  What did I see?

* Note - I'm going to use the terminology "automated test" in this experience report, because it's appropriate from my understanding of the period of which I'm writing.  Since James Bach and Michael Bolton have done recent work I'd really call these "automation checks" today.  However I want this experience report to be true to my understanding of the time.


Project Case - an experience report

Project Case as I've said was the first project I've worked on with a dedicated team of full-time testers. It was a bespoke product of about 10 years, with a rich functionality.  The test manager Stephen was a first in many ways for me - even though I was an internal transfer he put me through a barrage of questions on how I'd think to make tests for a product before accepting me on the team.  He took testing very seriously, and I did learn a lot from him.  Though I didn't always agree with him - but at the time, my understanding of testing, and more importantly being able to talk about it, meant I couldn't counter argue with him to the extent I could.

We were system testing for Project Case - and our tests were 100% automated.  Typically about 2 months before a new release we'd start scripting up new automation.  We'd even create scripts which would repeat defects that the customer had found.

At the start of system testing, we'd spend about 2 weeks testing new functionality, running tests on new features.  After that it'd be 6-8 weeks running regression.  Stephen was terrified about regression - which meant we ran every script we'd ever run, plus every script for every defect we'd ever found.

That in a nutshell was our project, now in more detail, I'm going to talk about where we found problems (you've probably already guessed some of them).

Automation wasn't really very efficient

So we had 100% automation - BOOM!  So I guess that meant we had a tight ship?  Actually the team was 12, and comparing to other teams I've since managed, I'd expect to be able to go through the same level of manual testing as we achieved with a similar sized team.  Maybe (shock-horror) even with a somewhat smaller team.

We weren't running faster and smarter, for reasons that will become obvious.

The team was hired foremost for their testing ability - not coding skills

This was one of the best teams of testers I've ever worked for.  And they were hired for their ability to test.

They were then made to write Visual Basic scripts.  And this is where it fell apart a bit - because many weren't very good coders.

Although I never met him on this project, a lot of it was written by John Bunting.  John Bunting, for Terry Pratchett fans, is a kind of Farnborough version of Bloody Stupid Johnson.  He's a guy who's code is so bafflingly strange and bad, I know of friends who were still trying to unpick what he's written 5 years on from his retirement.

But our automation tests were riddled with weird code - such as a function called "back to menu" which would repeatedly hit the escape button, take a screenshot, and check for the main menu.  Only sometimes it would repeatedly do this, miss the menu, then keep hitting escape and taking a screenshot.  If such a script was left running overnight, it would overload the database, and we'd need to call a DBA to fix before we could run anything the next day.

It's fair to say I spent a lot of time trying to address the technical debt within our automation, but ultimately the best I managed was band aiding the code.  I also tried to make the team familiar with Visual Basic coding standards, but in many ways the damage was done.  Thanks John.

We needed a good level of programming skill in the team.  In truth, that's why they got me.

We didn't really understand the system we were testing

Our main priority was running the legacy automation, and fixing problems as they occurred.  We spent more time fixing our scripts than actually using the product under test.

Although we had some testing heavyweights here, our attention was always on the automation and fixing it.  A constant problem thanks to the code of John Bunting Esquire.

Consequentially we never really understood the product under test to the level other test teams would have who'd manually tested.

We needed to be able to understand the system being tested, not the automation.

We didn't really understand the automation

These automated tests had been run for years, because they'd always been run.  They were large and meandering testing many things, and as lengthy automation even when they worked they'd take several hours.  When they failed, because of the quirks of our tool we'd have to start them running from scratch, whereas a manual tester could go "well I can adapt and finish from here on in".

We needed the automation to be simpler, quicker to run and with an obvious feature that it was trying to test.

The regression was too heavyweight

Often with a mature product "testing the core functionality" which people think of when they think of with regression testing.  We were trying to test "everything we've ever tested", which meant for each release it got harder and longer, and we needed more people.  Testing became a huge burden for the project.

We needed to focus down on what mattered, and dump everything else.  If we didn't think an automated script failure would end with a critical or high level defect raised, we probably should have dumped it.

Our automation was a monumental failure

I hate reducing things to metrics.  But for every one defect we were finding in the system, we were finding and fixing between ten and twenty issues in the automation code.

Think about what that means - we were testing our automation code more than we were testing our product.  That in itself is a massive failure.

We needed robust code which was easy to fix, and failed relatively infrequently.



Dr Ian Malcom says ...

Yesterday, when talking about automation, and this project in particular, I found myself quoting Jeff Goldblum from Jurassic Park,

Don't get me wrong, automation can be incredibly helpful when used right, and I hope to focus on that.  It can really help you, but it does so by aiding your manual testing efforts, and stop you running manual tests which are repetitive and boring - for more information please reread.

Here are some good questions to ask of your automation check you want to build ...

1. Do you want to run this automation check time and again?  If not, automation is the wrong strategy.
2. Do you have too many automation / does your automation take too long?
3. If this test failed, what level of defect would you raise?  If the answer to that is low, then does it make sense to check it frequently?
4. Is it clear what this script is supposed to check?  Try to make each automation script check a single thing - makes it simple.
5. If you have a wide variety of tests you want to run on a product, which you only expect to run once, manual testing is still the more efficient way to run them.

Sadly - I've learned this, and it cost my company of the time to learn this.  People like James Bach and Michael Bolton have learned this.  Local testers like Katrina Clokie, Aaron Hodder, Oliver Erlewein and Kim Engel have learned this.  But it seems through gossip and misselling, there are a few organisations who need to learn this for themselves all over.


And finally - revisiting that Dr Malcolm speech ....

Revisit that famous Jurassic Park speech here.  Listening to it last night, I realised with just a few tweaks about automation it would be all-too relevant.

Dr Malcolm: The lack of humility about testing that's being displayed here, uh... staggers me.

Manager: Well thank you, Dr. Malcolm, but I think things are a little bit different then you and I had feared...

Dr. Ian Malcolm: Yeah, I know. They're a lot worse.

Dr. Ian Malcolm: Don't you see the danger, inherent in what you're doing here? Test automation is the most awesome force the planet's ever seen, but you wield it like a kid that's found his dad's gun


Dr. Ian Malcolm: The problem with the engineering power that you're using here, it didn't require any discipline to attain it. You read what others had done and you took the next step to copy them. You didn't earn the knowledge for yourselves, so you don't take any responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could, and before you even knew what you had, you patented it, and packaged it, ... and now you're selling it, you wanna sell it. Well...

John Hammond: I don't think you're giving us our due credit. Our people have done things which nobody's ever done before...

Dr. Ian Malcolm: Yeah, yeah, but your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should.



If there is a single lesson from this, just be very clear when you want to test to be automation, if it makes sense, and if it should be tested.  Don't be a John Bunting.

Jutland Fallacies

This week commemorates The Battle Of Jutland - the largest naval battle in history that took place during The Great War between the navies of Germany and The United Kingdom.  In this piece we're going to look at the run up to the battle, some of the thinking of those in charge, and looking to see if fallacies were in place.  And even try to spot them!

Setting the context

At the start of the Great War, the British Empire covered the globe, serviced as it was by a giant navy which since Nelson and The Battle Of Trafalgar stood unopposed.  Britannia ruled the waves.

Such a large sea empire demanded a large navy - and Britain had the largest on earth.  Until recently it was large enough to challenge the next two sea powers combined.  However a new kind of battleship, the dreadnoughts had so revolutionised naval warfare with their huge guns and steam turbines that they practically rendered all other ships outdated and obsolete.

The British Empire was in a building frenzy to maintain dominance, but Germany was catching up fast by the outbreak of war.

The German Navy had sought to trap pockets of the British Navy and whittle them down, however at Jutland, the entire German Navy fell into a trap, being lured into a headlong battle into the entire British Navy.

The British Navy were in superior numbers to the German Navy - 151 ships to Germany's 99.  The British Navy were not only in prepared firing ranks to maximise their firepower, but were also one of the best trained and professional seamen in the world.

As the clickbait says "what happens next will amaze you ..."

Germany 3 - Britain 1

Jutland is a complicated battle to sum up.  From the actions on the day, the battle is a decisive German win.  When looking at "the big ships sunk" Germany will lose 1 battlecruiser, whilst Britain will lose 3.

But there is more to it than that - German battlecruiser Lutzow would be hit 24 times by British shells.  The damage was extensive and it was originally hoped she would make it back to port.  However eventually her handling was so poor she had to be scuttled.

On the British side though, there's a very different story - the battlecruisers Indefatigable, Queen Mary and Invincible all suffered a very similar and violent end.  In each, the ships exploded after just a handful of hits.  In each case, of the crew of over 1000, only a handful of men survived.

The day would lead a British Admiral to commend "there seems to be something wrong with our bloody ships today".  What had gone so wrong?

Tales from the magazine...

Diagram from Wikipedia 

To answer that question we need to cover off normal operation of a key component of a battlecruiser - the gun turret, and how it's fed ammunition from it's magazine.

The magazine - where shells and cordite explosives are kept have always been in the deepest parts of the ships. The theory being they're below the water line, and harder to hit with chance shots.

In peacetime onboard British ships, priming the big gun with ammunition was all about safety.  The shell room and magazine for cordite are highly armoured, and isolated from the lift shaft that feeds the turret by safety doors, to prevent the chances of a blast blowing through into either - a disasterous situation which would spell instant doom for the ship.

That makes sense on paper...

That was peacetime, but in 1916, this was war!

Within the modern warships of the time, there was a kind of iron triangle rule in place,

• Armour
• Firepower
• Speed

Pick two.  The battleship focused on armour and firepower, to the determent of it's speed.  The battle cruiser though picked more firepower and speed - it was seen as almost a naval form of cavalry to do lighting raids and get out, particularly picking off enemy shipping.

British battlecruisers focused more on having big guns, sacrificing armour compared to comparable German counterparts.  There was a logic to this - British battlecruisers could strike German battlecruisers from further away, where the German ships would not be able to strike back.  Their range would be their armour.

Being a better drilled and practiced navy, Britain could maintain a faster rate of fire from it's big guns than German, and during the war, this rate only increased.  And this too made sense to the Admirals, because if the Germans were being heavily shelled at a faster rate, they would struggle to reply.  So the British fleet's rate of fire would be their armour.

To make this rate of fire as fast as possible, it was decided in drills to stop the practice of shutting magazine and shell room doors whilst feeding munitions up to the turrets above.  They would be kept open to "keep feeding the guns".

This would of course be risky - these people weren't idiots.  But it had a logic to it - the more shots you got off at the enemy, the less time they'd have to respond, the quicker they'd sink the enemy.  Obvious.

But this also presented a problem - the sustained firing rate meant in a heated battle, a British ship run out of ammunition before it's German counterpart.  There was only one logical thing to do - to make sure you were carrying more ammunition - so it was decided to carry about 50% more ammunition in wartime.

Of course there was a small problem - there wasn't room in the current magazine for any more ammo.  So instead it was stored throughout the ship, wherever there was space.  Of course this wouldn't be in an armoured space, and was incredibly risky.  But there was a logic to this - this ammunition would be used first, and besides with the better range of guns, faster rate of fire and speed, it's not like the ship was in much of a risk of being hit at any rate.

F-A-L-L-A-C-Y spells BOOM!

On paper, each of those arguments seems to make a certain kind of sense,

• Big guns means being able to hit from a range you can't be hit back. So you'll be safe.
• If you hit them more, they have less chance to fire back. So you'll be safe.
• The more you shoot them, the faster they sink. So you'll be safe.
• Take more ammunition. So you'll be safe.
• Because you're so safe, you don't need to follow full safety precautions which only get in the way.

I love the definition Wikipedia has under fallacy, "a fallacious argument may be deceptive by appearing to be better than it really is".

To me, a fallacy is an idea which seems to initially make perfect intuitive sense ... just so long as you don't start thinking too much about it, and finding all the holes.  I've found Daniel Kahnman's Thinking Fast And Slow has been filled with superb examples where our fast, intuitive thinking can lead us astray, and where to be wary.

All those arguments make a certain logical sense.  What's missing though is that as you take each step of action, you're taking a course of action that's putting yourself at greater and greater risk.  The question not being asked is "does the extra aggressive punch this gives me take off comfortably with the risk?".

At Jutland the answer turned out to be a resounding "no".  With all the British battlecruisers lost, the explosion in the magazine happened soon after a chance shell hit the turret, setting off a catastrophic chain reaction - one which could have been prevented by following the normal peacetime safety guidelines.

HMS Lion - narrowly avoided a similar fate

This was almost the fate of the HMS Lion as well, but thanks to the quick thinking of Francis Harvey, he managed to shut the blast doors on the turret to prevent flash fires reaching the magazine below.  Although Francis himself perished in the explosion.

Lessons to be learned

It was only a few years ago we were looking at the series of bad judgments that caused the Titanic disaster.  There are a lot of similarities here - when we try and be too clever, we try and twist the logic of reality for our own purposes.

However as Richard Feynman pointed out several times, "nature cannot be fooled".  Your clever arguments hold no weight against pure scientific fact.  I like to think this is really about the core need for testers as well within IT projects - they're the people onboard whose job it is to try and think deep and ask some important "now hey, wait a minute" type of questions.

If every a course of action seems too neat, attractive and instinctual, that's a time to sound off your fallacy alarm, and wonder "okay - what could go wrong".  There's always a drawback, so find it, before it can find you!





Interested in the history of Jutland?

Want to know more, including how despite the British fleet being so badly hammered, it's also considered a strategic British victory?

I highly recommend this documentary on YouTube.

If you live in Wellington, and would like to talk more about the battle and it's history, my wargaming club Wellington Warlords are running the last game recreating the battle from 10am on Saturday 4th June 2016.

Testing for all?

I've talked a little in the past about qualifying as a teacher just after University.  Although I never spent much time in the profession, never the less I wouldn't give up that qualification for anything.  I learned a lot about myself and also about how people worked.

One assignment never feels too far away - we were given an essay to write called "science for all?".  And given the remit to explore diversity and inclusion in how we taught.  Each of us seemed to go our own ways with this, and some of the conversations we had on student teacher pub nights were the most illuminating learning I did on the course. It opened my eyes and my horizons.

I find it sad we have a "professional qualification" in testing which doesn't make us think as much as that single assignment on my teacher training di.  Do use the comments below or tweet to me ( with the hashtag #testingForAll ) to tell me what challenge you think of most when you think of "testing for all?", why is there a problem, and what can be done to cause real change?

Think inside and outside the box.

Memory Lane

I've previously written a lot about memory - but today I'd like to explore a particularly important one from long ago.

Being middle aged, and having moved around a lot, I have a many memories from different places, some going back to my early years.  Mostly they're well behaved and "remain in a state of slumber" ... and sometimes one will need a little jog before it goes back to sleep.

This was the Twitter picture which triggered one of my bittersweet ones,

My wife Collette and I have been together for over 20 years now.  That's so long that it's easy to take for granted.  But a memory of Liverpool is enough to stir those of our early days together.

I always found dating kind of awkward until I met Collette - with her there was never any need for pretense.  In came this woman with an incredible energy and personality to both match and at times complement my own.  We were both incredibly passionate people, although that passion didn't always align, and could cause some rough waters at times.

It was an important relationship - she always challenged me to be more than I could, an odd mix of loving me and pushing me.  But there was always in those early days a bittersweet ritual we had to live out.

Collette's parents were quite traditional, and we never wanted to upset or shock them.  So although she often stayed with me, she could never stay the night, having always to be back home come the morning.

Thus in the early hours, typically around 4am, one of us would wake in the others arms, wrapped together as we were in order to share a single bed.  Driving her home through the dead of night in Liverpool was always such an eerie experience - a city so vibrant during the day, so empty and desolate in those early hours.  The city slept whilst two lovers prepared to farewell each other.

We rode mainly in silence, occasionally Collette ghoulishly pointing out a street corner where a gangland slaying had occurred.  It was all to take our mind off the goodbyes to come.  Alone in the early morning, it felt we wrote our love on the streets, as each red light in our path was an opportunity to hold her soft, warm hand in the bracing morning, or to steal another kiss while we could.

All too soon she would be home, and I'd face the lonely journey home.  All the time thinking what a wonderful thing it would be to be able to wake up in the morning together.  It was like living in a fairy tale - I had someone I loved and who loved me very much, but I always had to give them up before the dawn ...




These days, I like Sundays the most.  Sundays there are typically no alarm clocks, no pressing reason to get out of bed.  Most Sundays you'll find me awake, but lying in bed, next to Collette, and remembering how to my 25 year old self, waking in the morning next to her was all he felt he needed to make him happy.

I think one of the saddest thing about being human is we yearn so much for things, but when we get them, we so rarely get to enjoy them or appreciate them.  All too often one sense of want is replaced with another - we're forever hungry and insatiable for some need.

Sometimes the right memory will help you remember how much what you have now is all you ever wanted.  Find yours, and never take it for granted.