Experiences in automation ... WeTestWorkshop

I have never lived anywhere quite like Wellington. The thing that constantly amazes me about this place is the sense of community amongst technical folk, aided by the various meetups which are organised by people who are passionate about their relative crafts.

I'm already a regular face at the AgileWelly events that are organised. I was thus noticably pleased when following KWST2, several testers decided we needed a more regular workshop, and thus the WeTest workshop was born.

This Thursday was the first event, sponsored by the consultancy I used to work for, Assurity. Katrina Edgar (the organiser behind the whole WeTest meetup) led with an experience report on the subject of test automation. It was a potentially tough crowd – about half the room were ex-developers turned testers, and 75% had experienced test automation in the last two years.

Katrina, like myself, was an ex-developer turned test automation specialist. She'd encountered 3 major projects so far in her career ...

On Project A, she was allowed to branch out into automation on a typically very manual task. She'd been given enough free reign to choose whether to automate or not. She used it to automate the very onerous task of setting up multiple configurations, which removed a lot of monotony from her tasks. The work was relatively low cost, and reaped high benefit. But that said, she felt just left to get on with in, and her automation was expressly that, HERS. It was never reviewed, and no-one else ever used.

Project B was more advanced. She came onboard and there was an automation framework already in place – a Jenkins machine which provided continuous integration tests using Selenium scripts. She felt where as Project A was more about “what are you testing” with the free reign on “how are you testing”, life here was quite the reverse. Everything was about “how can we test on Jenkins”.

The system had a Concordian front end to show whether tests had passed or not, and it was considered that if the test passed, it was “all good”. There were no follow-on manual tests, because it was believed “well our Selenium scripts test it all, don't they?”.

The problem was that the testing was highly technical, and to understand the script content, you had to be able to read Java. This meant not enough people did read through and understand the scripts and what they did. Testers would modify and make new scripts, but no-one ever reviewed them. So on the whole no-one could be sure whether these tests did test the core values of the project.

Project C echoed a lot of Project B. It was a similar system where everything had been done by automation. But it was a much older, legacy system, and all the original staff, and much of the expertise had moved on.

Thus the scripts were flaky, and needed a lot of maintenance by people who didn't really understand them all. A lot of time was spent fixing them, but no-one knew everything they did. But they'd always seemed to work before, so no-one wanted to tamper with them too much either.

Her experience report finished, the discussions around the room began. And this is where a peer-conference drastically differs from presentation-based events. It's much more interactive with many joining in, asking questions, sharing tales. Whereas in a presentation you walk out with on persons experience and ideas, at the end of a peer conference, you've had those ideas pooled together by everyone in the room.

Thus by the end of the 2 hours, we'd investigated and reached consensus in a way which was surprising to both myself and Katrina. In fact no-one could have predicted it – which is what can make these things so exciting. These were some of our take-homes by the end of the night …

Know why you are automating

Automation should be almost always about addressing a pain if you tried to do something manually. In particular it should use some strength of the computer against an area where a human being is much weaker and slower.

Here are some areas in which computer excel over humans, especially in a “testing” capacity (I will explain the use of quote marks later),

• they are much faster
• they can do the same task over and over again with no variance
• always do exactly what they're told

On the flip side, this is what an experienced human being can do which computers can't,

• does not need a script to start using the software
• uses software in the same way as an end user (if a human is going to use your end-product, you need a human opinion on it during testing)
• can investigate as they test
• can notice things in software even if they're not on the checklist

To make efficient use of automation (and thus get a return of investment on the time you spend automating), you need to be addressing a pain point in your testing, and you need to be doing something in your automation that computers can do well (from the first list) rather than something that humans do well. It also needs to be doing something that you're likely to do again and again - so once scripted, it'll save you time every time it's run.

If you're Agile, and 3 days of every sprint is taken with your testers running repetitious regression tests on a mathematical function, this is the kind of pain point you can and should automate to free up some of those 3 days of testing effort.

Know what you should be automating

When test automation was first introduced in the 1990s there was a belief that many test departments should have suites of 100% automation. Experiences of the last decade have really challenged that idea.

Test automation has a place, but it's not the alpha and omega of testing. In fact many like Michael Bolton believe automation tools should be called automated checkers over automated testers (hence the quotation marks before).

The reason for this is an automated script can only check for what it's scripted to check for. It will never tell you if a screen flashes pink and yellow unless you tell it to check for that. It will never notice the kinds of things that a human tester will go “well is it supposed to be like that?” where something is not necessarily against a requirement, but not quite right either.

The cyborg tester

I've heard the concept of the cyborg tester before, and this is what started to come out from people's experience with automation. I believe I've heard it from people like James Bach and Ben Simo on Twitter – the idea is that testing isn't about doing testing all by humans, and not by doing testing all by machines.

The cyborg tester is a fusion of both man and machine, using both blended together to produce superior testing.

Automated checks are fast, repeatable, and if done right, anyone can push “play”. But they miss a lot of defects a tester would find. They are best used essentially for unit testing between building a piece of code and giving it to a human tester.

We've all had scenarios where developers have delivered new builds daily – when asked if it passed testing, you are greeted with “well it built” (which does not mean it passed any kind of test). The test team start to use it, and there are major issues, with elementary functionality failing. That means anywhere from a half day to a full day of testing is lost because we have a bad build, and no capacity in some systems to rollback to a previously working build.

How much better then to include those kind of smoke checks as part of the build process, and if the software doesn't pass those checks, it's not deployed? Such a policy follows the “test early” philosophy, and means manual testers are protected from bad builds which are so fundamentally flawed it would force them to down tools until addressed. [A working old build allows more investigation than a new, broken one]

Such a system is one of synergy, allowing testers to continue investigating on a previously stable build until a useful new build with basic core functionality can be delivered.

Automation TLC

As alluded to by Katrina, and in line with comments I've had from Janet Gregory, the automation you are doing needs to be clear and visible to the whole project. Everyone should be encouraged to look at what you are doing, review, and give feedback, especially as to whether or not it addresses business and technical needs enough.

How can you be sure your automation really addresses the core business values of your end-product? You need that feedback to target the important stuff, and cut away anything which doesn't add value (otherwise you waste time running it, you waste money maintaining it).

But more than that, many places will automate phase one of a project and like Katrinas Project B and C, will say, “we're done”. Automation isn't a “we're done” thing. Automation in an ongoing commitment.

Every time you make changes to your code base, you need to have someone looking at your automation scripts and going “does anything here need to change”. That's how you keep your automation working and relevant. If you develop for a year, and only then start to use the scripts again, you might have a nasty shock (much like Project C) where nothing seems to work any more. You might even be tempted to bin it all and start again. At this point, the automation which was there to remove your pain in testing, actually becomes your point of pain!

But more than just making sure you have resources to maintain scripts, you have to ensure your scripts are maintainable. In software code, good practices are to have commenting within code to say what each pieces intent is, peer reviews of code and to even have coding standards of things you should try to avoid (forever loops anyone?). Being an ex-developer myself, these are things I encourage in any test automation project. Going around the WeTest workshop, it became clear I was not alone.

When can we get time to automate?

This was the final question of the night I was involved in (alas I had to leave for a train at this point).

But the comment would be one many would be familiar with, “we're going to flat out with our manual testing, we're having trouble creating our base of automation scripts”.

It's tempting to think it's possible to go flat out on a project and also be able to do process improvement as you go along. Sure you can make small improvements. But to achieve significant benefits you need to allocate effort, because you need to concentrate on that task, not run it in spare time (for more on this subject read this article).

If you are finding you have too much testing to do, and find it harder and harder to achieve deadlines you need to go back to our first point, look for the areas of pain that are taking time, and see if automation will help. You might have to say to your Project Manager “look we need to pause for a bit to get some breathing room here” and perhaps extend timelines or investigate other options – it's possible development need to pause to do some cleanup themselves. But you don't need forever, you just need enough automation to ease your painpoints, and then enough resource to keep it maintained when needed.

Overview

A fantastic first meeting, and looking forward to future – thanks to all those who turned up and made this such a memorable workshop! I certainly came away with a lot to think about, and have enjoyed revisiting it here ...

The following photos are taken from the WeTest site ...