AUTOMATION 27 - Goals, constraints, strategy

Technical level: **

I originally posted my planned curriculum here back in June.  It was based on my understanding of automation at the time.

I've used the series and the interest it's generated to engage with a lot of testers.  I've spoken with a lot of automators around Wellington, and when I've been in conference.  I've collected all their stories, sifted them, and clustered the points they've raised.

A lot of what they've said has been nice affirmations of what I already knew.  But truth is, we engage with others not only to confirm our biases, but hopefully to learn and to stretch ourselves.  Today we're going to revisit some material which probably should go right at the beginning.  Most of it doesn't undermine what I've talked about before, but it does expand on it nicely.

Goal

I talked about this previously here under "automation as a service", but I'm shifting focus a little to really think more firmly about goals.

We're used to people coming up to us when talking about automation and saying "oh my god, there is a great tool, a fantastic tool".  We tend to start with this magic tool that everyone says is so easy to use.

In all my conversations last year I noticed a common theme about people who had stories to tell me.  They were all what I've come to call "second generation automators".

They had a common tale which went something like "we started using this system ... it was record and playback because our testers didn't know how to code".  The rest of their story played out pretty much as one of my denial case studies - they'd invested heavily in the tool, but every time the code changed, all the tests broke, and the maintenance for that was painful.

Inevitably what happened is they had to bin what they'd done to date and start again.  But second time around despite being under incredible pressure to get underway, many of they used the time to plan out much more.

To do this many reflected on their goals.  A common theme of those goals were that they were trying to speed up the feedback process.  If a new build came out, they wanted to find the obvious problems quickly.  These tended to be around making sure they could complete an end-to-end transaction, and that key activities could be completed.

There were differences of opinion about whether the automation was there to evaluate a new build vs doing a deep regression test.  And we'll talk about that more precisely in a later article.

The two key constraints of automation

From their previous failed attempts at automation, there were two key constraints which groups had come to appreciate,

• Automation takes time to do well
• Automation needs some training/expertise to make maintainable

 These were immovable written-in-stone rules, for which they had to build strategies to cope with.  We'll talk more about them below ...


Automation takes time to do well

That record-and-playback tool they'd used before, that could write a script as you tested?  They'd got burned there because every time they're try to rerun it, it'd need a tweak or ten.

Generally you could create an automated script full of checks in the time it'd take you to run about a dozen manually.  If you were going to automate something you had to be sure it was something that had value in checking repeatedly.

Every group used what I call "the strategy of focus" to deal with this constraint.  They knew their automation was going to take a while - they'd never probably automate everything.  What they needed it to do was to check (a) critical elements and (b) anything that too a long time to do manually.

I had a great chat with an automation guru from TradeMe who had this great insight.  If you're working especially on an agile team, you always are working from your most critical elements to your least critical.  It can be a trap for testers to work on new features, and want to automate them.  But in truth the most critical features to your system are those you've already built!

Every group tried in some way to accept that the automation would be an ongoing concern.  They key solution was to prioritise and focus.  They all tried to involve testers in capturing ideas for automated scripts.  One of my favourite tales was of a group which kept a spreadsheet where testers could contribute ideas for script as they came to them, and the whole team would vote to prioritise what to build next.  Democracy in action!

And don't fall into my key trap.  I often find myself going that "everyone" means developers and testers.  Sure testers are a great source of ideas for automation scenarios - we excel at this.  But don't forget particularly to get business people in to help prioritise and come up with ideas.  Occasionally they'll say "this happened once in production", and the thing that happened really scared them, they'd like to know everything has been done to reduce the risk of it ever happening again.  That's the kind of thing that's worth doing.  Not necessarily every bug that made it to production, but the ones that gave the business the jitters.


Automation needs some training/expertise to make maintainable

Again, this came from the sting of "anyone can create scripts using our record-and-playback".  That had got groups into a mess.

There was an embracing of the fact that to write automation, you needed some kind of training and some kinds of coding skills.  But also you didn't want to pick up a tool for which you needed specialist bespoke training - what I like to call "Hogwarts for automation".

Generally there's a desire to use a commonly available language.  If you're a software shop that develops in Java, it probably makes a lot of sense to develop your automation in Java.  You might have some people around who not only can help, but are really good at creating and designing maintainable Java code (you're really hope).

[Side story, once for a customer I had to learn to problem in Lisp to use their automation.  A truly bizarre language, and one of the oldest programming languages still used]

Selenium Webdriver with Java is a really popular combination.  It was one of the reasons that I spun off my Java series.  But it's important to remember this is just a handy GUI automation, as we've talked about previously, finding room for good unit and API testing is really crucial (and we'll talk about this later).

Indeed this was an important takeaway.  Many groups had been lured into their first attempt at automation through the sales pitch of "a tool".  But they'd come to appreciate that automation was not a tool, but a toolbox.  Much like when an engineer comes to your house, she doesn't just bring one spanner, she brings a toolbox!

Unit testing, API testing, units testing are different tools in our automation toolbox, and it's important we learn to use the right one for the right job.  Which leads me to my next post ...




Want to know more?

I'm talking at the upcoming Unicom automation conference in Wellington.  I will be speaking about some of what I've learned not only from my experience, but through going out and talking to people about their automation strategies.  There'll be some other great speakers there as well!