AUTOMATION 3: Automation as a service

To me, this section you're reading is probably THE most important section of the whole series.  So take a moment to let that sink in.

Before we're going to launch into a look at the technology, the how-to's and the tools, we need to take a step back.  Let's take a moment to think of automation as we would any IT deliverable, and start by working out who the users and customers are, and what they expect to get out of the system.

It's a radical way to look at it, but something a previous manager encouraged me to ponder.  When we think of automation our first thought is "which tool is best/everyone else using?", and from then on we just follow so many degrees of wrong.

What I was encouraged to do was to look at automation not as "which tool" but to think of it as a service.  To which the key question is who are the customers, and what do they want?  Who provides the service?

Indeed automation is in many ways a business process for IT companies, and for any such process, the tool is an important part, but it isn't all of it.

What evolved from this, are four key roles within the automation service, which we'll discuss below.

The Manager

I might be a test manager / project manager / business unit manager.
I am driven to keep everything as efficient as possible in terms of resources and cost.
I am not necessarily technical in background.

Driving needs for automation

• I want to use automation so that I can reduce the number of hours my manual testers spend retesting.
• I want to have to spend minimal resources on maintaining the automation going forward so that I’m the most efficient with the projects time and money.
• I want to be able to see a high level status of the automation and when it’s been run so that I know if we have problems.

Questions you can expect from me,

• But isn’t Selenium the same but free?
• How much will it cost?  Is it a one off or ongoing cost?
• What support is there?

 
The Developer

I am focused on creating new software.
I am not necessarily aware of how testing works, and may not be in charge of automating software.

Driving needs for automation

• I want to get fast feedback on any new code I write so that the changes are still fresh in my head.
• I want to be able to see which automated checks have failed, so that I know where I need to address problems.
• I want to be able to look at a log file to understand if a failure is due to product or automation code so that I can be efficient in my work
• I want to be able to look into log files and get enough information about any failure so that I can investigate and fix any issues.
• I want to be able to run automation scripts so that I can check my software and find obvious failures
• I want to be able to run subsets of automated scripts when needed to get fast feedback on fixes and builds

Questions you can expect from me,

• What technologies does it work on?
• What environments can it work from?

The Tester

I am focused on using my time to manually test our product to the best possible standard.
I am not necessarily technical.
I am not seeking to replace all manual testing with automation, but want to use automation to avoid any unnecessary and time consuming checking where possible.

Driving needs from automation

• I want to be able to define new checks which I would like automated to reduce time consuming checking that I would otherwise have to perform again and again.
• I want to implement checks that give me confidence that a feature is behaving in a consistent way as when I initially assessed it.
• I want to be able to see at a high level all the checks that have been run, and their details so I know what the current status of the automation suit is.
• I want to be able to look at any automated check and know what it does so that I can understand what the automation is doing, and hence focus my manual testing elsewhere.
• I want to be able to look at an automated check log and understand that it’s doing so that any failures are meaningful to me.
• I want to be able to run a wide range of automated regression checks so that I cover many areas of testing, to avoid unnecessary manual testing.

Questions you can expect from me

• How do I define an automated check?
• Where can I see my automated check?
• Where can I see if my check passed?
• How do I view the automation steps?

 
The Automator

It is my job to create and maintain the automation framework.
I might also work as a tester or a developer.

Driving needs from automation

• I want to be able to write the code for automated check in a language I am familiar with so that I am comfortable with it, and don’t need additional training
• I want to version control my automation script so I can revert to old versions as required
• I want to be able to view the contents of automated checking logs so I can fix my scripts when things change
• I want to be able to use a modular design in my automated scripts so that why my test code fails due to changes in the product design I can minimise the amount of rework needed in my software
• I want to be able to run my automation scripts so that I know if my automation code works
• I want to be able to look at a log file to understand if a failure is due to product or automation script so that I can be efficient in my work

Questions you can expect from me

• What languages can it work in?

Using the four roles to help review your automation strategy

When setting up automation on a project, as we've discussed it's very easy to start with a trial, and just keep making scripts, and seeing how fast they run.

The four roles help you to review your overall process.  If you have a healthy process, everyone above will have their needs met, and be able to participate and benefit from it.

If however only one of those roles is really benefiting, to the determent of everyone else, then you have major problems, and you need to discuss it as a team, and find a way forward.

Some typical problems might be,

• Only the automator can run automation scripts and view output.  You've created a bottleneck.
• Automators create scripts without consultation with testers.  Testers hence have no idea what the automation regression covers, so have no strategy of working well with the automation.  Hence your hope of automation reducing what testing needs to cover goes out the window.
• Automation is useful when it works.  But when it fails, you really need to be able to see why.