Tuesday, June 14, 2016

AUTOMATION 4 - a zoology of automation

There isn't really "one type" of automation to assist testing.  However when we talk "test automation" most people typically mean "GUI automation".

It's important to learn about the different types, which we should cover by the end of this series.  Using the right automation task for the right job is important - as someone once told me "technically you can use a drill as a hammer ... but it's not a very good hammer".

Today we're just going to focus on a high level description and definition for each.  We'll spend quality time with each before moving on in later pieces.

Unit checking

Our friend Wikipedia says: unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested

That's kind of true, but also a little bit confusing.  I actually don't like confusing, because I will admit, I'm easily confused.

These days, most modern languages allow you to build in some forms of test into your code as you build it.  These tests allow you to check your code at it's most basic level.  It also allows you to run these checks incredibly fast.  There's an example coming up which will really help to illustrate this.

A great example of languages which allow "unit tests" is Java - look up the JUnit test framework, whose checks run as part of the build process (so developers find out really fast if there are problems).

API checking

Wikipedia saysAPI testing is a type of software testing that involves testing application programming interfaces (APIs) directly and as part of integration testing to determine if they meet expectations for functionality, reliability, performance, and security.

Most solutions today are built in layers or components.  They are a mixture of commercial off the shelf components, and bespoke development.  These components typically communicate between themselves using APIs, which are a form of information protocol.

Automation with an API tool allows you to perform checks on a system, by injecting control to a component directly using the API protocols.  Almost as a kind of remote control.  You can also get output via the API to check against.

It also allows you obviously to actually perform some activities in a system where a vital component is missing.  This component could be missing because it's not ready (those rascally developers doing the GUI, when will they learn?) or because it's something you can't simulate in your test environment.

A good example of a tool for API testing is soapUI - which out of the box is actually fairly manual, but mix it with some coding, and you've a powerful platform for API tests

GUI checking

This is the kind of automated check most often thought about when people think automation.

Wikipedia says ... well quite a lot really, and not easy to summarise.  Have a read yourself if you want.

I'll defined a GUI check as one which operates, manipulates and evaluates your web system using the actual front-end of your website.  I would like to add "as if an invisible entity was controlling", but I don't want to get burned for witchcraft.  Although ...

It has to be said, there's something odd when your tool allows you to watch the testing in action - it does look like you're machine is possessed.

Compared to unit or API tests, GUI tests are much slower (not many orders of magnitude faster than when I manually test), and have a tendency to break and need a lot of maintenance.  Small changes to the page design (or worse still, a bad design), will have you constantly tweaking your suite of scripts.

One example of this is of course Selenium IDE which is a record and playback tool.  This means you "hit record" and it learns to repeat back your actions during recording in a very basic XML script.

Such tools can be a danger unless you know what you're doing, because they don't make use of real strengths of a scripted language.  A good example is every time you create an account, you'll use slightly different data as a manual tester, these record and playback tools struggle with this.

Selenium Webdriver is much more powerful tool as it can be controlled by a programming language, taking advantage of many of the flexibility of such languages.  We'll explore later what that opens up for us.

Performance Checking

Before we go it's useful to just cover off performance tools.  Of all the tools, these are the ones I've the least hand-on experience with, but I have managed such testing, and do understand the principles.

In a nutshell they allow you to simulate webtraffic - JMeter for instance will send html requests to your system which simulate certain functions such as "registration", "login" or "transaction", and measures the response time from the system.

The faster / more requests it sends, typically the slower the response from the system.  Typically you're looking to see if it breaks down and become unacceptably long at certain loads.

Check out Oliver Erlewein's articles on Hello Test World for JMeter here.


  1. This comment has been removed by the author.

    1. Thanks Jim - well I learn something new every day.

      I chose zoology as I feel (certainly in what's to follow) we'll be studying them as different beasts. But yeah, this is probably closer to a taxonomy.

  2. For API testing, I like Postman, or just using cURL. Isn't SoapUI more for soap services testing (I know almost nothing about soap)?

    I find the test automation pyramid (originated by Mike Cohn, here's one of his posts: http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid) helpful when thinking about these kinds of automated checks.

    I like taxonomies like this. Also models like the pyramid and the agile testing quadrants. Take this list to your whole delivery team and talk about what automated regression checks that you need for your product.

    1. I'm doing a broad view here, and we're going to look a little closer at each. The pyramid is something I want to cover, it ties in with one of my iron rules. But I want to do so once we've explored these test types a bit more - ie SPOILERS! ;-)

    2. Don't take the Automation Pyramid as gospel. The way it is shaped comes from natural process and not some pre-planned methodology. Of course the base will be wider because of the nature and type of automated test (checks or whatever, splitting hairs in my opinion and remember what I said about splitters and clumpers) they are. Unit Tests are by nature singular in purpose and focus. Because of that you have a one-to-one relationship with the code itself and thus you will naturally produce more of them (thus the wider base). They are more atomic. Finally, these types of tests validate single conditions. Benefit of all of this is that a developer is testing their own code for correctness and robustness. It helps to eliminate "stupid" bugs in code sooner and alleviate the headaches testers later in the process incur when sloppy code is delivered. The next layer up is an integration layers where it is testing whole pieces of functionality and interactions with them. Not as atomic, but important because you need to know function and other programming interfaces are sound. This can be functions within the app, or functions between apps. This helps with proving robustness and reliability of the code. Finally at the top are Acceptance Tests (which can be UI or API/Services layer too). These are more process driven tests and if the other layers are in place you don't have to build as many as the other layers to test the software overall. You focus on proving things at the macro level.

      So if you are following a test driven development approach you will naturally build out your automation approach like the pyramid. I tend to think of it all as concentric circles of coverage, you build upon the other to provide the coverage of testing that you need.


  3. Ass Wr Wb Saya ingin berbagi cerita kepada anda bahwa sy seorang TKW dari malaysia dan secara tidak sengaja saya buka internet dan saya melihat komentar IBU LUSI yg dari singapura tentang AKI yg telah membantu dia menjadi sukses dan akhirnya saya juga mencoba menghubungi beliau dan alhamdulillah beliau mau membantu saya untuk memberikan nomor Togel/lottrey 4D dr hasil ritual/ghaib dan alhamdulillah itu betul-betul terbukti tembus dan menang RM.270.000 Ringgit ,kini saya kembali indon membeli rumah dan kereta walaupun sy cuma pembantu rumah tanggah di selangor malaysia, sy sangat berterimakasih kepada AKI MAULANA dan tidak lupa mengucap syukur kepada ALLAH karna melalui AKI saya juga sudah bisa se sukses ini, pesan AKI yg slalu sy ingat setiap manusia bisa menjadi kaya, hanya saja terkadang mereka tidak tahu atau salah jalan, Banyak orang menganggap bahwa miskin dan kaya merupakan bagian dari takdir, Takdir macam apa? Tuhan tidak akan memberikan takdir yang buruk terhadap kita semua, cobaan yang Tuhan berikan merupakan pembuktian seberapa kuat Anda bertahan di dalamnya. Tuhan tidak akan merubah nasib Anda jika Anda tidak berusaha untuk merubahnya sendiri, Jadi teman2 yg dalam ke susahan jgn pernah putus asah, kalau sudah waktunya tuhan pasti kasi jalan asal anda mau berusaha. AKI MAULANA adalah guru spiritual terkenal di indonesia yg bisa melakukan ritual ghaib seperti:
    1.Pesugihan bank ghaib
    2.Ritual tembus togel/lottrey
    3.Transfer janin
    4.Pelaris usaha, jodoh DLL
    jika anda ingin mengubah nasib seperti saya silahkan KLIK DISINI PESUGIHAN DUNIA GHAIB


    I want to share a story to you that sy a migrant worker from malaysia and accidentally I open internet and I see the comment LUSI mom from singapore about AKI who has helped him become successful and finally I also tried to contact him and alhamdulillah he wanted Help me to give the number Togel / lottrey 4D dr ritual / unseen and alhamdulillah it really proved translucent and won RM.270.000 Ringgit, now I am back indon buying house and train even though sy just housemaid in selangor malaysia, sy very thankful To AKI MAULANA and do not forget to give thanks to ALLAH because through AKI I also can be this successful, AKI message which slalu sy remember every human being can become rich, only sometimes they do not know or wrong way, Many people assume that poor and rich Is a part of destiny, what kind of fate? God will not give a bad destiny to us all, God's temptation is proving how strongly you endure it.God will not change your fate if you do not try to change it yourself, So teman2 deep into tusah jgn never despair, when it's time god must be the road to where you want to try.AKI MAULANA is a famous spiritual teacher in Indonesia who can perform the occult rituals such as:
    1.Presugihan bank ghaib
    2.Ritual translucent togel / lottrey
    3.Transfer fetus
    4.Paris business, matching DLL
     If you want to change the fate Please visit the website CLICK HERE WORLD WIDE GHAIB