Learning to use exploratory testing in your organisation ...

We have a certain view of test practises which have been passed to us from the waterfall project roots we've all had some experience of.  In many ways, these methods are familiar, especially to non-testers, and from this familiarity we feel a measure of comfort and security which sadly can often be unwarranted.

When veterans of waterfall such as myself are first faced with some of the concepts of exploratory testing our reactions can range from shock to abject horror.  Almost always though the initial reaction can be a negative one.

This article will help to define exploratory testing, de-construct what we do in scripted waterfall delivery testing, and talk about how we provide the same essential value with exploratory testing.  It will cover our approach to using exploratory testing as part of a system of testing, although of course other approaches can vary.  This is how we've found to get not only the best results, but the best buy in and turned our detractors into our allies.

Back when I did a series of articles on Building A Testing Culture for Testing Circus based on conversations I was having with other software testers, a major change and indeed the future of software testing was seen to be in the hands of increased use of exploratory testing.  And yet the idea isn't new - it was first coined by Cem Kaner in the 80s.  So what is it, and why is it becoming increasingly applicable to the way we're running software projects?

A few homegrown definitions

Let’s start by making a few definitions before we discover further!  These are the definitions I'm comfortable with, but you may find definitions will vary.

A test script is a set of instructions and expectations for actions on a system which a user then confirms when executing testing

Historically I've seen many a project where such scripting has meant we’d define every action of every button press, and detail every response.

Typically in waterfall, we would have a good amount of time to test … or rather we would ask for it, and whether we get it or not would be another tale.  

A product definition would come out of requirements and design, and whilst everyone starts to program against these, testers start to produce plans of what is to be tested, coupled with defining scripts for the new functionality based on these definitions.

Once done these plans and scripts are ideally put out for review – if you’re lucky you'd get feedback on them, which means you expand or reduce your coverage to include the areas which are key to the product.

When we run our scripts, typically life couldn't be simpler, all we do is go through, try each step, and tick or cross as we go to “prove” we've tested.

The problem with scripts is that they’re very brittle,

• they require a lot of time upfront (before the tester has access to a product) to scope to the level of detail discussed above.
• they are fallable in the fact that just as developers are not immune to misinterpreting product definition neither are testers.  And that means large amounts of rework.
• the requirements and design need to be static during the scripting and execution phase.  Otherwise again, you have a lot of rework, and are always “running at a loss” because you have to update your scripts before you’re allowed to test.
•  they depend on the testers imagination to test a product before they've actually seen a working version of it.

Unfortunately talk to most testers today, and they’ll report to you these are areas they’re under greatest duress.  The two biggest pressures they feel revolve around timescales being compressed and requirements being modified as a product being delivered.

Some testers have a desire to “lock down the requirements” to allow them to script in peace.  But obviously that involves to some extent locking out the customer, and although it’s something that we might feel is a great idea, it’s important to have an engaged customer who feels comfortable that they've not been locked out of the build process to have a successful project.

So testers have to be careful about not wanting to have a model for testing which works brilliantly in theory, but breaks down because of real and pragmatic forces on their project.

Exploratory testing has multiple benefits – one of the greatest being that it doesn't lock down your testing into the forms of tests you can imagine before you've even had “first contact” with a prototype version of software.

Exploratory testing is a method of exploring software without following a script.  There are many ways to perform it – the best parallel I have found is with scientific investigation.  You set out with an intention, and you try experiments which touch the area, devising new experiments as you go along and discover.

With exploratory testing there is more value in noting what you've actually done over recording your intentions.  Compare this with scripted testing, where you put the effort in ahead of time, and all you record during your testing is either a tick for a pass, or cross for a fail!

A test session is a high level set of behaviour that the tester should exercise when executing testing.  Often this can be a list of planned scenarios we think would be good to exercise.  But unlike a script, it's a set of things to try in testing, without getting bogged down early on with the step-by-step instructions of how that will be done.  It also is more a set of suggestions, with room for additional ideas to be added.

So what is exploratory testing?

There are many definitions out there for exploratory testing, and I'm going to add my own understanding of it.

Exploratory testing to my team is about

• building up an understanding first of what the core values and behaviour of a system is (often through some form of Oracle)
• using that understanding to try out strategic behaviour in the system to determine whether what we witness is unexpected

Oracles

An oracle is a guide for understanding how the system’s supposed to behave.  An obvious one we all know is a requirement document functions as an important oracle.  However it’s not the only oracle there is.

Recently when we were making changes to our registration page, we took a look at registration pages on Gmail, Hotmail, Facebook, Twitter.  Many of these had some of the features we used in our registration page, so it gave us a method of playing and getting familiarity with.

Especially if you’re producing a system that’s got a broad customer base, most users don’t have the advantage of reading a whole stack of requirements when they use your system.  Your product has to make sense, especially given similar products in the market.

This ability for a tester to look at a page and ask “does it make sense” is an important freedom that’s required in exploratory testing.  Sometimes saying “as per requirement” isn't enough, and we have to push further.

Recently we put together a system to allow a user to manage their own password reset when they’d forgotten their password.  All the required behaviour was put in front of a lot of people who signed off on what they wanted.  The requirement we had read that the email the user would receive would say “your account has been unlocked, and you can no login”, just as per requirement.  My tester dared to suggest that “you can now login” would perhaps make more sense, going beyond just taking the requirements to be the holy oracle of all truth, and using a bit of common sense.

Somehow that typo had got through a lot of people - they did indeed want "you can now login" - but then, that's the nature of testing.  I've seen much larger slips than that pass by uncommented before now ...

Example

You’re testing a login page, the requirement says, “When the user provides the correct password and username, the user is logged in”.

Within exploratory testing, it’s an expectation that the tester will be able to expand on this, using their experience with similar models,

This is a list of a tester using an Oracle based understanding and going beyond just the requirements to plan testing.  They'll typically expect,

• If I give an incorrect password for a username, I won’t be logged in
• If I give a correct password for the wrong username, I won’t be logged in
• On other systems the username I use isn't case sensitive – should it be here?
• On other systems the password I provide is case sensitive – should it be here?
• What should happen when I try to log in incorrectly too many time?  Does the account get locked or should it be locked for a period of time?

Skill Based Testing

This ability to think beyond just a one sentence requirement is part of the inherent skill which is a core need for exploratory testers, it calls for,

• An understanding of what the systems supposed to do.  Not just functionality, but the “business problem” it’s trying to address.
•  An understanding of how the system has worked thus far is helpful
•  An understanding of similar behaviour in other products

Exploratory testing is thus often referred to as “skills based testing”.

An argument often used in support of scripted testing over exploratory testing is that “if you have your testing all scripted up – then anyone can do it”.  Indeed such scripts it’s possible to provide to almost anyone, and they can probably follow them.

The problem tends to be that different people will interpret scripts differently.  Most inexperienced testers will tend to follow an instruction, and if what they see on screen matches, they'll just tick a box, without thinking outside of it (and hence would be quite happy with "you can no log in" example above because it was what the requirement said).

Lets explore this with another example ...

Test script

Action	Expectation	Pass/Fail
Click the link to take you to the login screen.	There are two fields for entry, Your account Password

The Test System

Here are 3 screens for the above script ... do they pass or fail?

Can you be sure that every tester would pick up on that from the script?  If so would it be a simple cosmetic defect, yes?

Sadly some people will combat this by trying to turn this into a version of the evil wish game, where you go overboard on explaining what you expect, so there' no room for ambiguity.  Hence an obsessed scripted tester might try and write the expectations as,

Action	Expectation	Pass/Fail
Click the link to take you to the login screen.	There are two fields for entry, Your account Password The text on the display is  all in Calibri font 11 all UPPER CASE black colouring is used for all text

Yes sirree - that will stop those 3 problems above occurring again, but it won't stop other issues.  Especially if it turns out the root cause for the problems is a disgruntled programmer who's tired of reading your scripts, and currently serving out their notice period until they move to another company where they do exploratory testing ...

"But our scripts are our training!"

This is a common objection to exploratory testing.  If you have detailed scripts, anyone can follow them, learn the system and be a tester right?

Most people I have spoken with have found that it’s far easier to have a handbook or guidebook to a product, which details at a light level how to do basic flows for the system, than to have high level of detail throughout your testing.

The counterargument to handbooks is that “anyone can follow a script” but as discussed previously to build up familiarity with the system and the values, you will always need some training time to get that.  If you just throw people at the “your account” problem you’ll get differing results, you need people to be able to be observant and in tune with the system, and that doesn't tend to happen when you just throw people blind at a system with only scripts to guide them.

The bottom line is that there are no shortcuts to training and supporting people when they’re new testers on their system.  If you’re going to train them badly, you’re going to get bad testing. 

One of the few exclusions we've found to this is areas which are technically very difficult - there are in our test system some unusual test exceptions we like to run, and to set up these scenarios is complex, and completely unintuitive (includes editing a cookie, turning off part of the system etc).  This is one of the few areas where we've voted to keep and maintain scripts (for now), because we see real value in the area, and with maintaining so few other test scripts, we can do a good job, without it chewing up too much time.  

But we've found we certainly don't need test scripts to tell us to login, to register an account etc - when the bottom line is our end users only get the instructions on the screen to guide them.  Why should a one off user not need a script to register, and yet testers like ourselves who are using the system every day require a script to remind us that if we enter the right combination of username and password, we're logged in?

Debunking the myths

A common complaint about exploratory testing is that it’s an ad-hoc chaotic bug bash chase.  You tell a team to “get exploratory testing”, and they’ll run around like the Keystone Cops, chasing the same defect, whilst leaving large areas of functionality untouched.

This is some people's view of exploratory testing

With such a view of exploratory testing, it’s no wonder a lot of business owners see it as a very risky strategy.  However, it’s also misleading.

Whilst such ad-hoc testing can be referred to as exploratory testing – it’s not what many people’s experience of exploratory testing is like.

Just because exploratory testing doesn't involve huge amounts of pre-scripting, doesn't mean that exploratory testing is devoid of ANY form of pre-preparation and planning.

You will often hear the words “session based” or “testing charter” being referred to – these are a way at looking at the whole system and finding areas it’s worth investigating and testing.  The idea is that cover your sessions, and you will cover the key functionality of the system as a whole.

Drawing up a map of sessions

Creating a map of sessions is actually a little harder than you’d first think.  It involves having a good understanding of your product and it’s key business areas.

Let’s pretend we’re working for Amazon, and you want to derive a number of sessions.  Two key sessions jump out at you right away,

• Be able as a customer to search through products and add items to the basket
• Be able to complete my payment transaction for my basket items

Beyond that, you’ll probably want the following additional sessions for the customer experience,

• Be able to create a new account
• Be able to view and modify the details of an existing user account
• Give feedback on items you ordered, including raising a dispute

Finally obviously everything can’t be usre driven so there is probably,

• Warehouse user to give dispatch email when shipment sent
• Helpdesk admin user to review issues with accounts – including ability to close accounts and give refunds

The trick at this point is to brainstorm to find the high level themes to the product.  Ideally each session has a set of actions at a high level that really encapsulates what you’re trying to achieve.

For more on mind mapping (I tend to be very much list driven in thinking), and I recommend Aaron Hodders notes on the subject.

Fleshing out the planned test session

For many skilled testers, just giving them the brief “be able to create a new account” will be enough, but perhaps a lot of detail came out of your brainstorming that you want to capture and ensure is there as a guideline for testing.

Let’s take the “be able to create a new account”, here are some obvious things you’d expect,

• Can create an account
• Email needs not to have previously been used
• Checking of email not dependent on case entered
• Password

o   Must be provided

o   Has a minimum/maximum length

o   Must have a minimum of 2 special characters

• Sends email recript

A session can be provided in any form you find useful – mind mapped, bullet point list, even put into an Excel sheet as a traceable test matrix (if you absolutely must).  Whatever you find the most useful.

Such bullet pointed lists should provide a framework for testing, but there should always be room for testers to explore beyond these points – they’re guidelines only.

Keeping it small

Typically not all test sessions are equal – the test sessions for “adding items to basket”, if you've got over 10,000 items to choose from (and no, you’re not going to test all of them) is obviously a bigger task than creating an account.  However if some sessions are too big, you might want to split them up – so you might want to for instance split “adding items to basket” to,

• Searching for items
• Retrieving details for items
• Adding items to basket

But really breaking it down is more an “art” than a set of rules, as you’d expect.

Test Ideas For Your Session

Okay, so you have a session, which includes some notes on things to test.  Hopefully with your experience in the project, and test intuition you have some ideas for things to try out in this area.

A common thing testers use to explore and test is called heuristics, which are testing methods.  Elisabeth Hendrickson has a whole list on her Heuristic Cheat Sheet.

But here are a few of those which should jump out at you,

• Boundary tests – above, on the limit, below
• Try entering too much/too little data 
• Use invalid dates
• Use special characters within fields
• Can a record be created, retrieved, updated, deleted

Using heuristics, a simple statement like “can I create an account” can be expanded to be as large/small as needed according to the time you have.

Working through a session

In running and recording test sessions we've gone right back to University.  For those of us who did science classes, an important part of our lab experience was our lab book.  We used this to keep notes and results of experiments as we tried them out.

They weren't wonderfully neat, and we’d often change and vary our method as we went along.  But we could refer back to them when we needed to consult with other students or our tutor.

For our group, these notes are an important deliverable as we move away from scripted testing.  Typically we’d deliver our scripts to show what we intended to test.  Now when required we deliver our notes instead showing what we tested (over intended to test).  Not every exploratory testing group needs to do this, but to us, it’s an important stage in building the trust.

Obviously a key part of this involves “just how much detail to put in your notes”.  And this is a good point.  The main thing you’re trying to achieve is to try our as many scenarios as possible on your system under test.

We've found a table in simple Microsoft Word is the most comfortable to use (you can make rows bigger as you need to put more details in).  Basically you put in your notes what your intentions are.

We couple this with using a tool to record our actions when we test.  Each row in our notes table covers about a 10-15 minute block of testing we've performed.  We use qTrace/qTest to record our actions and the system responses during this time.

This allows us to use our notes as a high level index into the recordings of our test sessions.  Obviously things like usable names for our recordings help – we go by date executed, but anything will do.

A screenshot from a typical session log

It should be noted of course that not all exploratory testing needs to be recorded to this detail.  

For my group, as we move away from scripted testing, this approach allows us to build up trust in the new system. The bulk of this documentation – screenshotting, and a button-by-button trace of what we do – we find qTrace invaluable in absorbing the load for us (it would slow us down like crazy otherwise).

Of course, any test we find ourselves doing again and again is a candidate for automation – and even here, if we have a suitable flow recorded, we can give a qTrace log to our developers to automate the actions for us.

Closing a session

When a tester feels that they've come to the end of a session, they take their notes to another tester, and discuss what they've tried, and any problems they've encountered.  The peer reviewer might suggest adding a few more tests, or if there have been a lot of bugs found, it might be worth extending the session a bit.

This peer reviewing is essential, and we encourage it to lead to some peer testing.  Basically no matter what your experience, there’s no idea that you can have that can’t be enhanced by getting the input from another individual who understands your field.

This is why for us, the method we've used for exploratory testing really confounds the standard criticism that “when you write scripts, they’re always reviewed” (even this is rapidly becoming the unicorn of software testing, with many scripts being run in a V0.1 draft state). 

In both using reviewing (through brainstorming) of when we plan our test sessions and also using a level of reviewing when we close a single session, we’re ensuring there’s a “second level of authentication”tof what we do.  Sometimes we wonder if our process is too heavy, but right now an important thing is that the reviewing, being a verbal dialogue, is fairly lightweight (and has a fast feedback loop), and both testers get something out of it.  Being able to just verbally justify the testing you've done to another tester is a key skill that we all need to develop.

Peer Testing

We try and use some peer testing sessions whenever we can.  I've found it invaluable to pair up especially with people outside the test team – because this is when the testing you perform can take “another set of values”. 

Back at Kiwibank, I would often pair up with internal admin end users within the bank to showcase and trial new systems.  I would have the understanding of the new functionality, they would have a comprehensive understanding of the business area and day-to-day mechanics of the systems target area.  It would mean together we could try out and find things we’d not individually think to do.  Oh, and having two pairs of eyes always helped.

Pairing up with a business analyst, programmer or customer is always useful, it allows you to talk about your testing approach, as well as get from them how they view the system.  There might be important things the business analyst was told at a meeting which to your eyes doesn't seem that key.  All this helps.

It’s important as well, to try and “give up” on control, to step aside from them machine, and “let them drive and explore a while”.  We often pair up on a testers machine, and etiquette dictates that if it’s your machine, you have to drive.  It’s something we need to let go of, with a lot of the other neuroses we've picked up around testing …

References

I don’t usually include reference to my blog pieces, but this piece has been a real group effort, and our understanding in this area has evolved thanks to the interaction and thought leadership from a number of individuals beyond myself.

Instrumental amongst these have been

• The Rapid Software Testing course from James Bach which really focuses on the weaknesses of scripting and adoption of exploratory testing, and is a road-to-Damascus, life-changing course I would recommend to any tester.
• The book Explore It by Elisabeth Hendrickson
• The many verbal jousting sessions I've enjoyed at WeTest
• My engaged team of testers at Datacom

The Power Of Pairing

In my previous piece on exploration, I touched upon the power of pairs, but "not by those words" (to quote James Bach).  I was reminded again of the theme this weekend, due to the following interchange ...

I'm a member of my local Les Mills gym.  Without doubt one of my favourite classes is Impact, a boxing themed class where we put on our gloves, and hit various punch bags and pads.  With a few bits of skipping, squats, press-ups and burpees thrown into the mix.  It's a very fun but challenging class, with a great atmosphere.

After class this week, I stumbled upon a History In Pictures photo on Twitter, and shared it with my Impact instructor Josette.

It's a picture of some women training on a rooftop in the 1930s.  I didn't notice much except that,

• It's black and white, and the kind of grainy quality of a 1930s photo
• The cars in the background look the right period
• Clothes and hairstyles seem period
• The girls all have really rather nice legs (I know, potentially a bit sexist, but the honest truth)

Have a look again?  Notice anything?

From my own knowledge, I know that for this historical period women's professional boxing would be rather unusual.  Only relatively recently have women been taken seriously in boxing - as Josette of course knows.

Josette loved the photo, but had an observation to make ...

"I wonder if they were part of a circus. Just the accessories on the floor for juggling and something that resembles a small trampoline."

My reaction to this was "huh?  What accessories?".  Darn that I was too distracted by legs to notice, but yes indeed, there they are!  Once you see that, your understanding of what you're seeing changes.  More than likely this isn't sparring proper, but more than likely practicing some form of staged/faked fight for entertainment (and probably titillation of a male targeted audience ... so maybe the legs ARE important).

So what's it all about?

We all know that two minds are better than one.  Likewise two sets of eyes.  Indeed the very existence of testing as a profession has come from people finding that we're more likely to find issues in software if someone writes the program, and a second person tests the software.  This studies have shown time and again this gets better results than just a single developer who "programs and tests".  That's because the developer who tests his own code uses a mental model of how the software should work to build it, but then use exactly the same model to test it.  We don't check in any way we've not previously expected, and that leaves us blind.

There are lots of pieces out there about the impact of inattentional blindness in software testing, especially from context driven testers.  To put it simply, we can get hung up looking for one thing, and miss something else that's under our nose.

Although here I joke about the girl's legs, I was too busy looking for clues to the pictures period (rooftops without aerials, cars, picture style, clothes hair) that I actually missed a lot of key things in it - namely the trampoline and various juggling items.

When we were out exploring Baring Hill, I noticed some things about the site, but my son had to bring my focus down to the pipes in the ground for me to notice them and follow them and use them to tell a story.

This reminds me of something I was told years ago when we were giving a statement about a traffic incident to a police interviewer.  The police try and gather eye witness statements from as many sources to piece together a picture of what's happened in any incident - usually everyone recounts the same events based on their perception, but not everyone tells it the same way, with some adding details that others fail to mention as relevant.  One baffling example was a case she'd been a part of where several people had independently described a suspect to her, but the eighth person mentioned "and he was blind".  In actual fact, the previous seven had forgotten to mention this key detail, although later all went "oh yes - he WAS blind".  Sometimes we're so focused (which is the same as saying distracted) on the little details - hair, shoes, clothes - we miss the big stuff.

In all these stories, having an extra perspective allowed the person to question or state things, to bring them to the table for discussion, and thus expand on one persons perspective.

This is the power of pairing.

Who to pair with?

Most people when they talk about pair testing usually mean testing between two peer testers.  I've not always had the luxury of another tester, especially when I'm working on small projects.

Sometimes I've paired up with another tester outside of my project to talk about my approach and bounce ideas off - it's always been useful for me to talk and justify my approach with someone I trust, and I often get ideas regarding fine tuning my focus.

There are a mix of people on a project though who you can get access to, and doing paired sessions with these people can be a great ways to exchange knowledge and values with.  Pairing up with these people for brief testing sessions where you demonstrate the system to them can be an enriching experience.

Here are some typical people you can look to pair up with, and what they can offer you,

Admin users - they may be in-house or customers.  Especially on legacy projects where you hear the words "not everything is documented", these people are a Godsend.  They usually belong to cultures where their knowledge is passed on verbally admin-to-admin as they are mentored in the system - often they know the obscure business processes that make the current system tick.  Spending time with them allows you to understand the key processes which keep the system ticking over.  Likewise when you have new functionality, they can really help to zero your time in on the pieces around it that would impact the business if they didn't work.  This can help you to focus your testing time on high value areas.

Business Analysts - they are quite literally living requirement oracles.  Except having been on the meetings that shaped the requirements, they understand the "whys" regarding the importance of key features. You'll learn a lot about your customer through them, "oh Karen asked for this because currently when they do X, they have to do Y first to activate it, and it takes them extra time to do so".

Marketing - an interactive session with marketeers helps them to understand how the system works, which helps them tighten down on how to promote features, as well as think about future enhancement.

Project Managers - a session with them will often help to shake out key concerns from the steering committee they report to, including potential horror stories from the past.  "Mark is worried that X might happen" or "last year they had problems when Y happened and brought down the system".  This allows you to be aware of areas where there are concerns, and make sure you're comfortable about how your testing is addressing them.

Junior Testers - paired sessions are a great way to coach junior testers.  Putting them in the driving seat gives them the opportunity to take the initiative, and to explain their approach.  It allows you the opportunity to suggest additional tests they might want to consider, or give feedback about what tests might be more important to run first.

What do they get from it?

Sharing is a two way experience.  Whilst they share these values with you, you're sharing yours with them.  Namely,

• Demonstrating what you're testing, especially new functionality
• Showing the methodology you use to test features in the system
• For admin users - helping to train them on how to use new features
• For business analysts and marketing - demonstrating how their requirements and requests Frwere realised
• For project managers - feeling engaged and involved in the testing being performed
• For junior testers - coaching

How do I know if I'm pairing or doing a demo?

This is an important distinction to be aware of.  If you're sitting with someone, and one of you is controlling all the conversation for the whole session, then you are not pairing.

Pairing is an interactive partnership.  There is a certain level of inquiry and challenge, without feeling of threat or accusation.  This is important - another party is essentially reviewing your testing as you perform it, and it's important to respond to these comments.  Some of this feedback may suggest other tests, which if feasible you should attempt to include.  [Sometimes if your session has a goal, and the suggestion would take you off topic, you might want to leave it until the end though or schedule another session]

If you're exercising your testing, or guiding your pairee and they simply don't respond, then you're not pairing - you are essentially demoing.

Here are some potential areas of challenge - notice that most of them suggest areas for extra tests, which if easy to accommodate you should include,

• "You used a keyboard shortcut - I didn't know that existed - we're told to use the icon to launch".  Extra test - show them the system launched via the icon, and teach them the shortcut.
• "The server admin machine is ancient and still has a Netscape browser, did you check this screen is usable in Netscape?"  A potential additional test, maybe using an virtual machine of an older operating system.  [Although then again likely to need some set up and investigation that you might want to return to it later]
• "You showed me a currency transaction in Yen, but 95% of our transactions come in US dollars, Great British Pounds or the Euro - did you cover them?"  Possible here to either show them, or show them results where you did.  But they're trying to make sure you covered the test conditions which are most likely to occur.
• "That looks different to our current system - I didn't think this page was changing?"  Could be worth chasing up with your BA if you don't know, otherwise fill them in on the change.

It's important to take these things in the spirit of pairing.  If you're challenged and there's something you didn't think about, don't go on the defensive, but instead realise it's valuable feedback you've got, and the whole reason why you're spending time doing this.  If you think a suggestion is genuinely (and I mean this) unhelpful, try and explain as tactfully as possible - but remember your goal here is to influence, to win them over to your side, not to make them feel stupid.

One example of this was a project a few years ago.  We had just 12 options our customer could choose to add to our product when they set up an account, and they followed the below rules,

• The customer needed to select at least 1 option
• The customer could select options in any order
• The customer could select up to all 12

My business owner wanted me to be sure that I covered ALL options.  If you've read The Elf Who Learned How To Test, you know how much I warn about ever saying the word ALL.  I spent some time explaining how permutations worked in mathematics, and showed them some of the different ways you could arrange 12 products etc.  I then worked through an Excel spreadsheet to show them how many permutations of product there were - the number was something like 100 billion.  Their mouth fell open, because they had no idea when they said "all permutations" it actually meant so many.

At this point, I talked to them about what I had done,

• Gone through the Business Intelligence for the current system.  About 80% of people had just 1 option.  Then about 18% had 2 options.  Less than 0.25% had 3 or more.  Hence I did a lot of testing with 1, 2 and 3 options.
• I had covered some scenarios where all 12 or just 11 options had been chosen.
• I had tried out different option numbers each time I did exploratory testing, and not found any issues thus far.

I have to thank James Bach's Rapid Software Testing course for this kind of engagement with non-testers, but it helped a lot to make what I was spending time on more understandable.  I was very open to suggestions of distinct scenario, usually phrased "what about when ...", but I never tried to deceive them that ALL options were going to be tested or even achievable.

When this level of engagement is done well, it helps others to understand the challenge and skill of testing because it makes testing visible.  When it's not then typically organisations often find themselves believing that "they have a testing problem - that's why our products are late / have bugs".  Don't be that organisation.

Are you exploring?

The following is a write up of a talk I gave at the Christmas Wellington Test Professionals Network ...

If you follow me on Twitter, you probably have noticed I have a deep passion in history, which comes across in the sheer volume of history related facts that I retweet.  I'm not alone in this passion, as indeed my son shares the same interest.  He loves to read about it, to visit exhibits, but most of all to “experience” it.

He’s sometimes a bit disappointed with moving to New Zealand – England after all has history going back thousands of years, there are battlefields galore, and sites dating back to Roman or Norman times.  New Zealand in contrast was only colonised a few hundred years ago by Maori pioneers, with colonial Britain arriving in the 19th Century.

But when he heard that there was a World War II bunker at Baring Hill, near where we live in Wainuiomata, he knew he wanted to visit it.  We knew it was there, there’s a basic website warning us it’s a moderately difficult walk.  However there's no real information on the bunker itself, and Google doesn’t really provide us with much more to go on.

England itself is filled with remnants of fortifications thrown together to defend the country in a war that (thankfully) never came.  Likewise the Baring Hill bunker doesn't exist in history books because no documented history ever happened there, and yet it still existed in some form never the less.

This meant we just simply didn't know what to expect.  We still had images in our mind from our educated guesses.  We knew there was a lot of fear during World War II of invasion from Japan, and looking at a map, the position would overlook Wellington's harbour.  Thus we expected,
•    it to be the site of a giant fortification like those used by Germany to make “Fortress Europe”.
•    it was probably the site of a gun station – we didn't expect to see a cannon still there, but maybe some fixtures / turntable to show it was once there.
•    soldiers (maybe some form of home guard) would drive to the bunker to “do a shift” and then return home at the end of it

In other words, something a bit like this,

Our expectations

All the information we had to go on ...

Testing with personas ...

Last year whilst i was still at Kiwibank, I was lucky to attend a morning lecture by local company Optimal Usability on "Putting personas to work".

Many people who've worked with me will know I am not a morning person.  However I was glad to make this early morning event, mainly as within Kiwibank I got to network with a whole load of people I'd normally not have a huge amount to do with - Market Managers, Customer Experience etc.

A persona is mainly used for marketing - they are supposed to be a core of about 4-6 characters (who often have a bit of stereotype about them) who represent a cross section.  We had several at Kiwibank, and when we'd look at changing or offering a new service we'd look to see which of those personas would want in, and which might want to go elsewhere.

For both marketing and testing it's a powerful tool - it's where testing steps into the realm of method acting, and you try to see an application through another persons eyes.  This touches an important feature - your software might well functionally do everything that it's supposed to - but is it going to be something that people are going to feel comfortable using?

Personas should be a one-pager, of something people can instantly connect to, because everyone has met a "Pam" or "Tony" or "Chris" in their life (hence why here a bit of stereotype is handy is it's shortcut language).

Personas can be made from detailed market research to tie into a targeted customer base.  Whilst reading Elisabeth Hendrickson Explore It! she mentions personas as a tool for exploratory testing.

Whilst talking to Lisa Crispin about "software culture" recently she mentioned about how many of her team enjoy roleplaying.  In fact "exploratory testing with personas" is very much like role-play testing.  I couldn't resist trying to draw up some character sheets (I'm an ex-roleplayer, so don't act so surprised).

I wanted to imagine I had some form of "software management tool" (which we've all encountered at some point).  Maybe it was HP's Quality Centre.  Perhaps something like Pivotal Tracker.  Or New Zealands home grown Enterprise Tester.

I was going to work from scratch (although I have zero market research budget), then remembered it might be fun to revisit the people I call "the world's worst Agile Team" from An Agile Murder Mystery.   Maybe this group wasn't an idealised group for "understanding your market", but from a testing perspective they would allow you to thrash your software in ways you'd never before imagined ...

Mrs Peacock – Business Owner 

“I just want to know what the current status is” 

Personal Computer 
Mrs Peacock uses a Windows 7 laptop.  She’s out of the office a lot, and uses a variety of wi-fi points (some slow) to login.

Browser 
Internet Explorer 8

Usage rights 
View ability only

Tool habits
Mrs Peacock is mainly interested in seeing the big picture of the project, and wants to be able to see clearly how things are going, and it be able to take her into any issues.  She wants to see how things have changed.
She uses her laptop a lot on slow connection.  She also depends on the trackpad (she doesn’t carry a mouse with her), so she relies a lot of shortcut keys and wants any icons to be easily clickable.

Colonel Mustard – Project Manager 

“I need to be able to assign work items”

Personal Computer 
Mustard has been upgraded to a new Windows 8 laptop, and has been known to take an iPad into meetings to check updates.

Browser 
Internet Explorer 10

Usage rights 
Create work items and assign tasks

Tool habits 
Mustard is keen on being able to set work tasks to his team, see what they’re working on/progressing.  He sets pieces for all the team, but never himself works on end-to-end individual pieces.

He likes to be able to perform the same actions on multiple elements over having to keep repeating the same action.

Reverand Green - Architect 

“I wonder what happens if …” 

Personal Computer 
Green has made a big noise about being allowed to have a Macbook Pro, even though most of the company is on variants of Windows.

Browser 
Safari

Usage rights 
Full Administration Rights

Tool habits 
Green often plays around with any new features, and will often try and come up with new ways to do things.
Forever optimising the work-flow.  Often this means changing the flow for which there is currently data in “the old system”, and hits migration issues.

Miss Scarlett – Business Analyst 

“What’s next?”

Personal Computer 
On a desktop running Vista.  Don’t ask her how she feels about that, as she feels she’s due an upgrade.

Browser 
Opera

Usage rights 
Basic user – rights only over tasks assigned to her

Tool habits 
Scarlett doesn’t look at the big picture tool, just goes straight into and views the tasks assigned to her.  If work items aren’t assigned to her, she won’t go looking for work.

Prof Plum – Programmer 

“I don’t have time for this …”

Personal Computer 
Difficult to say – he has a cluster of a lot beneath his desk, running different flavours of Linux, and connect to a bigger Samba server that no-one is quite sure of the location of.  Certainly there are so many monitors on his screen, the company made major savings getting him to turn them off over Christmas … 

Browser 
Firefox … with so many plugins you don’t want to know.

Usage rights 
Basic user – rights only over tasks assigned to him

Tool habits 
Prof Plum finds spending time updating his tasks an annoying diversion.  He dislikes intently being asked after changing something “do you really want to do this? OK / Cancel”.  He’d much rather it just does it, and there’s an undo button if he changes his mind.

When it comes to filling out any form, he’ll always do it with the minimum mandatory fields possible.

Mrs White - Tester 

“I don’t see what was wrong with the old way of doing things?”

Personal Computer 
Two years ago Mrs White was upgraded to a Windows Vista desktop, which had issues during updates.  She didn’t like it, and rescued her old XP machine, which IT is now struggling to find parts to keep going.
Because her eyesight isn’t the best, she runs the machine on low resolution so it’s easier for her to see the icons (people have tried to tell her there are better ways to do this, but she’s sticking to it).

Browser 
Internet Explorer 7

Usage rights
Basic user – rights only over tasks assigned to her

Tool habits 
Mrs White spends a lot of time getting things right before submitting (unlike Prof Plum, she tends to fill in EVERYTHING).  Sometimes this means things get timed out.  It’s important to her that she doesn’t lose her work.

She’s also a creature of habit.  She knows a lot about the business area, but she really struggles with anything new.  She will make a lot of noise if changes mean she can no longer do something she used to be able to do.  She doesn’t like naming conventions or icons to change.

A source of testing inspiration?

Reading through that you probably have an idea of a few basic tests you'd like to run - cross browsers and computers for start.

But for Mrs Peacock you want to be able to login and quickly see "the big picture" and drill down.  Whilst Miss Scarlett wants to just be able to quickly see what's been assigned to her.  Can the system do both well without compromise?  Can this release cover Rev Greens desire to experiment vs Mrs Whites need for familiarity?

I found in writing these people out, I obviously went back to my source article on the Agile Murder Mystery.  But the key thing for "filling them out" was to find a picture of someone I felt "had the qualities".  Once I could see "who this person was like", a lot of the other details fell into place.  You will also notice in my choice of photos, I'm also a sci-fi geek (like you didn't know already).

If you want to find out more about personas - how to make them etc, the original presentation slides are still up here,

http://www.slideshare.net/slideshow/embed_code/14323974?hostedIn=slideshare&page=upload

And Optimal Usability do all kinds of blogs, articles and even a newsletter about areas of usability, including personas.

You may not have a market research budget - in which case, just do what I've done, guess, then put in front of a business owner or marketeer and see what feedback they give ... go broad in your guesswork, and put some things which are deliberately provocative in there ...

• "do we care about XP users?"
• "what about all these browsers?  Are we just going to go with IE?  Why shouldn't we just go with IE?"
• "why would someone want to access from an iPad browser?"
• "should this be accessed outside an organisations server from public wi-fi?  Should it be in the cloud?  Do we have strong enough security?"
• "companies don't work much on Linux do they?"

Have fun!

The science of software ...

Back in 2002, I had my own form of Buccaneering.  As a developer on a UNIX aircraft project, I'd been in the buisness for over 5 years, and noticed a few parallels between some fundamental laws of physics and software engineering.

I'd written these up, and had them pinned on my desk as "The Talks Physical Laws Of Software Engineering".  Unfortunately I've lost all copies of them, so I'm recreating them from memory ... as you can see they're not comprehensive and were meant more for fun, and from a development over testing perspective.

But what they're an example of the concept of Buccaneering, taking a parallel idea from (in this case) physics, and bringing it into the software engineering context.  Of course none are particularly ground-breaking (although we were caught out by rule 6 at the time) ...

1)  Problems in code (defects) will remain, unless work is done on them [Newton's First Law Of Motion]  Really, defects just don't go away ...

2)  As you are doing work to remove defects, you are also doing work to potentially introduce new defects [Newton's Third Law of Motion]

3) Software that is developed without any monitoring will tend towards chaos [Second Law Of Thermodynamics]  You are just going to hope that everything is good?  Let me know how that works out for you ...

4) Small deviations applied over long enough distances cause massive errors [Trigonometry]  So try and find any defects early.

5) Any system will have levels of uncertainty [Heisenberg's Law of Uncertainty]  Of course you want to minimise uncertainty (which is part of what testing's about), but you cannot remove it altogether!  It will always be there in unknown finite levels.

6) The more closely you observe an event, the more likely you are to be impeding it [Heisenberg's Law of Uncertainty]  Coined as we learned that trying to rerun defects using a debugger could prevent the problems re-occuring - mainly as many debuggers of the time would initialise data stacks which would otherwise contain random data.


Maybe you've read these, and come up with a couple more examples in your head?  Congratulations!  You're thinking outside the box ...

Rapid exploratory website testing ...

I was faced with an interesting challenge today at 12:50pm.  We have a minor project which I'm not involved directly with, and hasn't needed any testing staff.  However they've had a new supporting website produced to provide information … this had been delivered that morning - could I spend half an hour checking it out for anything "obvious".

This was an ideal opportunity to really stretch my muscles and give it a going over exploratory testing style.  I knew nothing of the website, though maybe that wouldn’t be a disadvantage, I’d evaluate it much as I could within the time allowed.

1:00pm, the link came through.

Opening the page, I began my analysis.  The site was essentially a marketing toy, telling prospective customers about a new service which was being provided, and would allow them to register an interest.  It detailed all the features which were coming, why it would make life a lot easier, as well as links to both supporting partners and associated stories about the initiative.  It also had a top menu bar which allowed you to jump to parts of the page you were interested, which dealt with a particular aspect of the new service.

Several areas had icons, which if you held your mouse over, would allow bubbles to expand, giving a more detailed graphical explanation.

Starting with the basics, I attempted to click every link, every button on the page, making sure it went to the right target.  Two of the links to supporting partners could not be selected with the left mouse button, but could with a right button click menu [Defect 1].

I tried out the webpage in Internet Explorer.  The menu bar buttons did not take you to the right part of the page at all, which was most curious [Defect 2].

I opened the website using Chrome and Firefox (the browsers I had available).  The page looked identical in all browsers.  However in these two browsers the menu bar buttons DID work as expected [Revised defect 2 with this information].

Dragged my mouse over the icons that opened graphical explanations, confirmed they made sense and didn't behave oddly if close to the browser view edge.  I did wonder why one story had two links to website when others only had one (inconsistent) [Defect 3].

Read through the website – did it make sense?  I noticed that two sentences in the same paragraph were virtually identical, and referred to supporting partners in an inconsistent manner [Defect 4].

There was a field to register interest by adding your email address.  Tried a valid email, it was accepted (good).  Tried one junk email, and got told it was invalid (good).  Tried a couple of variations of invalid emails, not all were rejected.  Noted it as a possible problem  [Defect 5].

The bottom of the page had a huge, but empty grey box which just looked messy [Defect 6].

At this point I thought I was done.  Then I had a moment of slyness.  Thinking about how the menu worked for Firefox and Chrome was a little suspicious - I know web developers tend to love working and testing out on these browsers.  Likewise, I also know how web developers love their big, high resolution screens.  So I went into my personalised settings, and dropped my screen resolution to 600 x 800.  The page now no longer fitted to the screen, and some buttons on the menu bar became mangled with some icons missing altogether [Defect 7].

I emailed my list of discoveries to the project manager.  It was 1:30pm and time for lunch.



That was testing at it’s most raw, and a lot of fun (and a nice break from the meetings and documentation of the rest of the day).  For the product, it was a perfectly pitched piece of ah-hoc testing.  I defected everything I thought could be an issue - of those, there are about 3 things which need doing (the non-selectable links, menu not working in IE and probably the behaviour in low screen resolutions), the rest are more about consistency and look, which might be equally important if there's time.

The issue with the menu bars was discovered by a BA during the same time.  But where they reported it, I managed to define it was an Internet Explorer issue, and not one on Chrome and Firefox.  This made me realise that testers are more than people who "find problems" (their BA, a very talented and smart woman did that), however being a tester, it was my nature to go further than just find the problem, but "define the problem".

A most interesting exercise for sure ...