I was taking another look at Lessons Learned In Software Testing on Amazon today. I've used the work copy extensively, and really want "my own" edition, but I'm torn between having a paper copy or an electronic version (I use my Kindle a lot).
I have to admit I've not really finished it yet - but I do find it full of useful insights. I often read things in there that I'm afraid to say I agree with, because I've seen companies and projects learning the hard way.
What interested me was the reviews, or rather the profile of the reviews ...
Most people think it's an amazing and indispensable book ... except 2 people who think it's dreadful (out of 50, that's about 4%). Fascinated, perhaps with a shade of rubber neck syndrome, I had to look at these reviews. Maybe these "thought leaders" of testing had an interesting take on the book, that I should not be dismissive of.
This is by far the more interesting ...
Promises Much, Delivers [little]
This book was praised by several colleagues as THE way to work on testing methods and thinking. After reading it and talking with each of them, it was apparent they were excited based on false credentials about ideas that were easy and comfortable but ineffective long term.
This book is VERY dangerous to a serious testing organization because it focuses on minimal documentation (which means in 6 months when you're asked if you tested X and you can't remember, you'll get 5mins to get out of the building), downplays automation in regression testing (what!!?), and admits openly that it is proposing ideas that are NOT proven (contrary to what the title states) but rather are ideas that "seem to be working" (see pg 176) but no formal nor long term studies support their claims.
Well, long term studies that have already been done directly contradict their findings: process is driven by a need to be effective and if you don't know what you're doing before you do it, then you don't know what you did when you're done. ... This is a book for those who advocate ad hoc testing to their own discredit and need a means of justificating their apathy and laziness to those who actually know effective testing techniques.
Now that is a bad review worth having! Sadly I am surprised that this form of opinion represents only 4% of people - though the figures are going to be skewed by the fact that people who feel as the above individual are (I seriously doubt) unlikely to buy this book.
But this view, represented by, let's call him for simplicity Rex, is one many software testers have to face.
The phrase that struck me with a level of absolute terror on Rex's thought leadership style was that piece here, "it focuses on minimal documentation (which means in 6 months when you're asked if you tested X and you can't remember, you'll get 5mins to get out of the building)".
That isn't a model of testing - that's a model of QA, in this case Quality Auditors. In his model, QAs go around as almost accountants of software recording, noting, writing. Constantly writing more and more reports and documents which just stack up.
I have to admit early in my career I worked for a project like that (to be honest we didn't know better). When we ran a "formal test", we needed three people - one to read the script, one to perform the action, and a third person who "witnessed" our test,
at the end of a successful step, the reader would tick a box in the printed hardcopy of the test script.
at the end of a successful test case, we'd each sign our relevant roles
at the end of a successful test phase, all our signed test scripts would be filed in our office in case of audit
And sometimes we'd have the customer come along to witness one of our tests - so we'd have a special signature box for those occasions too!
Pretty soon of course, the office just couldn't keep up with all the paperwork we were generating. In truth we were probably now a fire hazard. We looked like something from the movie Brazil ...
What we eventually learned was that having up to four people witness every test step, was kind of inefficient. Oh sometimes it was worth having two people testing together to challenge each other, but mainly it was a waste, slow, boring. At the end of my two years there I heard two testers running one of my scripts and I was actually ashamed of how slow and boring I'd made testing on the project.
The problem with the Quality Audit model of testing is this,
just because you're recording everything you're doing doesn't mean you're doing the right thing. It just means you can defend what you did. And if you missed doing something because (a) people wouldn't give you feedback or (b) you were committed to following your scripted planned process, even though you found issues, then too bad.
you're to blame for software. Notice in Rex's take on testing, if there's a problem with the software, it's not the project manager, the developer or the BA who is made to pack up their desk. It's the tester. The "QA tester" here is the fall guy - he owns quality, so if the product has issues it's because he didn't add quality at the end - fire him! Why would anyone want to work in that role? Personally I'd rather be a Wedding Planner in The Game of Thrones!
it creates the idea of testing being "just a very bureaucratic layer of software development". We had three people performing one test - and doing it by slowly reading instructions and performing them. Did we get the best value from those three testers using that model? Hell no! But the documented process was to die for, and would keep our auditors happy! This leads to the idea many managers have that testing is full of waste. Yes we do need to document anything important (see my piece on exploratory testing for more), but to give the most value for our testing time, we need the most amount of time either with hands on the software, trying out different ideas and pathways.
Of course the problem with the last point on bureaucracy is inevitably testing finds itself squeezed, and it's very tempting to do the wrong thing in that position (you keep the level of documentation, but test less yes?). As I've said, I'm a huge fan of using tools like QTrace which just make notes for me as I test, so I don't need to either follow a script or take time consuming screenshots as I go. Sometimes I know I have a problem area, and I want to try a few things out, and having it recorded helps be go back and show someone. But even so if a recording doesn't show something, I don't just save it "just because" I want to avoid building up a huge library of files if I don't think there's anything useful in that file.
Testing isn't about auditing software, it's about trying things on your software (the actual doing, not the documentation). Probably the documentation that matters the most are bug reports - but even so I see test managers who have a hard time coping with teams who deal with defects that can be fixed easily "using sticky notes" because they can't use their tool to "see what defects were fixed 6 months ago".
I'm glad I see less and less "thought leaders" like Rex. Their ideas to software and to testing are pure bureaucratic toxicity. They imagine a career path that as long as you can generate a documented trail of evidence you can claim the high ground with any project stuff up. I think somewhere down the line, Rex is in for a nasty shock ...
I'll wrap up with a superb and concise quotation from Scott Barber to one of the reviewers,
"This review makes it clear that what you really want is to not have to think. Testing is a thought-engaged activity. I suggest that if you want to be told the "only" way to do a thing, that testing might not be the best career path for you."
For me, grammar is a bit of an odd thing. I was a child of the 70s, and the thinking of the UK educational system I was a part of was that you taught people how to use the English language more through repeated use and examples, than applying terms and rules and assessing them on that.
It's interesting, because I do a good job with writing (not perfect, but hey, this blog and a couple of books isn't bad). Grammar I'm not bad on, however I'm actually partly dyslexic, and my spelling is pretty terrible - thankfully I've learned a few tricks to hid it, such as a reliance on the red underline on Word documents to warn me I might have made a boo-boo. [Typically I go ballistic when someone turns that feature off, as it's part of my coping mechanism]
I've been able to use the language quite well, to University academia standards and beyond. But people are sometimes surprised to know I struggle to know what a verb and a noun is. We were taught to use the English language without the labels. In fact in many ways I've learned more recently about this through my son as he goes through education himself (and as per the norm, names and labels are back in vogue).
It's impossible to avoid some of the grammar memes which are floating about social media. Mainly I find them quite fun, and educational. Although I find some can be posted from grammar bores.
This though has to be one of my favourites,
Now this is a grammar rule that I definitely know - the difference between,
your - meaning an item that belongs to you such as your ball, your vanity, your blog
you're - a contraction of "you are", examples being you're vain, you're annoying, you're welcome
What's fascinating though is despite be knowing this rule quite well, I'll often reread something I've written on here or in one of my books, and notice despite knowing this rule, despite having reviewed my material (in the case of my books, externally), I will occasionally make the mistake of using the wrong form.
This fascinates me - indeed it's the basis of software testing. Even when we thoroughly know a form like the English language, we'll occasionally make an "oops". So any form of writing can be subject to these errors, and this obviously goes for coding as well.
If people were capable of never making these forms of mistakes, we'd need less testers, proof-readers and the like. But of course, it's human nature to make these mistakes - and whilst we can take actions to minimise them, we can never take action to remove them altogether.
It's a very funny sketch, and many people have commented that "I have been in that meeting". Business people don't get what technical people say - drum roll and big laugh please!
Now here's the challenge. If you were the technical person in that room, how would you try and take charge of that meeting and prevent that car crash from happening? The technical expert tries his best, but could you do better?
The challenge from me to you, is that rather than laughing "business peeps iz sooo stupidz", how could you turn situations like this around? Because I guarantee you that you will be faced with at least one such meeting - if you have your sights on being senior, it'll happen more and more ... because you're the expert remember!
When I left Assurity to join Kiwibank, part of the fun and challenge was leaving a technical environment to one where I would be one of the few technical people in a predominantly business area. I talked a little about this experience when I looked back at test estimation. There's no doubt that like The Expert, I made a whole heap of mistakes, mainly by assuming my audience had a similar background in IT to myself. But instead I often worked with people to whom the IT solution was a means to an end, and that meant I had to get into their headspace a little first.
Though I never got perfect, I certainly got better. And a major way of doing that was replaying conversations like the one in The Expert and asking "well ... how could that have gone better". Give it a go ...
I'm very pleased to announce that a group of Whirlwinders including myself will be doing an evening of comedy on Thursday 29th May at the The Jolly Pub in Paraparaumu.
If you'd like to support this worthy cause, tickets are $10 each (including spot prizes), and you can order yours by contacting Martin Sloman. If you can't make the event, I'd still encourage you to take a look through the Whirlwind website, and view the amazing work that Whirlwind's behind.
Make no bones about it, I love April Fools Day. At Thomson Marconi, the first company I used to work at, the monthly site newsletter in April would always be looked forward to. Departments would sneak in the occasional "fake story" which you'd start out reading, and bit by bit would lure you into believing the incredible.
Back in my 20s, I myself got cautioned at EDS for circulating a fake HR memo, which although many found hilarious, looked a little too real for some. If you have a team like mine at Thomson Marconi, where you can share good humour, April 1st helps you to really share that important team spirit (though there's a reason it's only once a year - too much pranking can get tiresome). Indeed the cautionary tale from me getting into a bit of hot water at EDS is that it only takes one person to complain, and then you might have to explain your prank to some higher ups!
One of my favourite activities on April 1st is looking at newsites, and this year as I did, I noticed a perculiar activity going on in my brain. In many ways April Fools Day introduces an element of critical thinking we don't exercise as much on the other 364 days of the year. Looking at the list of news items, I know that one of them (or maybe two) are likely to be a hoax, and so I read them, challenging the logic.
There is certain pattern to potential hoaxes - no-one would surely "hoax" a plane crash or a disaster where people were killed, would they? I thought this years candidate might be a story about Gwyneth Paltrow, as she was talking about her childcare arrangements and going on tour with Coldplay for the benefit of her children. However, this was a week after Gwyneth announced her "conscious uncoupling" (or separation to you and me) from her husband, Coldplays lead singer Chris Martin. Unfortunately with celebrities, a variation of Poe's Law comes into effect, and "it is difficult or impossible to tell the difference between an expression of sincere extremism and a parody of extremism". Maybe not so much Poe's Law, as Paltrow's Law.
Some of the greatest pranks ever pulled (not necessarily on April Fools Day) include the hysteria some Americans experienced listening to Orson Welles War Of the Worlds broadcast in 1938 and when the UKs Panorama (a respected journalistic program) ran a story in 1957 on a Switzerland family harvesting from their spaghetti tree. Both of these stories came from a time when there were limited radio and TV sources, and thus any information provided by them was seen as authoritative. For The War Of The Worlds, the play was presented in the first half as a series of interrupts to a seemingly live program of "Ramon Raquello and His Orchestra". The radio audience were used to plays that "sounded like plays", rather than what is in many ways the first mockumentary.
Media awareness helps
Back in 1993, I took a media awareness module as part of my teacher training. During the week long module, the world watched the final tragic moments of the Waco siege being played out. Our workshop had copies of all the UK papers that week, and so it allowed us to see the spectrum of how the story was being told. And believe me all the accounts did not add up to quite the same tale. Some papers were quite restrained, giving an almost clinical account of events. The UK tabloid press typically focused on more lurid details of usually the Branch Davidians strange sexual practices, and focusing on the deaths of the children involved (which sadly I can't read about without thinking about Damien from Drop The Dead Donkey).
One quite startling non-story which was given prominence on the day in one such tabloid was Uri Geller's claim that if only the authorities had let him speak to the cult involved, he could have used his abilities to end the siege. This did not seem a hard news story, but just a piece of opportunistic self-promotion, and I found myself not only disgusted by Uri behaving like this (but then don't psychics make a living saying they can contact your beloved lost ones?) but the paper for giving it the time of day.
The events of that week, and our focus on how media were portraying it allowed me to become much more savvy about this. When is information being portrayed to us as information but secretly is serving a secret, self-serving agenda, such as with the Uri Geller tale? As with the news, if you trust one source implicitly you'll never be sure if you're getting facts or an editorial take on events that serves someone's soapbox. The only way to not be taken in is to take a critical thinking approach, to inquire, and get other sources of information to compare and contrast, and then settle for the one you feel fits best.
My son is currently working towards taking history at University in New Zealand. He's learning at the moment that doing a report based on one set of information is amateurish at best. Whenever possible, he's looking for multiple testimonies on modern events, and contrasting them. Most often he's asking the question of "who is giving me this information".
The who he tells me is really important, and that was reinforced with a recent video we watched - Akira Kurosawa's Rashomon. The film is about a monk and a woodsman trying to make sense of events - a local samurai has been killed, and his wife raped. The problem is that everyone's story doesn't add up - everyone has given a different, completely contradictory tale, but one which claims themselves to be innocent.
Having done my piece as a juror in an actual bodily harm case, this is very much as things are in real life. There are the victims friends who tell one tale. The defendants friends who tell another. And then those in the middle who to an extent cast a shadow of doubt over both sides.
So how can we ever know the truth of anything?
Scary isn't it? How can we tell when someone in our work is selling us a tool or methodology which will serve our needs, or is selling something for their interests which will cause us to fail?
This is one of the reasons why Aprils Fools Day is to me such a special day. It's probably the day where we look at news stories and we're deliberately expecting a lie, and so we approach it differently. This year for April Fools, I wrote an article for Sticky Minds called QA Management: All Bark And No Bite? The story was a tale of how an agile team was using a retired guard dog (it was going to be a police dog at first) to collect the team together for meetings, and keep developers and testers in line. Said like that it's far fetched, but I found a way to tell the tale a piece at a time, so each piece made sense with what I'd said before.
In the office, we can see this manifesting itself in certain subtle traps. It might be that the virtues of a methodology or tool are being oversold to us, and we hand over cash to impact on a nasty surprise. I managed to sell a big lie through a whole lot of little ones. Yup - kind of feeling a little bit despicable about that ...
So here are a few guidelines against falling into traps when we have things presented to us ...
Does it sound too good to be true?
Oh, c'mon, we all have one of those exercise machines at home where we fell for the advertising that "it was the one piece of equipment we'd need to get into shape". Typically it's in the garage, under a whole heap of other unused stuff. We've all fallen for the hype on at least one thing, and if we're lucky we learned from it that some things which sound too good to be true often aren't.
If someone isn't telling you about the cons or limitations of something they're trying to get you to use, be very afraid. Because there are always limitations! At the very least try a bit of a Google search before leaping, and look for people's negative stories ...
Reputation
A quick Google on someone is always useful, as is asking around. Have other people followed this persons advice and regretted it? I know James Bach has talked about how important within software testing a tester's reputation is, far more than any qualification.
It's true - many people have for instance favourite actors/directors. I will typically go out of my way to watch a Peter Jackson, Quentin Tarantino or Tim Burton movie. That's typically because I've enjoyed their past movies. And based on enjoying those movies, I know their next movie is likely to be a good one, or at least I expect it to appeal to me.
Likewise from past movies, I don't really like the work of Tom Cruise, Ben Stiller, Will Ferrell or Ricky Gervais. If you ever see me at a movie that stars all of them, then it's likely the hall on the way to the movie theatre will have scratchmarks frin my fingernails, as I will have had to be dragged there by the devils minions themselves. Or these guys ...
But seriously, ask people who've done business with this individual for their thoughts - and do it directly, not through testimonials. Did they feel it was useful - did they have any surprises? Did they understand the benefits?
And don't forget the reputation of the publication is important as well. If someone is promising a brand new method of software deployment in The Onion, what do you think?
Timing
If it's on, or near 1st April, be afraid ...
And finally ...
The very topic of "how can you determine the usefulness of a source" came up at the end of my workshop on Testing Gravity at WeTest Workshops, where I presented these two videos about "gravity in space" ... we couldn't go there ourself, so we needed to rely on secondary sources,
So two clips about space, gravity and space stations. What's the difference?
Well, while they both try and represent some accuracy about life in space (and what it can be like). The fact is if you look into the first clip, you find it's from Stanley Kubricks 2001: A Space Odyssey, and whilst it's attempting to be accurate, it's not real evidence of life in space, just a theatrical mock up. You know this because you look up and Stanley Kubrick is a director, and he filmed 2001 very much on Earth.
The second clip however contains Chris Hadfield. And a good search online will tell you he's an actual astronaut, and that was filmed not in studio, but on the International Space Station (ISS). So what you're seeing isn't a representation of what life should be like in space, it's the real thing. It's valid as evidence.
Oh of course you can go further on this and say "well how do we know that he really did go into space, and the ISS, isn't just on a Hollywood studio backlot somewhere?". Well a lot of people have testified to him going up there, people have seen rockets heading to space, you can track the ISS and view a fast moving star going over your house which moves like no airplane we've ever seen. Now that's not concrete proof (but then again in life what is?), but it's certainly enough to suggest that the second clip is "the read deal" and has potential to be genuine. Or of course, a really elaborate (and expensive) hoax ...
We have a certain view of test practises which have been
passed to us from the waterfall project roots we've all had some experience of. In many ways, these methods are familiar, especially to non-testers, and from this
familiarity we feel a measure of comfort and security which sadly can often be
unwarranted.
When veterans of waterfall such as myself are first faced
with some of the concepts of exploratory testing our reactions can range from
shock to abject horror.Almost always
though the initial reaction can be a negative one.
This article will help to define exploratory testing,
de-construct what we do in scripted waterfall delivery testing, and talk about
how we provide the same essential value with exploratory testing.It will cover our approach to using
exploratory testing as part of a system of testing, although of course other approaches
can vary.This is how we've found to get
not only the best results, but the best buy in and turned our detractors into
our allies.
Back when I did a series of articles on Building A Testing Culture for Testing Circus based on conversations I was having with other software testers, a major change and indeed the future of software testing was seen to be in the hands of increased use of exploratory testing. And yet the idea isn't new - it was first coined by Cem Kaner in the 80s. So what is it, and why is it becoming increasingly applicable to the way we're running software projects?
A few homegrown definitions
Let’s start by
making a few definitions before we discover further! These are the definitions I'm comfortable with, but you may find definitions will vary.
A test script is a set of instructions and expectations for
actions on a system which a user then confirms when executing testing
Historically I've seen many a project where such
scripting has meant we’d define every action of every button press, and detail
every response.
Typically in waterfall, we would have a good amount of time to
test … or rather we would ask for it, and whether we get it or not would be another tale.
A product definition would come out of
requirements and design, and whilst everyone starts to program against these,
testers start to produce plans of what is to be tested, coupled with defining
scripts for the new functionality based on these definitions.
Once done these plans and scripts are ideally put out for
review – if you’re lucky you'd get feedback on them, which means you expand or
reduce your coverage to include the areas which are key to the product.
When we run our scripts, typically life couldn't be simpler,
all we do is go through, try each step, and tick or cross as we go to “prove”
we've tested.
The problem with scripts is that they’re very brittle,
they require a lot of time upfront (before the
tester has access to a product) to scope to the level of detail discussed above.
they are fallable in the fact that just as
developers are not immune to misinterpreting product definition neither are
testers.And that means large amounts of
rework.
the requirements and design need to be static
during the scripting and execution phase.Otherwise again, you have a lot of rework, and are always “running at a
loss” because you have to update your scripts before you’re allowed to test.
they depend on the testers imagination to test a
product before they've actually seen a working version of it.
Unfortunately talk to most testers today, and they’ll report
to you these are areas they’re under greatest duress.The two biggest pressures they feel revolve
around timescales being compressed and requirements being modified as a product
being delivered.
Some testers have a desire to “lock down the requirements”
to allow them to script in peace.But obviously
that involves to some extent locking out the customer, and although it’s
something that we might feel is a great idea, it’s important to have an engaged customer who feels
comfortable that they've not been locked out of the build process to have a successful project.
So testers have to be careful about not wanting to have a
model for testing which works brilliantly in theory, but breaks down because of
real and pragmatic forces on their project.
Exploratory testing has multiple benefits – one of the
greatest being that it doesn't lock down your testing into the forms of tests
you can imagine before you've even had “first contact” with a prototype version
of software.
Exploratory testing is a method of exploring software
without following a script.There are
many ways to perform it – the best parallel I have found is with scientific
investigation.You set out with an
intention, and you try experiments which touch the area, devising new
experiments as you go along and discover.
With exploratory testing there is more value in noting what
you've actually done over recording your intentions.Compare this with scripted testing, where you
put the effort in ahead of time, and all you record during your testing is
either a tick for a pass, or cross for a fail!
A test session is a high level set of behaviour that the tester should
exercise when executing testing. Often this can be a list of planned scenarios we think would be good to exercise. But unlike a script, it's a set of things to try in testing, without getting bogged down early on with the step-by-step instructions of how that will be done. It also is more a set of suggestions, with room for additional ideas to be added.
So what is
exploratory testing?
There are many definitions out there for exploratory
testing, and I'm going to add my own understanding of it.
Exploratory testing to my team is about
building up an understanding first of what the core values
and behaviour of a system is (often through some form of Oracle)
using that understanding to try out strategic behaviour in
the system to determine whether what we witness is unexpected
Oracles
An oracle is a guide for understanding how the system’s
supposed to behave.An obvious one we
all know is a requirement document functions as an important oracle.However it’s not the only oracle there is.
Recently when we were making changes to our registration
page, we took a look at registration pages on Gmail, Hotmail, Facebook,
Twitter.Many of these had some of the
features we used in our registration page, so it gave us a method of playing
and getting familiarity with.
Especially if you’re producing a system that’s got a broad
customer base, most users don’t have the advantage of reading a whole stack of
requirements when they use your system.Your product has to make sense, especially given similar products in the
market.
This ability for a tester to look at a page and ask “does it
make sense” is an important freedom that’s required in exploratory
testing.Sometimes saying “as per
requirement” isn't enough, and we have to push further.
Recently we put together a system to allow a user to manage
their own password reset when they’d forgotten their password.All the required behaviour was put in front
of a lot of people who signed off on what they wanted.The requirement we had read that the email the user would receive would say “your account has been unlocked, and you
can no login”, just as per requirement.My tester dared to suggest that “you can now login” would perhaps make more
sense, going beyond just taking the requirements to be the holy oracle of all truth, and using a bit of common sense.
Somehow that typo had got through a lot of people - they did indeed want "you can now login" - but then, that's the nature of testing. I've seen much larger slips than that pass by uncommented before now ...
Example
You’re testing a login page, the requirement says, “When the user provides the correct password and username,
the user is logged in”.
Within exploratory testing, it’s an expectation that the
tester will be able to expand on this, using their experience with similar
models,
This is a list of a tester using an Oracle based understanding and going beyond just the
requirements to plan testing. They'll typically expect,
If I give an incorrect password for a username,
I won’t be logged in
If I give a correct password for the wrong
username, I won’t be logged in
On other systems the username I use isn't case
sensitive – should it be here?
On other systems the password I provide is case
sensitive – should it be here?
What should happen when I try to log in
incorrectly too many time?Does the
account get locked or should it be locked for a period of time?
Skill Based Testing
This ability to think beyond just a one sentence requirement
is part of the inherent skill which is a core need for exploratory
testers, it calls for,
An understanding of what the systems supposed to
do.Not just functionality, but the
“business problem” it’s trying to address.
An understanding of how the system has worked
thus far is helpful
An understanding of similar behaviour in other
products
Exploratory testing is thus often referred to as “skills
based testing”.
An argument often used in support of scripted testing over
exploratory testing is that “if you have your testing all scripted up – then
anyone can do it”.Indeed such scripts
it’s possible to provide to almost anyone, and they can probably follow them.
The problem tends to be that different people will interpret
scripts differently.Most inexperienced
testers will tend to follow an instruction, and if what they see on screen matches, they'll just tick a box, without thinking outside of it (and hence would be quite happy with "you can no log in" example above because it was what the requirement said). Lets explore this with another example ...
Test script
Action
Expectation
Pass/Fail
Click the link to take you to the login screen.
There are two fields for entry,
Your account
Password
The Test System
Here are 3 screens for the above script ... do they pass or fail?
Can you be sure that every tester would pick up on that from
the script? If so would it be a simple
cosmetic defect, yes?
Sadly some people will combat this by trying to turn this into a version of the evil wish game, where you go overboard on explaining what you expect, so there' no room for ambiguity. Hence an obsessed scripted tester might try and write the expectations as,
Action
Expectation
Pass/Fail
Click the link to take you to the login screen.
There are two fields for entry,
Your account
Password
The text on the display is
all in Calibri font 11
all UPPER CASE
black colouring is used for all text
Yes sirree - that will stop those 3 problems above occurring again, but it won't stop other issues. Especially if it turns out the root cause for the problems is a disgruntled programmer who's tired of reading your scripts, and currently serving out their notice period until they move to another company where they do exploratory testing ...
"But our scripts are
our training!"
This is a common objection to exploratory testing. If you have detailed scripts, anyone can follow them, learn the system and be a tester right?
Most people I have spoken with have found that it’s far easier to have a
handbook or guidebook to a product, which details at a light level how to do
basic flows for the system, than to have high level of detail throughout your
testing.
The counterargument to handbooks is that “anyone can follow a script” but
as discussed previously to build up familiarity with the system and the values,
you will always need some training time to get that.If you just throw people at the “your
account” problem you’ll get differing results, you need people to be able to be
observant and in tune with the system, and that doesn't tend to happen when you
just throw people blind at a system with only scripts to guide them.
The bottom line is that there are no shortcuts to training
and supporting people when they’re new testers on their system.If you’re going to train them badly, you’re
going to get bad testing. One of the few exclusions we've found to this is areas which are technically very difficult - there are in our test system some unusual test exceptions we like to run, and to set up these scenarios is complex, and completely unintuitive (includes editing a cookie, turning off part of the system etc). This is one of the few areas where we've voted to keep and maintain scripts (for now), because we see real value in the area, and with maintaining so few other test scripts, we can do a good job, without it chewing up too much time. But we've found we certainly don't need test scripts to tell us to login, to register an account etc - when the bottom line is our end users only get the instructions on the screen to guide them. Why should a one off user not need a script to register, and yet testers like ourselves who are using the system every day require a script to remind us that if we enter the right combination of username and password, we're logged in?
Debunking the myths
A common complaint about exploratory testing is that it’s an
ad-hoc chaotic bug bash chase.You tell
a team to “get exploratory testing”, and they’ll run around like the Keystone Cops, chasing the same defect, whilst leaving large areas of functionality
untouched.
This is some people's view of exploratory testing
With such a view of exploratory testing, it’s no wonder a
lot of business owners see it as a very risky strategy. However, it’s also misleading.
Whilst such ad-hoc testing can be referred to as exploratory
testing – it’s not what many people’s experience of exploratory testing is
like.
Just because exploratory testing doesn't involve huge
amounts of pre-scripting, doesn't mean that exploratory testing is devoid of ANY form of pre-preparation and planning.
You will often hear the words “session based” or “testing
charter” being referred to – these are a way at looking at the whole system and
finding areas it’s worth investigating and testing.The idea is that cover your sessions, and you
will cover the key functionality of the system as a whole.
Drawing up a map of
sessions
Creating a map of sessions is actually a little harder than
you’d first think.It involves having a
good understanding of your product and it’s key business areas.
Let’s pretend we’re working for Amazon, and you want to
derive a number of sessions.Two key
sessions jump out at you right away,
Be able as a customer to search through products
and add items to the basket
Be able to complete my payment transaction for
my basket items
Beyond that, you’ll probably want the following additional
sessions for the customer experience,
Be able to create a new account
Be able to view and modify the details of an
existing user account
Give feedback on items you ordered, including
raising a dispute
Finally obviously everything can’t be usre driven so there is
probably,
Warehouse user to give dispatch email when
shipment sent
Helpdesk admin user to review issues with
accounts – including ability to close accounts and give refunds
The trick at this point is to brainstorm to find the high
level themes to the product.Ideally
each session has a set of actions at a high level that really encapsulates what
you’re trying to achieve.
For more on mind mapping (I tend to be very much list driven in thinking), and I recommend Aaron Hodders notes on the subject.
Fleshing out the
planned test session
For many skilled testers, just giving them the brief “be able to
create a new account” will be enough, but perhaps a lot of detail came out of
your brainstorming that you want to capture and ensure is there as a guideline
for testing.
Let’s take the “be able to create a new account”, here are
some obvious things you’d expect,
Can create an account
Email needs not to have previously been used
Checking of email not dependent on case entered
Password
oMust be provided
oHas a minimum/maximum length
oMust have a minimum of 2 special characters
Sends email recript
A session can be provided in any form you find useful – mind
mapped, bullet point list, even put into an Excel sheet as a traceable test matrix
(if you absolutely must).Whatever you
find the most useful.
Such bullet pointed lists should provide a framework for testing, but there
should always be room for testers to explore beyond these points – they’re
guidelines only.
Keeping it small
Typically not all test sessions are equal – the test
sessions for “adding items to basket”, if you've got over 10,000 items to
choose from (and no, you’re not going to test all of them) is obviously a
bigger task than creating an account.However if some sessions are too big, you might want to split them up –
so you might want to for instance split “adding items to basket” to,
Searching for items
Retrieving details for items
Adding items to basket
But really breaking it down is more an “art” than a set of
rules, as you’d expect.
Test Ideas For Your
Session
Okay, so you have a session, which includes some notes on
things to test.Hopefully with your
experience in the project, and test intuition you have some ideas for things to
try out in this area.
A common thing testers use to explore and test is called
heuristics, which are testing methods.Elisabeth Hendrickson has a whole list on her Heuristic Cheat Sheet.
But here are a few of those which should jump out at you,
Boundary tests – above, on the limit, below
Try entering too much/too little data
Use invalid dates
Use special characters within fields
Can a record be created, retrieved, updated,
deleted
Using heuristics, a simple statement like “can I create an
account” can be expanded to be as large/small as needed according to the time
you have.
Working through a
session
In running and recording test sessions we've gone right back
to University. For those of us who did
science classes, an important part of our lab experience was our lab book. We used this to keep notes and results of
experiments as we tried them out.
They weren't wonderfully neat, and we’d often change and
vary our method as we went along.But we
could refer back to them when we needed to consult with other students or our
tutor.
For our group, these notes are an important deliverable as
we move away from scripted testing.Typically we’d deliver our scripts to show what we intended to
test.Now when required we deliver our
notes instead showing what we tested (over intended to test).Not every exploratory testing group needs to
do this, but to us, it’s an important stage in building the trust.
Obviously a key part of this involves “just how much detail
to put in your notes”.And this is a
good point.The main thing you’re trying
to achieve is to try our as many scenarios as possible on your system under
test.
We've found a table in simple Microsoft Word is the most
comfortable to use (you can make rows bigger as you need to put more details
in).Basically you put in your notes
what your intentions are.
We couple this with using a tool to record our actions when
we test.Each row in our notes table
covers about a 10-15 minute block of testing we've performed.We use qTrace/qTest to record our
actions and the system responses during this time.
This allows us to use our notes as a high level index into
the recordings of our test sessions.Obviously things like usable names for our recordings help – we go by
date executed, but anything will do.
A screenshot from a typical session log
It should be noted of course that not all exploratory
testing needs to be recorded to this detail.
For my group, as we move away from scripted testing, this approach allows us to
build up trust in the new system. The bulk of this documentation –
screenshotting, and a button-by-button trace of what we do – we find qTrace invaluable in absorbing the load for us (it would slow us down like crazy otherwise).
Of course, any test we find ourselves doing again and again
is a candidate for automation – and even here, if we have a suitable flow
recorded, we can give a qTrace log to our developers to automate the actions
for us.
Closing a session
When a tester feels that they've come to the end of a
session, they take their notes to another tester, and discuss what they've
tried, and any problems they've encountered.The peer reviewer might suggest adding a few more tests, or if there
have been a lot of bugs found, it might be worth extending the session a bit.
This peer reviewing is essential, and we encourage it to
lead to some peer testing.Basically no
matter what your experience, there’s no idea that you can have that can’t be
enhanced by getting the input from another individual who understands your
field.
This is why for us, the method we've used for exploratory
testing really confounds the standard criticism that “when you write scripts,
they’re always reviewed”(even this is rapidly becoming the unicorn of software
testing, with many scripts being run in a V0.1 draft state).
In both using reviewing (through brainstorming) of when we
plan our test sessions and also using a level of reviewing when we close a
single session, we’re ensuring there’s a “second level of authentication”tof
what we do.Sometimes we wonder if our
process is too heavy, but right now an important thing is that the reviewing,
being a verbal dialogue, is fairly lightweight (and has a fast feedback loop),
and both testers get something out of it.Being able to just verbally justify the testing you've done to another
tester is a key skill that we all need to develop.
Peer Testing
We try and use some peer testing sessions whenever we
can.I've found it invaluable to pair up
especially with people outside the test team – because this is when the testing
you perform can take “another set of values”.
Back at Kiwibank, I would often pair up with internal admin end users
within the bank to showcase and trial new systems.I would have the understanding of the new
functionality, they would have a comprehensive understanding of the business
area and day-to-day mechanics of the systems target area.It would mean together we could try out and
find things we’d not individually think to do.Oh, and having two pairs of eyes always helped.
Pairing up with a business analyst, programmer or customer
is always useful, it allows you to talk about your testing approach, as well as
get from them how they view the system.There might be important things the business analyst was told at a meeting
which to your eyes doesn't seem that key.All this helps.
It’s important as well, to try and “give up” on control, to
step aside from them machine, and “let them drive and explore a while”.We often pair up on a testers machine, and
etiquette dictates that if it’s your machine, you have to drive.It’s something we need to let go of, with a
lot of the other neuroses we've picked up around testing …
References
I don’t usually include reference to my blog pieces, but this
piece has been a real group effort, and our understanding in this area has
evolved thanks to the interaction and thought leadership from a number of
individuals beyond myself.
Instrumental amongst these have been
The Rapid Software Testing course from James Bach which really focuses on the
weaknesses of scripting and adoption of exploratory testing, and is a
road-to-Damascus, life-changing course I would recommend to any tester.
Having met Rob Sabourin at STANZ 2013, I was really pleased to hear he was returning to New Zealand to teach Software Education's Testing Mobile Applications course, which I've got one of my team on. Rob is a great, very charismatic speaker - if you ever get the opportunity, I really recommend hearing him speak, you'll not be disappointed.
Fortunately for Datacom, it turned out that Rob had a free day, and asking around I managed to get enough interest across the company to bring him in to run his one day "Just In Time" workshop, that focuses on managing your time and resources to perform risk based testing.
Unfortunately for me, this is the second course I've organised that, much as I'd love, I couldn't justify sending myself on. [Organising courses for your company doesn't mean you just put yourself forward for all the cool stuff, that would be a bit greedy]
However, I managed the next best thing, which was taking Rob to lunch, and getting to talk to him. I love catching up with other testers, whether famous speakers like Rob, people from WeTest workshops or colleagues within Datacom. Networking, and sharing your experiences and approaches with other testers is such a great way to grow your experience, as well as find allies and kindred spirits.
So we talked testing, Canada, snow (somehow the topic always turns to snow when you're talking to a Canadian), and before I knew it, fish. I started to gather fishing gear and giving fishing a go last year after being encouraged by one of my BA's at Kiwibank. I have to say sadly, I'm pretty rotten.
But for Rob, fishing in a passion - he showed me pictures barely a week apart where he'd been ice fishing, and where he'd been fishing off Florida. Ice fishing was certainly the most interesting - he's allowed to have 10 fishing rods per person in his party, and they'll sit in a heated hut, and watch their rods (because more people = more rods, he's always keen to have company when fishing it seems). With limited rods, he has to think about how he spaces them - if he gets a lot of bites, he's developed a knack for knowing if he thinks he's exhausting the fish in that part of the ice (they're getting wise, and moving elsewhere), or if it's worth to move more rods nearby.
Obviously that has all kinds of parallels with testing - we only have limited time and resources to test, and we have to make choices about where we put our rods under the software ice to fish for problems. Some areas we can choose to space out a lot, some we focus a lot more resources on. But the trick is to make an estimation, make it clear your reasoning to the rest of your fishing party, and go for it ...
Of course this brings me to another point - fishermen's tales! Rob never once talked about a fishing trip where he talked about the number of fish he brought in. Fishermen don't seem to count.
Fishermen don't tell you the number of fish they landed, they tend to tell you about one or two, but they typically have one or two things going for them - usually it's because they're a particularly rare fish, or because it was the biggest fish they'd caught that trip. No, fishermen don't tend to tell you how many sardines they land - the talk is about tuna, kingfish, marlin.
Even that, Rob tells me (I'm a rubbish fisherman remember), is a choice. If you go fishing and you want big fish, you have to make a choice, you choose the bait and hook to catch big fish - little fish will slip right through.
I was lucky to catch the last 15 minutes of this workshop - even though I thought I was experienced enough in the area Rob was talking about, he presented a few ideas and techniques which I went "ooh - never thought of doing it like that". I'm glad to have had a couple of my team on there, which means they might well get the opportunity to champion with me some new ways of doing things - it's always exciting to have the team feeling engaged and excited about new ideas!
So my takeaways for the day were without a doubt that when you know time and resources are constrained,
think about where you want to concentrate and priorities your efforts
don't communicate by counting all your defects, talk about the individual biggest issues you have
don't go out fishing for sardines on day one - look for the big fish. Worry about the smaller defects when you've overfished the big ones
.jpg)
