Back to basics 1: Testing with oracles and heuristics

Reading back over my blog for the year, there's been some interesting and philosophical items.  But following Let's Test Oz it's time to get back to some good old fashioned basics of testing!  So here we go ...

Setting out the rules and objectives

On the Tuesday night at Let's Test Oz we were talking about oracles over dinner, particularly around three fundamental aspects which are the core of most online systems now,

• registration
• log in
• account self management

Doesn't matter if you're Amazon, Facebook, Twitter or Hotmail, these elements are likely to be within your system somewhere.  So we're going to go through and explore each one, with a list of how I'd approach testing any such system.

I'm going to use as the core system under test Twitter, so if you've never used Twitter, now's a good time to get a handle.  Now obviously some screens will change once this is published, but it's likely the core basics will stay the same (so roll with it).

Oracles

Now obviously I don't work for Twitter, so I have no idea of what requirements they have for their processes.  Thankfully though requirements aren't the only way to guess how a system should operate.  There is a testing superpower I call "common sense" (which to be fair, is fairly uncommon) or James Bach calls oracles which is useful here.

As I discussed a while ago, an oracle is, "guide for understanding how the system’s supposed to behave".  Whilst it's true a requirement is an oracle, they're not the only ones.

An oracle is something that helps you to determine if a test has passed or fail.

Let's give you a sneak peak into part 3 on logging in.  You enter your username and an incorrect password.  I have no requirements here - but what do you think will happen next?  [No, I'm not going to say until part 3 - but really hope you've guessed]  And how do you know you're right?

In part because you've used other systems with these core elements in, and expect the behaviour from one to be pretty close to the behaviour from another.  And partly because it makes a kind of sense.  If you can log into an account with any old password (oops - spoilers?) - then really, what's the point of your login screen?

Heuristics

Together with some oracles to guide us, I'm going to use heuristics to generate test ideas.  Heuristics are rules of thumb used to test an element of a system.  Typically for instance when you see a name field, you'll be generating test ideas such as,

• what if I leave it blank?
• what if my name is REALLY long?
• what if I've got apostrophes and double-barrelled elements to my name?
• what if my cat runs along the keyboard and it's a jumble like 89dee8(#EH#(*E?  [If you look at the pattern those letters make across the keyboard, then it's likely the cat was really breakdancing]

All these test ideas are called heuristics, they are rules of thumb where you know systems can get into difficulty.  So they're a good idea to test.

Occasionally I meet a worried customer who'll desperately ask me "your testing will include some negative testing, won't it?".  It's the sign of a customer who has had someone work for them going "I log in with my name and password ... the login page is now tested", it means that the tester is likely to have not used enough heuristics to generate test ideas.

And let's be frank, sometimes you don't have enough time to apply every test heuristic, but you should try out as many as you have time for, often starting with the ones you know are likely to cause the most trouble!

During this series I'll be using Elisabeth Hendrickson's excellent heuristic cheat sheet as the source for a few ideas ... if you've never come across this page before, (a) print it and (b) laminate it.  It's that useful!