Writing a kick-ass defect!

I was asked by a graduate tester about just what should go into a good defect report.  It's a really good question, and to my surprise, something I've never written about, although in a way whether we give defect reports in a written form, or a verbal form, thinking about the information you provide when reporting a defect part of the key role of a tester.  After all, if we're not able to tell someone else, that bug isn't going to go away any time soon!

Obviously, different project will have different defect templates.  However you're bound in your career to find places where you'll find none at all, so thinking about what you need to provide is a valuable skill.

I've written many defect reports, but I've also been a developer who's fixed them.  So I know what's helpful, and what's not.  Something I really encourage new testers to read is a humorous meme of supposed log book communications between pilots and ground crew.  The pilot is vague about the issues, so the ground crew are vague about their resolution back.  Don't let this happen to you!

As with most things - writing a great defect answers some very generic questions of What-How-Why-Where-When!

So let's start the ball rolling - you're using your companies new trial piece of software, and something just doesn't seem right about it.  Let's start defining it ...

What

In a nutshell, what is the problem?  This should be a real brief summary of what the problem is.

I like to think of it in terms of elevator pitches.  Imagine you've just got into the ground floor elevator with your project manager.  You get off at the 3rd floor, she gets off at the 4th.  And she asks you "hey, I heard you found a defect this morning?".

You have 3 floors, and about 20 seconds to summarise what you've encountered.  This is a real skill in summarising what you've found to a one sentence tag.  But believe me, defects like episodes of Friends are remembered as "the one with the...".

If you have a super and concise "what the problem is" summary, it probably belongs as your title.  That way, anyone reading through your defect system will go "ah".

Hint - when you go to talk to a developer about a defect you've logged, it's best not to quote just the number.  very few people remember bug 666.  That "the one where ..." summary will be the best way you have to job their memory!

How

Okay - you've summarised the problem.  But how did you cause it to happen?

The how is a way of "repeating your steps".  Ideally with a defect, you'll be able to repeat a series of steps, and repeat the unusual behaviour.

Repeatability of a defect is great.  But sometimes you just can't repeat the behaviour.  What then?  Well that's when you have to use your judgement - often depending on how much of an issue you think it was.  It's always worth talking through with a developer.

This is one of the reasons I like to use a screen recorder when I can, because it records exactly what I did and where the issues happened.  It is alright though to have defects that can't be repeated, but it's best to put on it "I'm not able to repeat".

Oh - and they say a picture is worth a thousand words, so never underestimate the power of a screenshot!

Why

Why is this a problem?  For you to be raising this as an issue you must find there's something you don't like about what you've seen.

Sometimes it's very black and white, "the requirement says the page should display your account details, and instead it displays nothing".  Likewise, if you encountered a blue screen of death, you can be pretty sure that's not "as per design".

But sometimes you might raise a defect because there is something going on which doesn't feel right.  It bugs you (another word we use for defect).

Your description of why will lead you to another description of a defect which is important - it's severity.  Typically in projects there are more defects and oddities than time to fix.  So people tend to focus on the things which are causing the most pain.

Severity is how severe a problem something is.  If you're causing your machine to blue screen regularly, that's pretty severe, and going to impact what you can test.  If you have a spelling mistake, that's less of a problem, and certainly not going to impact your testing too much.  However as it doesn't take much of a spelling mistake to make a swear word (as the test manager who emailed me with a mispelled request for "defect counts" found out).  And although it's true to say we testers live to report these kind of defects, they can actually have a functional impact - if you have a system which generates an automated email which includes a swear word, it's likely some e-mail filters are going to put it in the junk pile!

Generally though - although there are grids and standards for defect severity, I've found you just tend to pick this up through experience.  In truth, if you get everything else right about your defect and leave the severity blank, most people can choose appropriately from the information you've provided.  But experience helps you to find tune this.

Where and When

When it comes to retracing your steps where it happened and when it happened helps.

Where - well just that.  You might have a couple of test environments, so it helps to know that.  If you're on a web application, the kind of browser (and version) usually helps.  And indeed the machine.  All this information goes double for mobile devices of course!

When can be handy too - sometimes it's known for someone to be doing a release to an environment, and forget to tell testing.  Shocking huh - but it does happen.  Or indeed it allows a developer to look through the logs around about that time to see if anything peculiar was going on in the logs.

Back to basics 4: Account self management

You can find the "game rules" for this series here.

Today, we're going to jump in with some oracle ideas about account self management.  Note that some of these ideas as well as "testing expectations" could also form "design ideas" as well ...

Oracles

At my team we have a series of character cards and one of them is as below, "user who needs their account editing" ...

This is a range of functions which allow you to change some of the details of your account.  The bottom line for a company, it really helps if certain very simple functions a user can modify for themselves over ringing a support line for.  It's easier for them, and means you're not wasting money on call centre staff to do basic jobs.

Make it so a customer has to wait 30 minutes on the phone in a queue listening to Cat Stevens because they've forgotten their password, and more than likely the end result is your customer deciding they don't really need your system that badly.

Account Hijacking Danger!

There are some account details which if changed can lead to account hijacking.  This is where if someone uses a machine in at an internet cafe after you and you're still logged in, they could potentially "hijack" your account.  We'll talk a bit about those when we encounter areas where it could be dangerous.

Obviously a lot of the below depend on context of "what service you are delivering" and the level of rigour you need around it.  Lets look at some details you might want to change ...

Changing Name

People change name a lot.  The most obvious one is after marriage when it's tradition the bride change her surname to the groom's.

But there can be other reasons as well - for instance I have to admit not being too in awe of the surname "Talks", as my school life pretty much consisted of every teacher saying at the start of the year, "Michael Talks ... I hope he doesn't".  I thought I had it bad until at University I met a guy called Nicholas Lunt who'd wanted to be a teacher, but decided otherwise "because of what my name rhymes with".

Now if you're Twitter or Facebook or any social media, this should be a fairly easy thing to do.  However I do know some social media make have safeguards - you can't for instance change your name to the name of someone famous like "Kate Middleton", "Arnold Schwarzenegger" or "Donald Trump" without flashing some ID to prove that's really your name.  Facebook also has an issue with the surname "Talks", which it thinks is so rare "it's just a joke".  Meh, thanks.

The difference for this would be if you have an online bank.  For that given the ability for fraud by just changing your name, you'd want some more rigour and a "come in and show proof of name change".

Changing Date of Birth

As far as I know, there is no way you'd ever want to change your date of birth.  Okay maybe you'd want to younger, but there's no legal reason.  If your system offers you to, then this is a bit of a no-no.

Changing Gender

We can get a bit schoolboy giggly about this.  But having experienced the other side of this, and the difficulty of changing gender through friends like Violet, having the world recognise your new gender if different from birth gender is a big deal.  People who do need to be treated with respect and compassion, and not the butt of a joke.

The laws in the UK and NZ recognise the right of people to legally change their gender to that which they associate with, regardless of birth gender, so typically your system needs to as well.

Changing Password (Hijacking Danger)

Yes, it makes sense to change passwords occasionally.  But to avoid the hijack scenario, it's generally good to ask for the old password first to prevent anyone from doing it.  It also makes sense to send an email/text to say the password has changed, just in case you go "wait a minute, I did not change the password on this!".

Changing Email Address / Phone Number (Hijacking Danger)

Your email address and mobile phone number are typically used in "forgotten password" scenarios where you say you've forgotten your password, and a temporary one is emailed to you.  If an unscrupulous party sets the email to one they control, then they're just a set away from hijacking your account.

Hence it makes sense as for password to have this change password protected, but also for there to be a "changed email address" notification sent to the OLD email address.  And similar for the mobile phone number.

Hopefully I don't have to explain why it needs to be the old email address to you - if not, have a good think about it!

Similar logic to this applies to changing your mail address.


Heuristics And Test Ideas

This is the last in a series looking at oracles, heuristics and test ideas.  Our first exercise quite thoroughly set out the details for testing account creation.  The second exercise, regarding logon gave some more room for you to try things out yourself.

This time around, with those examples in your mind, and our expectations above clearly set out, you should be ready to fly this one solo!

Do use the comments below, or find me on Twitter to say how it went.  Hopefully you've found this useful.  We're really good at talking about testing, but sometimes it's useful to have a few fleshed out examples, especially if you're relatively new to testing.  Typically as the work we do is confidential, it's not like we can even "just take what we do home and post online".

Back to basics 3a: Mindmapping that last set of tests

You can find the "game rules" for this series here.

Yesterday I went through a lot of the options for creating a series of test ideas for our sample login page.

Sadly for an article like that, a lot of things had to, by necessity, be written out the long way.  As a rule of thumb looking at all those test ideas, I estimated there would be about 3 days to script it all up formally (depends on the level of detail you add).

A lot of people are proposing using mind maps to guide testing, but within Wellington, most people have come to hear of their use through talks by Aaron Hodder which you can read about here or here.

Yesterdays piece took about 90 minutes all told to write.  I managed to put much of the test ideas into this mind map in about 10-15 minutes.  Much quicker.  Although of course the mind map lacks detail, and the "discussion of why we're testing" commentary I needed to add to yesterdays document (because it's written to be an instructive guide).

Turning a test idea into a limb of a mind map is definitely "a skill" over a set of rules.  Some of those test ideas were a couple of sentences long, and a limb of a mind map won't handle that kind of detail.  You've got to be able to condense it down, put something that captures the essence of the test idea.  But will make sense when you look at it again in 2 days going "what did I mean there?".

If you don't have a Twitter account, do get one.  Twitter with it's 140 character limit is great for getting you into the habbit of condensing complex ideas into a sentence!

Aaron suggests using a mind map to help plan out and guide your testing.  If you print it out and tick/scrub out elements as you test them.  That way it shows what you've done, and what's left to test.  A useful trick, and one worth bringing to your attention.

Also, show it around (maybe even put it on the wall), and ask people if they have any better ideas.  After all, two heads are better than one - unless you're this guy ...

Back to basics 3: Login

You can find the "game rules" for this series here.

I'm looking at some core examples of website functionality and test ideas for them based on oracles and heuristics.  Last time we looked at registering / creating an account.

One key heuristic you hear in use a lot is CRUD or create, retrieve, update delete.  This is the standard lifecycle of a lot of items you can create in a computer system.

In many ways an account is no different, we

• create - when we register for an account (part 2)
• retrieve - every time we login to the system (this part)
• update and delete - when we need to modify our account (part 3)

So we're really going to do the CRUD testing over the whole series of these articles.

Oracles


Occasionally I come across someone who'll ask about testing the login "I enter my username and password, how hard can it be?". I need a meme to help me explain this, but ...

Really our oracle is not only "how we've seen login systems work before", but also our understanding of why the login exists.  The login exists to allow me access to my account, but to also keep others out.  Thats why we have the password!

So any testing needs to be around the rules of when I should be able to log in, and when I shouldn't.  With computing power being what it is now, we don't today have systems that allow unlimited tries to login.  Typically there's a rule out there that if someone forgets their password a set number (typically 3-5) consecutive times, then the account is locked.  This stops someone just nuking the account with password attempts until it breaks open.

Locking can mean different things - it could be the user has to contact the helpdesk to get it unlocked.  It could be it remains locked for a hour (limiting the hacker to 48 attempts to crack the password a day, not great but still better than unlimited).  But however, it fundamentally limits the number of tries you make, because any successful login after so many fails will be treated as a fail.

Also we need to think of not just login but also logout.  You want the system to auto log you out after inactivity.  You don't want to be able to logout, then someone log you in again by just hitting the back button.  If your login url is "yourapp/login" and your account url is "yourapp/account?userID=1234" you don't want to be able to get into the account just by changing "login" into "account?userID=1234" on the browser address bar.  [You may laugh, but it's been done].  

I have to admit the last two there "back button doesn't relogin" and "changing address bar to hack in" are blurring the line of understanding that I have between an oracle expected behaviour and a heuristic.  But in a way, it doesn't matter - you don't get exam questions on this - the important thing is that it's used to generate a test to try.

Finally, people do forget their password once in a while, there should be an option to say "send me a temporary password" via email to login to the system.

Oh - did I mention that the password should be obscured as you enter it so your neighbour can't just watch you enter it?

Heuristics

Actually not that many - there's just a few on-screen variables here.  You should probably try that the maximum length of username entered matches the maximum length from the registration page.  And likewise for the password field.

You should want to try a variety of character types used for password and username to check nothing's getting stripped or objected to.

Test 1: Happy Day or "I can login"

The following combinations will allow a user to login to their system,

• Enter their username exactly as entered into the registration page and with correct password
• Enter their username, but with different capitalisation and with correct password
• Enter their phone number starting 0064 for phone number area code and with correct password
• Enter their phone number starting +64 for phone number area code and with correct password
• Enter their email address exactly as entered into the registration page and with correct password
• Enter their email address, but with different capitalisation and with correct password

Should probably mention something about "password is always obscured", but you surely have that by now?  If that requirement shocks you, you need to try out using more web systems for familiarity!

Test 2: I can't login

If the username/phone number/email address is entered with the following, the user is not let into the system (with a suitable error message given),

• Incorrect password
• Correct password for another account
• A password which has since been changed (old password)
• The correct password but using different capitalisation
• The correct password but using the first x characters of an x+1 character length password
• The correct password but using x+1 characters for an x character length password (add a space at the end/start)
• A temporary password that has been used once.
• A temporary password, when another temporary password has been subsequently ordered.

If I give a username/phone number/email address for which there is no account, it makes sense the error message is suitably generic.  Otherwise likewise, scammers/phishers can go "this email has an account, but this one doesn't".


Test 3: Locked out

Lets say the lock out number is 5.

The basics

• I login with the wrong details for an account 5 times.  I login with the correct details, but I'm not let in. [It takes 5 consecutive failed logins to lock]
• I login with the wrong details using a combination of username/email/phone number 5 times but incorrect password each time.  I login with the correct details, but I'm not let in. [5 consecutive failed logins against the account not the handle you're using]
• I login with the wrong details for an account 5 times.  I login with the correct details once and am let in.  I login with the wrong details for an account 3 times.  I login with the correct details once and am let in.  [It takes 5 consecutive failed logins, not 4 .... and they must be consecutive]

Trying out the unlock feature

• I login with the wrong details 5 times. The account is unlocked.  I can login. [I can unlock my account]
• I login with the wrong details 5 times. The account is unlocked.  I login with the wrong details for an account once.  I login with the correct details once and am let in   [Once locked once, it doesn't trigger for every failed login since]
• I login with the wrong details 5 times. The account is unlocked.  I login with the wrong details for an account 5 times.  I login with the correct details, but I'm not let in.  [Locking the account isn't a one shot deal]

Test 4: Forgotten password

Lets order a new one!  Click the button, to get the above page.

• You should get a new password via email if you enter, the email / phone number / username there.  And by "you should" I mean you are going to test all those options ... not just one of them!  
• An unused email / phone number / username version of any of them should be rejected
• Email and username should probably be case insensitive (miketalks being treated the same as mIketALKs)

If you've ordered a temporary password,

• If you use it to log in you are successfully logged in but made to change your password immediately
• You probably should still be able to use your old one, and if you login with the old (good) password, it should make the temporary password unusable (try it)
• You should be able to copy and paste the temporary password from the email into the screen
• There should probably be a time limit on how long the temporary password can be used for.  And guess who's going to test it?
• If you order a temporary password twice, the first one should be made unusable.  And hint - the temporary password should be unguessable, not "newPassword01".
• You can't use a temporary password to login twice

Test 5:Logout

When you are inside the system, you are logged out,

• After a set period of inactivity.  Stopwatch ready, because guess what you're doing.
• If you close the tab/window, and then reopen it, you're taken to the login window to find you're logged out.
• If you select the "logout" button.

You cannot use the back button to get back into the system once logged out.  Neither can you just type in the address bar for your account and bypass the login page.

Two tier variant

Some systems have what's called a "2-tier login" system, where you use a password, then a token is sent to your mobile device (or from a token system), which you have to use.  This like a temporary password has one use, a new one can be requested (invalidating the old one), and has a limited time for which it can be used.  All variables which can be tested.

The principle is it's safer to login using a password (which could be broken) together with an item you're known to have on you (token or mobile).  If your system has an SMS stub, you're in luck, and you can use a variety of mobile numbers, but still read the token details from the raw SMS stream.

If not, then it means sadly you only ever have one phone number you can use - your own.  Suddenly that suite of phones for testing seems a good idea.

Observation

There's a heck of a lot to test there.  I've hung at work for an hour and a half whilst the trains are out writing this (we have some extreme weather in Wellington right now).  Even so it's been a lot of typing, and my fingers are somewhat sore.  To script it up in full would be at least 3 days, and for not much more benefit.

Now your turn

Have a play with your account, and again try this out on Facebook, Hotmail etc.  Get a real feel for the behaviour, and as always comment below if you think there's anything you've noticed or would like to add!

Back to basics 2a: Did you know what I missed yet? Browsers!

You can find the "game rules" for this series here.

Yesterday I was posting a series of ideas for testing a registration page.  In my head there was a really important area though that I missed.

I've been tempted to just go back an update the page, but I do like the idea of just admitting to being in a rush and missing it.  Because even with experience, we're not infallible as testers, and under deadline pressures we can end up in a rush and miss something (in my case, my son wanted to do some World Of Tanks on the laptop).

There is actually one heuristic that's missing from Elisabeth Hendrickson's cheat sheet, and it gives an indication of the age of that sheet (2006 was when it was last updated).

Device/browser heuristic

Yup - back in 2006, IE was the dominant browser, and there was a wee upstart browser called Firefox which a lot of geeks raved about.  But generally it wasn't considered to be too important or mainstream.

Welcome to 2014 - we have a host of browsers such as Chrome, Firefox, Safari.  And we still have IE, but which version is important - IE6?, IE11?, every possible version?

Do we need to test on Windows 8, Windows 7, Vista?  Do we need to test on Apple and Linux machines?

And that's not even going into mobile devices or tablets?  Are we going to choose out a few versions of iOS or Android which are important?

I covered some aspects of pure PC browser testing here and here in 2013, which are useful places to get a feel for some of the measures.  But even then I didn't cover web testing on mobile devices which themselves have their own complications - the biggest of which being it's free to install a new browser on your machine, but if you need to test on a mobile device, then you have to justify the purchase of the device for testing or else see if you can bring in a mobile specialist.

The best thing to do is to work out which are the most currently popular devices out there - even better, ask marketing for what their target market is.  Don't trust to devices you have in your testing pool to be representative of your market!  [And that market pretty much changes every time Apple makes an announcement]  A good place to look through is Synapse Mobile who have links to what are the most popular mobile device platforms at present.

I never got to meet him, but when I lived in Farnborough, Stephen Janaway was just down the road from me testing for Nokia, and he has some excellent insights into mobile testing.  One of his pieces on building a testing lab of mobiles is here.

As discussed back last year, what you're looking for when you're cross-browser testing (considering you've narrowed down your mobile and browser options to a suite to test on) is,

• Are all my fields present?
• Can I enter data/select items as expected?
• Is information displayed to me consistently?
• Does it look ugly and unusable?  Sometimes alignment can be all over the place, and it makes the screen look ugly.  And something that's ugly often undermines trust in the legitimacy of the website.
• How does the screen look on different resolution settings?
• How does the screen look on maximum vs resize of the browser?  Or when I minimise the page?
• Does the description of the page on the tab bar make sense?
• What happens if I use the back/forward/refresh buttons?
• Do all error messages get displayed appropriately?

Playing around with mobile browsers, several interesting additional areas I've found are,

• What happens if I click a drop down box?  Apple and Android have their own built in methods for dealing with these, and sometimes it can't handle drop down boxes with large amounts of text well.
• How does the mobile browser handle being turned sideways from landscape to portrait and back again during an operation?
• How useable is the keyboard in landscape and portrait mode?
• Apple and Android don't really have pop-up boxes.  Does your system have any?  How does it handle them?

Most of these test ideas will of course apply not just to the registration but all the pages ongoing!

Back to basics 2: Registration

You can find the "game rules" for this series here.

Registration is the first time that a user encounters your system - make it too difficult or painful, and they won't get past this.  Most people - and especially developers - when faced with "testing" this page will just enter their own personal details, hit "create" and if it works, then job done.

However there's certain things you'd want to try out beyond that.  Having a system that allows you to login is okay - as long as you're going to be the only person who uses the system (unlikely).  As a tester your mission is to try out a representative sample of scenarios which will cover all the other potential genuine and malicious users.

Okay - this is Twitters current registration page ...

Oracles

As discussed we don't have any requirements from Twitter, but having done this on other systems, we have a good indication of expected behaviour, and this forms our oracle.  Let's write out some of the basics,

• The system should set up a brand new account when done (but we won't test this fully until part 3) and an email sent to user
• If the system doesn't like what I'm doing, it should at least fail gracefully with an error message that gives me meaningful information
• My email/username should not have been used by anyone else before.
• The terms and conditions should be clearly readable
• All fields are mandatory - if not entered, no account will be created
• Email field must be valid according to rules from here (that page forms an oracle)
• Password will need to be of a set complexity (but exact rules unknown)
• The text you enter into the password field should be obscured to prevent someone from leaning over your shoulder and seeing it

Not at use here, but what I'd expect on this kind of page,

• I need to confirm I'm above a certain age (usually by entering your birthday)
• I need to agree to terms and conditions before creating my account (here they have put that "by selecting 'create my account' you are agreeing to our terms and conditions)

Picking out heuristics

From the cheat sheet, areas which seem to be in play here,

• strings - most of the fields are strings to enter, so most of the areas there apply to each field
• date and time - would apply if we had a date of birth field (we're going to pretend there is)
• boundaries - would apply for date of birth if we had a minimum age (again, going to pretend there is)
• input method - makes sense to ensure we can copy and paste to fields

What we're going to do now is to combine these to generate test ideas.  Don't be afraid to try them out with me.  Here goes ...

Test 1: Looking holistically ...

We're going to look initially at the system holistically, and then break down to be put each element under duress.

Let's start off with a few really basic tests ...

• Enter your details, including all the fields, confirm that account created (might need to check the database for this) and potentially an email is generated when account created.  According to the CRUD heuristic, you might also like to once created to retrieve the account in some way.  We'll cover some of this in part 3, but you might check in the back end to make sure the data entered is really that the system stores (not just for this test, but for other variants where an account is created).
• Enter absolutely no data, and try and create.  Expect an error message to be created.  [It actually 'wags' the 'create my account' box at you to show an error, but close enough]
• Create all the registration information, bar one field which is kept blank. Try to create.  It should warn you.

Test 2: Name field

Okay so it can work for some people from those tests.  Lets try out some other options.  We're going to exercise the name field under duress now.

The most obvious things to apply here from the cheat sheet are strings and input methods. So ...

• We've tried it being blank already.  But let's make it really long, and see what happens.  There are some tools out there to make really long strings.  But I like going "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMONPQRSTUVWZYZabcd...".  Typically eventually the field just stops, and I remember what letter it stopped at.  I then create the account, and retrieve the name (either from the account or by looking in SQL) and check it stops at the same letter, ie the name hasn't been unnecessarily truncated
• The field "the quick brown fox jumps over the lazy dog" is helpful as it includes all the letters of the alphabet - if you use this and it's returned in full, then there's no crazy truncation of letters or after spaces.  It's useful to also include in UPPER CASE.
• Not all names are alpha characters.  Some people have an apostrophe or insist on having a double barrelled name.  And yet their names are valid names.  How does your system handle that?
• Some people will insist on being French or German.  Yes there is a world of accented characters out there - àáâãäåçèéêëìíîðñòôõöö - how does your system cope?
• Can you copy and paste your name into the field?
• How does it handle characters you don't ordinarily associate with names?  Things like numbers or special characters like '123...' or '!@#*$(%*'.  Eliminating these characters from entry into your system helps security - particularly the * is used in Unix and SQL can can cause all kinds of mischief.
• If my name is entered and including spaces, make sure that "Mike Talks" is retrieved as "Mike Talks" and not truncated on the space down to "Mike".
• What happens if I start my name with a SPACE or use multiple spaces?

Test 3: Email field

As with the name field, you can try really long fields, copy and pasting etc.  But there are also special rules for email fields.

A lazy (but useful) basic test would be to try the valid email addresses found here and the invalid email addresses found here.  Or at least try your own variants.

As mentioned in the oracles, if you include an email address previously used, you'd expect some kind of warning.

Also would expect some "case insensitivity" in the checking of email.  If I've used an email "testsheepnz@tester.co.nz", I don't expect to be allowed to use "testsheepNZ@tester.co.nz".

Test 4: Password field

As said, blank should be unacceptable.  Typically as mentioned in oracles, passwords have to be longer than 6 characters, and include not just lower case alpha, but some upper case of numberic/special characters are mandated.

Tests for passwords which are too short or too long should be considered, or those which break the rules.  Also include copy and pasting passwords in.

Test 5: Username

Again too long/too short variants again need to be used.  If you try to go too long it should warn you, and you should try out using numbers and special characters to see whats allowed and what's discouraged.

As with email address, if you try to use an account already used it should warn you.  And likewise employ a case insensitive check to prevent you from creating "testsheepnz" when there is already a "TestSheepNZ".

Test 6: Date of birth (bonus feature)

Okay, Twitter doesn't have date of birth, but a lot of places do.

To throw some very basic spanners in the works, you should try and enter,

• A completely garbage date, confirm the system warns you it's not a valid date
• Include a date in the future for date of birth
• Try the 29th February for some valid/invalid leap year options
• Try some dates that don't exist such at the 30th February or the 32nd September
• Try first/last valid dates in the months
• Try entering dates in different date format - ie 12/01/94 or 12 01 1994 etc, to see how the system copes
• If the system asks you to select a date from a GUI, try different options, including selecting days that aren't there.

Typically most systems have business rules that "you need to be 13 or more to sign up for an account".  This is a great time to try some boundaries,

• Try someone who is minimum age to the day
• Try someone who is minimum age minus one day
• Try someone who is minimum age plus one day
• Try someone who is over 100 years old.  Pre year-2000, many computer systems stored date of birth year as a 2-digit field.  So people over 100 sometimes cause a bit of a fit in legacy systems.

Test 7: Terms and conditions

Yeah, most people just go "yadda, yadda, yadda" for the terms and conditions.  Typically you should not be able to create an accoutn unless you select you're accepting the terms and conditions.  But also they need to be displayed well.

On Twitter it doesn't seem to be very well displayed, until you click in the box, and it expands like so ...

Nice!

Wrapping up ...

As you can see just from this very basic pass, we have created a whole ream of test ideas for a very basic operation.

You can see how a lot of testers (especially context driven ones) will look at these ideas and prefer to either have a check list of them or mind map.  It's going to take a lot of time to write those all up as formal scripts, but you probably have a good idea how to execute them right now (which makes one wonder 'so what benefit exactly do I get from scripting?').

In fact, have a go at trying a few tests on the Twitter registration right now.  Then take a look at the registration pages for Facebook, Hotmail, Instagram - pretty much anything you can find (that doesn't involve a legal obligation).  Drop in the comments below any oddities you found, or indeed any different oracles or heuristics you used when you did ....

Happy testing!

Back to basics 1: Testing with oracles and heuristics

Reading back over my blog for the year, there's been some interesting and philosophical items.  But following Let's Test Oz it's time to get back to some good old fashioned basics of testing!  So here we go ...

Setting out the rules and objectives

On the Tuesday night at Let's Test Oz we were talking about oracles over dinner, particularly around three fundamental aspects which are the core of most online systems now,

• registration
• log in
• account self management

Doesn't matter if you're Amazon, Facebook, Twitter or Hotmail, these elements are likely to be within your system somewhere.  So we're going to go through and explore each one, with a list of how I'd approach testing any such system.

I'm going to use as the core system under test Twitter, so if you've never used Twitter, now's a good time to get a handle.  Now obviously some screens will change once this is published, but it's likely the core basics will stay the same (so roll with it).

Oracles

Now obviously I don't work for Twitter, so I have no idea of what requirements they have for their processes.  Thankfully though requirements aren't the only way to guess how a system should operate.  There is a testing superpower I call "common sense" (which to be fair, is fairly uncommon) or James Bach calls oracles which is useful here.

As I discussed a while ago, an oracle is, "guide for understanding how the system’s supposed to behave".  Whilst it's true a requirement is an oracle, they're not the only ones.

An oracle is something that helps you to determine if a test has passed or fail.

Let's give you a sneak peak into part 3 on logging in.  You enter your username and an incorrect password.  I have no requirements here - but what do you think will happen next?  [No, I'm not going to say until part 3 - but really hope you've guessed]  And how do you know you're right?

In part because you've used other systems with these core elements in, and expect the behaviour from one to be pretty close to the behaviour from another.  And partly because it makes a kind of sense.  If you can log into an account with any old password (oops - spoilers?) - then really, what's the point of your login screen?

Heuristics

Together with some oracles to guide us, I'm going to use heuristics to generate test ideas.  Heuristics are rules of thumb used to test an element of a system.  Typically for instance when you see a name field, you'll be generating test ideas such as,

• what if I leave it blank?
• what if my name is REALLY long?
• what if I've got apostrophes and double-barrelled elements to my name?
• what if my cat runs along the keyboard and it's a jumble like 89dee8(#EH#(*E?  [If you look at the pattern those letters make across the keyboard, then it's likely the cat was really breakdancing]

All these test ideas are called heuristics, they are rules of thumb where you know systems can get into difficulty.  So they're a good idea to test.

Occasionally I meet a worried customer who'll desperately ask me "your testing will include some negative testing, won't it?".  It's the sign of a customer who has had someone work for them going "I log in with my name and password ... the login page is now tested", it means that the tester is likely to have not used enough heuristics to generate test ideas.

And let's be frank, sometimes you don't have enough time to apply every test heuristic, but you should try out as many as you have time for, often starting with the ones you know are likely to cause the most trouble!

During this series I'll be using Elisabeth Hendrickson's excellent heuristic cheat sheet as the source for a few ideas ... if you've never come across this page before, (a) print it and (b) laminate it.  It's that useful!