Tuesday, September 23, 2014

Back to basics 3a: Mindmapping that last set of tests


You can find the "game rules" for this series here.

Yesterday I went through a lot of the options for creating a series of test ideas for our sample login page.

Sadly for an article like that, a lot of things had to, by necessity, be written out the long way.  As a rule of thumb looking at all those test ideas, I estimated there would be about 3 days to script it all up formally (depends on the level of detail you add).

A lot of people are proposing using mind maps to guide testing, but within Wellington, most people have come to hear of their use through talks by Aaron Hodder which you can read about here or here.


Yesterdays piece took about 90 minutes all told to write.  I managed to put much of the test ideas into this mind map in about 10-15 minutes.  Much quicker.  Although of course the mind map lacks detail, and the "discussion of why we're testing" commentary I needed to add to yesterdays document (because it's written to be an instructive guide).

Turning a test idea into a limb of a mind map is definitely "a skill" over a set of rules.  Some of those test ideas were a couple of sentences long, and a limb of a mind map won't handle that kind of detail.  You've got to be able to condense it down, put something that captures the essence of the test idea.  But will make sense when you look at it again in 2 days going "what did I mean there?".

If you don't have a Twitter account, do get one.  Twitter with it's 140 character limit is great for getting you into the habbit of condensing complex ideas into a sentence!

Aaron suggests using a mind map to help plan out and guide your testing.  If you print it out and tick/scrub out elements as you test them.  That way it shows what you've done, and what's left to test.  A useful trick, and one worth bringing to your attention.

Also, show it around (maybe even put it on the wall), and ask people if they have any better ideas.  After all, two heads are better than one - unless you're this guy ...


Monday, September 22, 2014

Back to basics 3: Login

You can find the "game rules" for this series here.

I'm looking at some core examples of website functionality and test ideas for them based on oracles and heuristics.  Last time we looked at registering / creating an account.

One key heuristic you hear in use a lot is CRUD or create, retrieve, update delete.  This is the standard lifecycle of a lot of items you can create in a computer system.

In many ways an account is no different, we
  • create - when we register for an account (part 2)
  • retrieve - every time we login to the system (this part)
  • update and delete - when we need to modify our account (part 3)

So we're really going to do the CRUD testing over the whole series of these articles.

Oracles


Occasionally I come across someone who'll ask about testing the login "I enter my username and password, how hard can it be?". I need a meme to help me explain this, but ...


Really our oracle is not only "how we've seen login systems work before", but also our understanding of why the login exists.  The login exists to allow me access to my account, but to also keep others out.  Thats why we have the password!

So any testing needs to be around the rules of when I should be able to log in, and when I shouldn't.  With computing power being what it is now, we don't today have systems that allow unlimited tries to login.  Typically there's a rule out there that if someone forgets their password a set number (typically 3-5) consecutive times, then the account is locked.  This stops someone just nuking the account with password attempts until it breaks open.

Locking can mean different things - it could be the user has to contact the helpdesk to get it unlocked.  It could be it remains locked for a hour (limiting the hacker to 48 attempts to crack the password a day, not great but still better than unlimited).

Also we need to think of not just login but also logout.  You want the system to auto log you out after inactivity.  You don't want to be able to logout, then someone log you in again by just hitting the back button.  If your login url is "yourapp/login" and your account url is "yourapp/account?userID=1234" you don't want to be able to get into the account just by changing "login" into "account?userID=1234" on the browser address bar.  [You may laugh, but it's been done].  

I have to admit the last two there "back button doesn't relogin" and "changing address bar to hack in" are blurring the line of understanding that I have between an oracle expected behaviour and a heuristic.  But in a way, it doesn't matter - you don't get exam questions on this - the important thing is that it's used to generate a test to try.

Finally, people do forget their password once in a while, there should be an option to say "send me a temporary password" via email to login to the system.

Oh - did I mention that the password should be obscured as you enter it so your neighbour can't just watch you enter it?

Heuristics

Actually not that many - there's just a few on-screen variables here.  You should probably try that the maximum length of username entered matches the maximum length from the registration page.  And likewise for the password field.

You should want to try a variety of character types used for password and username to check nothing's getting stripped or objected to.






Test 1: Happy Day or "I can login"

The following combinations will allow a user to login to their system,

  • Enter their username exactly as entered into the registration page and with correct password
  • Enter their username, but with different capitalisation and with correct password
  • Enter their phone number starting 0064 for phone number area code and with correct password
  • Enter their phone number starting +64 for phone number area code and with correct password
  • Enter their email address exactly as entered into the registration page and with correct password
  • Enter their email address, but with different capitalisation and with correct password
Should probably mention something about "password is always obscured", but you surely have that by now?  If that requirement shocks you, you need to try out using more web systems for familiarity!

Test 2: I can't login

If the username/phone number/email address is entered with the following, the user is not let into the system (with a suitable error message given),
  • Incorrect password
  • Correct password for another account
  • A password which has since been changed (old password)
  • The correct password but using different capitalisation
  • The correct password but using the first x characters of an x+1 character length password
  • The correct password but using x+1 characters for an x character length password (add a space at the end/start)
  • A temporary password that has been used once.
  • A temporary password, when another temporary password has been subsequently ordered.
If I give a username/phone number/email address for which there is no account, it makes sense the error message is suitably generic.  Otherwise likewise, scammers/phishers can go "this email has an account, but this one doesn't".


Test 3: Locked out

Lets say the lock out number is 5.

The basics

  • I login with the wrong details for an account 5 times.  I login with the correct details, but I'm not let in. [It takes 5 consecutive failed logins to lock]
  • I login with the wrong details using a combination of username/email/phone number 5 times but incorrect password each time.  I login with the correct details, but I'm not let in. [5 consecutive failed logins against the account not the handle you're using]
  • I login with the wrong details for an account 5 times.  I login with the correct details once and am let in.  I login with the wrong details for an account 3 times.  I login with the correct details once and am let in.  [It takes 5 consecutive failed logins, not 4 .... and they must be consecutive]


Trying out the unlock feature

  • I login with the wrong details 5 times. The account is unlocked.  I can login. [I can unlock my account]
  • I login with the wrong details 5 times. The account is unlocked.  I login with the wrong details for an account once.  I login with the correct details once and am let in   [Once locked once, it doesn't trigger for every failed login since]
  • I login with the wrong details 5 times. The account is unlocked.  I login with the wrong details for an account 5 times.  I login with the correct details, but I'm not let in.  [Locking the account isn't a one shot deal]
Test 4: Forgotten password



Lets order a new one!  Click the button, to get the above page.
  • You should get a new password via email if you enter, the email / phone number / username there.  And by "you should" I mean you are going to test all those options ... not just one of them!  
  • An unused email / phone number / username version of any of them should be rejected
  • Email and username should probably be case insensitive (miketalks being treated the same as mIketALKs)

If you've ordered a temporary password,
  • If you use it to log in you are successfully logged in but made to change your password immediately
  • You probably should still be able to use your old one, and if you login with the old (good) password, it should make the temporary password unusable (try it)
  • You should be able to copy and paste the temporary password from the email into the screen
  • There should probably be a time limit on how long the temporary password can be used for.  And guess who's going to test it?
  • If you order a temporary password twice, the first one should be made unusable.  And hint - the temporary password should be unguessable, not "newPassword01".
  • You can't use a temporary password to login twice

Test 5:Logout

When you are inside the system, you are logged out,
  • After a set period of inactivity.  Stopwatch ready, because guess what you're doing.
  • If you close the tab/window, and then reopen it, you're taken to the login window to find you're logged out.
  • If you select the "logout" button.

You cannot use the back button to get back into the system once logged out.  Neither can you just type in the address bar for your account and bypass the login page.

Two tier variant

Some systems have what's called a "2-tier login" system, where you use a password, then a token is sent to your mobile device (or from a token system), which you have to use.  This like a temporary password has one use, a new one can be requested (invalidating the old one), and has a limited time for which it can be used.  All variables which can be tested.

The principle is it's safer to login using a password (which could be broken) together with an item you're known to have on you (token or mobile).  If your system has an SMS stub, you're in luck, and you can use a variety of mobile numbers, but still read the token details from the raw SMS stream.

If not, then it means sadly you only ever have one phone number you can use - your own.  Suddenly that suite of phones for testing seems a good idea.

Observation

There's a heck of a lot to test there.  I've hung at work for an hour and a half whilst the trains are out writing this (we have some extreme weather in Wellington right now).  Even so it's been a lot of typing, and my fingers are somewhat sore.  To script it up in full would be at least 3 days, and for not much more benefit.

Now your turn

Have a play with your account, and again try this out on Facebook, Hotmail etc.  Get a real feel for the behaviour, and as always comment below if you think there's anything you've noticed or would like to add!

Sunday, September 21, 2014

Back to basics 2a: Did you know what I missed yet? Browsers!

You can find the "game rules" for this series here.

Yesterday I was posting a series of ideas for testing a registration page.  In my head there was a really important area though that I missed.

I've been tempted to just go back an update the page, but I do like the idea of just admitting to being in a rush and missing it.  Because even with experience, we're not infallible as testers, and under deadline pressures we can end up in a rush and miss something (in my case, my son wanted to do some World Of Tanks on the laptop).

There is actually one heuristic that's missing from Elisabeth Hendrickson's cheat sheet, and it gives an indication of the age of that sheet (2006 was when it was last updated).

Device/browser heuristic

Yup - back in 2006, IE was the dominant browser, and there was a wee upstart browser called Firefox which a lot of geeks raved about.  But generally it wasn't considered to be too important or mainstream.


Welcome to 2014 - we have a host of browsers such as Chrome, Firefox, Safari.  And we still have IE, but which version is important - IE6?, IE11?, every possible version?

Do we need to test on Windows 8, Windows 7, Vista?  Do we need to test on Apple and Linux machines?

And that's not even going into mobile devices or tablets?  Are we going to choose out a few versions of iOS or Android which are important?

I covered some aspects of pure PC browser testing here and here in 2013, which are useful places to get a feel for some of the measures.  But even then I didn't cover web testing on mobile devices which themselves have their own complications - the biggest of which being it's free to install a new browser on your machine, but if you need to test on a mobile device, then you have to justify the purchase of the device for testing or else see if you can bring in a mobile specialist.

The best thing to do is to work out which are the most currently popular devices out there - even better, ask marketing for what their target market is.  Don't trust to devices you have in your testing pool to be representative of your market!  [And that market pretty much changes every time Apple makes an announcement]  A good place to look through is Synapse Mobile who have links to what are the most popular mobile device platforms at present.

I never got to meet him, but when I lived in Farnborough, Stephen Janaway was just down the road from me testing for Nokia, and he has some excellent insights into mobile testing.  One of his pieces on building a testing lab of mobiles is here.

As discussed back last year, what you're looking for when you're cross-browser testing (considering you've narrowed down your mobile and browser options to a suite to test on) is,

  • Are all my fields present?
  • Can I enter data/select items as expected?
  • Is information displayed to me consistently?
  • Does it look ugly and unusable?  Sometimes alignment can be all over the place, and it makes the screen look ugly.  And something that's ugly often undermines trust in the legitimacy of the website.
  • How does the screen look on different resolution settings?
  • How does the screen look on maximum vs resize of the browser?  Or when I minimise the page?
  • Does the description of the page on the tab bar make sense?
  • What happens if I use the back/forward/refresh buttons?
  • Do all error messages get displayed appropriately?
Playing around with mobile browsers, several interesting additional areas I've found are,
  • What happens if I click a drop down box?  Apple and Android have their own built in methods for dealing with these, and sometimes it can't handle drop down boxes with large amounts of text well.
  • How does the mobile browser handle being turned sideways from landscape to portrait and back again during an operation?
  • How useable is the keyboard in landscape and portrait mode?
  • Apple and Android don't really have pop-up boxes.  Does your system have any?  How does it handle them?

Most of these test ideas will of course apply not just to the registration but all the pages ongoing!


Saturday, September 20, 2014

Back to basics 2: Registration

You can find the "game rules" for this series here.

Registration is the first time that a user encounters your system - make it too difficult or painful, and they won't get past this.  Most people - and especially developers - when faced with "testing" this page will just enter their own personal details, hit "create" and if it works, then job done.

However there's certain things you'd want to try out beyond that.  Having a system that allows you to login is okay - as long as you're going to be the only person who uses the system (unlikely).  As a tester your mission is to try out a representative sample of scenarios which will cover all the other potential genuine and malicious users.

Okay - this is Twitters current registration page ...


Oracles

As discussed we don't have any requirements from Twitter, but having done this on other systems, we have a good indication of expected behaviour, and this forms our oracle.  Let's write out some of the basics,
  • The system should set up a brand new account when done (but we won't test this fully until part 3) and an email sent to user
  • If the system doesn't like what I'm doing, it should at least fail gracefully with an error message that gives me meaningful information
  • My email/username should not have been used by anyone else before.
  • The terms and conditions should be clearly readable
  • All fields are mandatory - if not entered, no account will be created
  • Email field must be valid according to rules from here (that page forms an oracle)
  • Password will need to be of a set complexity (but exact rules unknown)
  • The text you enter into the password field should be obscured to prevent someone from leaning over your shoulder and seeing it

Not at use here, but what I'd expect on this kind of page,
  • I need to confirm I'm above a certain age (usually by entering your birthday)
  • I need to agree to terms and conditions before creating my account (here they have put that "by selecting 'create my account' you are agreeing to our terms and conditions)

Picking out heuristics

From the cheat sheet, areas which seem to be in play here,
  • strings - most of the fields are strings to enter, so most of the areas there apply to each field
  • date and time - would apply if we had a date of birth field (we're going to pretend there is)
  • boundaries - would apply for date of birth if we had a minimum age (again, going to pretend there is)
  • input method - makes sense to ensure we can copy and paste to fields


What we're going to do now is to combine these to generate test ideas.  Don't be afraid to try them out with me.  Here goes ...

Test 1: Looking holistically ...

We're going to look initially at the system holistically, and then break down to be put each element under duress.

Let's start off with a few really basic tests ...
  • Enter your details, including all the fields, confirm that account created (might need to check the database for this) and potentially an email is generated when account created.  According to the CRUD heuristic, you might also like to once created to retrieve the account in some way.  We'll cover some of this in part 3, but you might check in the back end to make sure the data entered is really that the system stores (not just for this test, but for other variants where an account is created).
  • Enter absolutely no data, and try and create.  Expect an error message to be created.  [It actually 'wags' the 'create my account' box at you to show an error, but close enough]
  • Create all the registration information, bar one field which is kept blank. Try to create.  It should warn you.

Test 2: Name field

Okay so it can work for some people from those tests.  Lets try out some other options.  We're going to exercise the name field under duress now.

The most obvious things to apply here from the cheat sheet are strings and input methods. So ...
  • We've tried it being blank already.  But let's make it really long, and see what happens.  There are some tools out there to make really long strings.  But I like going "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMONPQRSTUVWZYZabcd...".  Typically eventually the field just stops, and I remember what letter it stopped at.  I then create the account, and retrieve the name (either from the account or by looking in SQL) and check it stops at the same letter, ie the name hasn't been unnecessarily truncated
  • The field "the quick brown fox jumps over the lazy dog" is helpful as it includes all the letters of the alphabet - if you use this and it's returned in full, then there's no crazy truncation of letters or after spaces.  It's useful to also include in UPPER CASE.
  • Not all names are alpha characters.  Some people have an apostrophe or insist on having a double barrelled name.  And yet their names are valid names.  How does your system handle that?
  • Some people will insist on being French or German.  Yes there is a world of accented characters out there - àáâãäåçèéêëìíîðñòôõöö - how does your system cope?
  • Can you copy and paste your name into the field?
  • How does it handle characters you don't ordinarily associate with names?  Things like numbers or special characters like '123...' or '!@#*$(%*'.  Eliminating these characters from entry into your system helps security - particularly the * is used in Unix and SQL can can cause all kinds of mischief.
  • If my name is entered and including spaces, make sure that "Mike Talks" is retrieved as "Mike Talks" and not truncated on the space down to "Mike".
  • What happens if I start my name with a SPACE or use multiple spaces?


Test 3: Email field

As with the name field, you can try really long fields, copy and pasting etc.  But there are also special rules for email fields.

A lazy (but useful) basic test would be to try the valid email addresses found here and the invalid email addresses found here.  Or at least try your own variants.


As mentioned in the oracles, if you include an email address previously used, you'd expect some kind of warning.

Also would expect some "case insensitivity" in the checking of email.  If I've used an email "testsheepnz@tester.co.nz", I don't expect to be allowed to use "testsheepNZ@tester.co.nz".

Test 4: Password field


As said, blank should be unacceptable.  Typically as mentioned in oracles, passwords have to be longer than 6 characters, and include not just lower case alpha, but some upper case of numberic/special characters are mandated.

Tests for passwords which are too short or too long should be considered, or those which break the rules.  Also include copy and pasting passwords in.

Test 5: Username



Again too long/too short variants again need to be used.  If you try to go too long it should warn you, and you should try out using numbers and special characters to see whats allowed and what's discouraged.

As with email address, if you try to use an account already used it should warn you.  And likewise employ a case insensitive check to prevent you from creating "testsheepnz" when there is already a "TestSheepNZ".

Test 6: Date of birth (bonus feature)

Okay, Twitter doesn't have date of birth, but a lot of places do.

To throw some very basic spanners in the works, you should try and enter,
  • A completely garbage date, confirm the system warns you it's not a valid date
  • Include a date in the future for date of birth
  • Try the 29th February for some valid/invalid leap year options
  • Try some dates that don't exist such at the 30th February or the 32nd September
  • Try first/last valid dates in the months
  • Try entering dates in different date format - ie 12/01/94 or 12 01 1994 etc, to see how the system copes
  • If the system asks you to select a date from a GUI, try different options, including selecting days that aren't there.
Typically most systems have business rules that "you need to be 13 or more to sign up for an account".  This is a great time to try some boundaries,
  • Try someone who is minimum age to the day
  • Try someone who is minimum age minus one day
  • Try someone who is minimum age plus one day
  • Try someone who is over 100 years old.  Pre year-2000, many computer systems stored date of birth year as a 2-digit field.  So people over 100 sometimes cause a bit of a fit in legacy systems.
Test 7: Terms and conditions

Yeah, most people just go "yadda, yadda, yadda" for the terms and conditions.  Typically you should not be able to create an accoutn unless you select you're accepting the terms and conditions.  But also they need to be displayed well.

On Twitter it doesn't seem to be very well displayed, until you click in the box, and it expands like so ...


Nice!

Wrapping up ...

As you can see just from this very basic pass, we have created a whole ream of test ideas for a very basic operation.

You can see how a lot of testers (especially context driven ones) will look at these ideas and prefer to either have a check list of them or mind map.  It's going to take a lot of time to write those all up as formal scripts, but you probably have a good idea how to execute them right now (which makes one wonder 'so what benefit exactly do I get from scripting?').

In fact, have a go at trying a few tests on the Twitter registration right now.  Then take a look at the registration pages for Facebook, Hotmail, Instagram - pretty much anything you can find (that doesn't involve a legal obligation).  Drop in the comments below any oddities you found, or indeed any different oracles or heuristics you used when you did ....

Happy testing!

Back to basics 1: Testing with oracles and heuristics

Reading back over my blog for the year, there's been some interesting and philosophical items.  But following Let's Test Oz it's time to get back to some good old fashioned basics of testing!  So here we go ...

Setting out the rules and objectives

On the Tuesday night at Let's Test Oz we were talking about oracles over dinner, particularly around three fundamental aspects which are the core of most online systems now,
  • registration
  • log in
  • account self management


Doesn't matter if you're Amazon, Facebook, Twitter or Hotmail, these elements are likely to be within your system somewhere.  So we're going to go through and explore each one, with a list of how I'd approach testing any such system.

I'm going to use as the core system under test Twitter, so if you've never used Twitter, now's a good time to get a handle.  Now obviously some screens will change once this is published, but it's likely the core basics will stay the same (so roll with it).

Oracles

Now obviously I don't work for Twitter, so I have no idea of what requirements they have for their processes.  Thankfully though requirements aren't the only way to guess how a system should operate.  There is a testing superpower I call "common sense" (which to be fair, is fairly uncommon) or James Bach calls oracles which is useful here.

As I discussed a while ago, an oracle is, "guide for understanding how the system’s supposed to behave".  Whilst it's true a requirement is an oracle, they're not the only ones.

An oracle is something that helps you to determine if a test has passed or fail.


Let's give you a sneak peak into part 3 on logging in.  You enter your username and an incorrect password.  I have no requirements here - but what do you think will happen next?  [No, I'm not going to say until part 3 - but really hope you've guessed]  And how do you know you're right?

In part because you've used other systems with these core elements in, and expect the behaviour from one to be pretty close to the behaviour from another.  And partly because it makes a kind of sense.  If you can log into an account with any old password (oops - spoilers?) - then really, what's the point of your login screen?

Heuristics

Together with some oracles to guide us, I'm going to use heuristics to generate test ideas.  Heuristics are rules of thumb used to test an element of a system.  Typically for instance when you see a name field, you'll be generating test ideas such as,
  • what if I leave it blank?
  • what if my name is REALLY long?
  • what if I've got apostrophes and double-barrelled elements to my name?
  • what if my cat runs along the keyboard and it's a jumble like 89dee8(#EH#(*E?  [If you look at the pattern those letters make across the keyboard, then it's likely the cat was really breakdancing]
All these test ideas are called heuristics, they are rules of thumb where you know systems can get into difficulty.  So they're a good idea to test.

Occasionally I meet a worried customer who'll desperately ask me "your testing will include some negative testing, won't it?".  It's the sign of a customer who has had someone work for them going "I log in with my name and password ... the login page is now tested", it means that the tester is likely to have not used enough heuristics to generate test ideas.

And let's be frank, sometimes you don't have enough time to apply every test heuristic, but you should try out as many as you have time for, often starting with the ones you know are likely to cause the most trouble!

During this series I'll be using Elisabeth Hendrickson's excellent heuristic cheat sheet as the source for a few ideas ... if you've never come across this page before, (a) print it and (b) laminate it.  It's that useful!

Friday, September 19, 2014

The Let's Test Oz experience ...

The Personal Journey


There were a lot of nerves on my behalf leading up to the Let's Test Oz conference this week.  My wife was diagnosed with severe anxiety much earlier in the year, made much worse by the death of her mother.

Leaving her whilst I went to the neighbouring country for the best part of a week was worrying, but I also knew it had potential to be good for her to know that she could cope and was more capable than she knew.  It could be a great point on her recovery, or a point where she badly relapsed.  Thankfully we had a backup plan in our number one son, who is a very capable and level-headed man at sixteen now.

It was great that at Let's Test I had a couple of people who knew this backstory - I was checking into the hotel with physical and emotional baggage - and were there to check on me and understand if I didn't want to be super-social all the time.  I think at times this support net is what the online testing community has learned to weave best.

What I'm going to do is pick up on a few of the best moments and revelations from the conference, to condence my experience ...

The Train Ride


I had a two hour train ride in the morning with tester Kim Engel, fresh from our conversation about flat earth's.  It was a great journey, and a great conversation to get me in the mood - I almost talked myself hoarse.  It reminded me one of my favourite pastimes with my son is travelling, because whether hiking or driving, it gives you opportunity to talk and explore.  With my son it's about exploring history and ideas around it.  With Kim it was exploring aspects of testing and mental health that we've both had personal journeys with.

With such conversations it's actually often a disappointment when you reach your destination, because you've enjoyed the journey too much.  Yup - I was actually slightly sad to arrive (but not for too long).

Coaching Testers Workshop


I've spoken previously about James Bach and Anne-Marie's workshop on coaching testers.  It had some great ideas circulating, and James had found some examples from movies of people coaching others.  I had to say I feel slightly ashamed that I have not yet watched the Magnificent Seven, despite liking Westerns - I need to remedy this at some point!

Great focus was given to understanding yourself as a coach, who you are and what your behaviour is.  How do you interact with people?  What's important to you in others?  Then to look at the person who is looking for coaching, and asking what they need in a coach.  Sometimes that's not you.  James and Anne-Marie talked frankly about their coaching, and that occasionally they will recommend an individual goes to the other for coaching.

As always with a good workshop, this included reams of hands-on.  We logged in anonymously to Skype and we got to work with other people in the room, taking turns to be the coach and to be the student.

I'll be absolutely blunt, I thought the person I was coaching knew me, and was playing a game.  I kept getting incredibly frustrated but trying to be calm with them.  When it came to debrief it turned out that the person I paired with had never used Skype before and was a slow typist.  I learned a valuable lesson there that especially online you need to get some form of check-in about the student, how they feel and what their backstory is, rather than "leap into" the coaching.

It's a lesson I really should know - but it's amazing how the session helped to reinforce that, and instead I made my own assumptions that it was someone trolling me.

So you wanna be a boxer?


SoftEd ran a couple of boxing training sessions which were absolutely superb.  I occasionally do something similar in Wellington (I've talked about Josette, one of our instructors here).



Boxing training is a very intimate kind of training.  You pair up with someone, and take turns doing exercises,with one person using gloves to hit, and another holding the pads being hit (you hit the pads, not the person!).  You have to both be mentally in a similar zone, and develop a kind of rhythm with each other.

That makes it oddly quite a social activity.  The best kind of pairing is when you're both supporting the other with "try moving your stance", "we're half way, don't give up", "c'mon, keep going, nice" - it was some of Bach and Charrett's coaching tips in miniature.

Half the success of any conference like this is being able to mingle with people who you don't know.  Meet new people, make new allies.  The boxing sessions proved to be a great "meet and greet" event, with conversations with other boxers strung out through the rest of the conference.

Which brings me to this tweet ...


The boxing was one method (I'll talk about the other in a moment), but there were the experienced keynote speakers touring the conference, and there were other speakers such as myself.  But mingling and listening I realised something important, "everyone has an experience report and a story inside them, just keep your ears open".

As I said in that tweet, in some ways listening to the raw stories from others was a great opportunity to really spread my net over the conference.

Put yourself around people with passion


The other method of putting myself around a bit came at lunchtime.  It all felt like being a bit at High School with "who shall I sit with?".  Occasionally I just really needed to eat and dash, and I'd just sit alone.  But I tried to use it as an opportunity to sit with new people, and introduce myself.

I however earned myself the "special little snowflake" achievement on Monday lunchtime though - finding a table of people I didn't know, and asking if it was okay to sit with them.  They were from a completely different conference!

But this actually led to a bit of a revelation.  They were a conference of ultrasound operators and were curious about who we all were.  So I got to tell them a little about software testing, as well as ask them a bit about their conference.

My interest was hightened a bit.  Recently my brother and his wife had a daughter.  One thing that surprised me was the ultrasound. When my son was "under development", we had a couple of ultrasound pictures of him, and they were a bit like one of those 3d puzzles.  If you stared at really hard, you couple perhaps make out a skull.  Ish.

But for Thea, her pictures were strikingly clear, the technology had really come along leaps and bounds.  So surely being an ultrasound operator was a lot easier now?  Wrong.

Turns out they do more and more checking with ultrasound as it's such a non-invasive proceedure.  They were spending a conference looking at example images of different ailments such as a damaged appendix.  In their normal routine they might not see some of these examples, so it was all about improving their ability to look and and recognise issues.  They were using the conference to broaden their experience from other operators, so when they went back to work on the Thursday they were just that bit sharper.

The improvements in technology made some things a lot easier.  But at the end of the day it required a human eye and human judgement.  And damn it - wasn't the same true about testing's relationship to technology over the same timeframe?  Some things had got easier, but at the end of the day, it's about a human eye and human decision making.

This led me to an important understanding - I learn a lot about testing, but not always from testers.  I'm quite a talker, but I'm a good listener too.  In fact when we go touring around New Zealand, my wife despairs of me as I really enjoy going into quiet shops and talking at length to the owners, where we come from and finding out some of their history.

I am attracted to spending time with people with a passion, and an ability to animatedly talk about that passion.  It doesn't have to be anything I'm interested in - in fact often it helps if it's not.  Just this year I've spun pieces off from conversations with Josette my boxing instructor and Lotz my musician friend.

Testing has a good few parallels, and if you listen out, there is knowledge out there from people going through similar experiences which you can Shanghai, and add to yours!

An interesting chat with Erik Peterson leads to some self-reflection ...


I had an interesting lunchtime chat with Erik Peterson where we talked about heuristic models for testing.  I came to the realisation that I'm heavily dependant on an "experiental model" (although I do use others) - basically "when I used to program, I once saw this happen" or "I've seen a bug like this in a similar system".

That's of course okay, as long as you realise it's fallible.  And it's greatest fallibility is you aren't looking for a bug you've never experienced before - you have a blind spot to anything you've never seen or heard.  It also made me look at some of my writing - overall my writing heavily leans towards a "series of experience reports", occasionally postulating a model from this experience.

It's an interesting look in the mirror at my way of thinking - also tying into my previous post about trying and failing.  It's like I'm drawn to having a pool of experiences to base judgements on.

My talk


Yup - I'm not being egotistical here, but not only did I enjoy giving my talk on "deprogramming the cargo cult of testing", but to my shock, I walked out with an expanded take on it.  Some of the questions asked allowed me to think and explore the subject in ways I'd not expected.

The topic was really talking about the system of testing we've put into place over the last 12 months, and I talked about it back in my piece on exploratory testing.  We put together a new way of testing when we moved to being agile, but we engaged with our customer to talk to them about what they felt they got out of our old methodology?  What did they feel they get from a test plan, a test script or a test report?

The point was this was to form a matrix of values from our customer - this meant whatever approach we took for testing it needed to address these values in some manner.  If it didn't then we weren't done with our approach, it wasn't hitting the needs, and we needed to rethink.  But not only that, we had to make sure we were making "how our testing worked" visible to the customer.

An example of this would be how the customer saw test scripts both as "proof of testing" and "training material".  We ended up using qTrace to record our sessions as "proof of testing" and for "training material" sharing an internal testing handbook we already had, and making sure we kept it up to date sprint on sprint.

Someone noted the piece tied in a bit with Keith Klain's keynote where he talked about avoiding being overly whiny or self-centred about testing's problems, but understand the person you report to "has problems and needs" that you don't know of.  To try and go to them, not with more problems, but trying to help and aid them.

The bottom line to this approach was that we made sure we had an evangelical fervour to delivering real value to our customers, in a testing approach that we felt accurately addressed their needs.  In this I really was pleased we seemed to carry on the spirit of Alessandra Moreiraa's talk about engaging and influencing people.  In fact the conclusion from our talk was it would be a mistake to wait for a major shift from waterfall to agile before engaging with a client to ask if the testing you're performing is really "ticking the boxes" from both the client and the test team point of view.

Final thoughts ...

These are the things that really stuck with me - a very interesting conference, with a lot to take home.  I sadly missed the Fiona Charles keynote at the end of the Wednesday, which I was looking forward to.

There was a lot to take in, but also fun to be had along the way - one of the funnest activities being Joanne Perold and Carsten Feilberg's workshop where we replicated problems in communication by using a Lego building exercise to replicate the the software building process.  This was an exercise I would love to try again with different rules to see if it causes some of the outcomes I expect.  Likewise the boxing and the coaching activities were nicely hands on.

The team behind Let's Test Oz really did an excellent job in making this happen - the venue and food was amazing, everything ran well, and everyone seemed to come to the conference ready to really share and engage.  I made sure before writing this, I sent an email to the key players, asking them to circulate to all who needed to read it.

Thanks guys!

A tale of two tutors ...

Several of my recent blog posts have talked back to my experiences at University, and I'm somewhat shocked to find out that 25 years have since passed, and yet I still feel I'm out on an active path of learning.

University was a formative time for me in a lot of ways, as it is for many people. Apart from the independence aspect, it's also a time where you learn there isn't a textbook syllabus anymore.  Instead there's a whole world of different ideas and theories in your chosen discipline, many of which contrast, and education at this point becomes trying to reason between this knowledge.

An important role in shaping me as a human being through these years came from two very different individuals who were my tutors in physics and astronomy at the University of Sheffield, and this post is about who they were and where that influence came from ...

Professor Fred Combley



Fred was my tutor for physics.  It's hard to think of him as "Prof Combley", he was Fred, a very warm and gentle person, whose office was almost always open to any student.

His approach for our third year course in cosmology was really unique.  He didn't teach a single lecture. We did!

Now that might seem like a very lazy lecturer, but it was both inspiring and revolutionary.  Each week, three students were given an area to prepare a lecture for, and we'd present the next week.  We'd have to go to the library to research, and we had a session booked with Fred to discuss the topic.  We were in fact given the keys to our own learning.

I loved doing it, and loved sitting and exploring the topic with him.  Versus the many other lectures we attended that seemed to get droned out by writing out mathematical formulae by rote, this was us, the empowered student, picking out what was important and what to say.  And the information from those lectures stuck.  I loved it so much that it led me to trying teaching, and is one of those reasons why I'll almost always put my hand up to speak or present when given an opportunity.

But what stuck with me most of all was as I said was his warmth.  I wasn't a model student at all.  At school I was the top of the class in physics, but at University within a group of my peers, I was decidedly below average at the subject, and that caused me a lot of stress.  I felt almost robbed of my identity.  In fact I failed exams in my first two years and had to do resits (I did say that failure was my natural style).  But with perseverance I got through - although despite not being a star student, as I said Fred always had a time for me, and more respect than I felt I deserved at the time.  I always felt he believed in me, even when I didn't believe in myself, and that was important getting through some tough times.

I bumped into him again three years later - I was popping back to the University of Sheffield to have some lasers etched for my project at the University of Essex.  [Essex was partnered with Sheffield, who had superior clean room facilities].  He saw me in the corridor, and to my amazement remembered me (as a physics tutor I imagined I was one in a sea of faces), and dragged me into his office to catch up - he was very pleased to hear I was still in physics and enduring (although I was still failing exams).

Thanks to the wonder of the internet (Fred was involved with CERN, so got it first), we kept in touch afterwards, with an email a year to catch up and tell him about my new career in software, and the interesting things I was learning.

Then one day in 2001, I realised he was no longer on the staff at Sheffield University, and smiled as I realised he must have finally retired.  Sadly no, as another of the teachers at Sheffield, Dr Susan Cartwright had to inform me that he was due to retire, but alas an aggressive cancer had killed him before that could happen.

I was gutted.  Anyone who knew him couldn't help but be gutted ...

Prof David Hughes


David Hughes was my tutor for astronomy and in many ways the polar opposite to Fred.  I mentioned him of course in previous blog posts ...

David was a hero of mine, I'd seen him speak on astronomy on The Sky At Night, and he was simply brilliant in the televisation of the Giotto probe intercept with Haley's Comet.  I've put up links before (such as this) of him speaking - he is without doubt the greatest speakers I've ever heard, able to convey incredible concepts in a simple and entertaining form.

But there was a problem - we really didn't get on well, at all.  I talked in my previous post that when we were asked to plot out a graph of 200 points, I asked why we weren't using a computer for the task, to which he replied ...

"You can feed a computer a string of numbers, and it can add them and divide them, and multiply them faster and more accurately than any human will.  It will even draw a mean graph line through them.  But it will never go "uh-oh, that piece of data looks out of place".  Only human beings can look at, and if need be, ignore data that could be erroneous.  If you just feed numbers into a computer without an intrinsic understanding of the data and the measurements you're using, you're essentially cutting out human judgement and intuition.  You're not here to learn how to enter numbers into a machine, but how to see those patterns for yourself, and trust to your own judgement over that of a computer."

In that blog, I mentioned how inspirational that quote is, and how I return to it again and again.  But what I didn't tell you was how I responded to it at the time.  I hated it, and I didn't like him for what he said.  I felt belittled over my idea to use a computer program (I was good at using computers for such tasks) and annoyed at him over it.

It didn't help that unlike Fred, David seemed to have his favourite students to whom he was always joking and telling funny stories to.  However when I had to spend time with him, he always seemed to be being on my case and being incredibly pushy.

It took me years though to realise that compared to those student who got the funny tales, that I got the best from David Hughes.  I had excuses galore to hide behind, but he had a habit of blasting them away, and not letting me settle for "not giving it a go", frequently pushing me with "you can do better than this".

Another conversation I remember having was of me whining about a 1000 word essay we had to complete in a week, and how that was almost impossible.  He told me bluntly that if I did my research and got the ideas in my head, I could easily sit down an knock out that length (as he often did for astronomy magazines) in just a couple of hours.

And damn, he was right!  I've been working on this blog post for about an hour, and Microsoft Word tells me I'm over 1200 words.  Every time I do a word count on a blog or magazine piece, I just remember that conversation, and go "damn, you're right again!".  So it's fair to say he has been a major influence in my writing on testing.

But he wasn't always so gruff, when I started to appreciate how he'd not let me sabotage myself with ideas that "it can't be done", I also remembered him talking to me when I failed an astronomy paper, telling me not to let it worry me too much, and it was a mark that we could work on and "make alright" next time.

Unlike the tragic story of Fred, Prof David Hughes is retired, but still talking about the subject with which he has boundless passion, and has a nice gig doing astronomy lectures on cruise ships ...

Models of tutoring and coaching

These two characters were going through my head for the last week, and I've realised how much in any mentor or coach we need elements of both of them.  David's pushiness to not allow me to settle for second best or to sabotage myself with negativity before I'd begun and Fred's gentle nature to help pick you up and gently say "well, what do you think you can do better?".  A mixture of hardness and softness dictated on which is needed to coax out the best from an individual.


Everyone probably need a bit of pushing and a bit of picking up, but the focus for different individuals is completely different, because those people, how they view themselves is different - making learning and coaching itself a contextual activity. It makes me think of Shifu, the master and trainer in Kung Fu Panda.  He originally tries to teach Po how to do Kung Fu in the exact same manner he's trained all his other students.  But the training is disastrous, and Po eventually gets really dejected.  However when Shifu sees Po in the kitchen climbing shelves to find cookies, he realises that Po has more raw ability than he originally thought.  He comes up with a training scheme different to the others, and which focuses on what motivates Po.

In many ways, all these elements turned out of be part of the model that James Bach and Anne-Marie Charrett presented at Let's Test Oz 2014 as part of their workshop on coaching testers.  However instead of the two data points in my life called "Fred" and "David", they've been researching, trying out models, and looking for fundamental truths about what makes good coaching.  I'm going to resist the urge to paraphrase them but it was without doubt the session at Let's Test that I found myself taking the most away from.  If you have the chance to ever attend their workshop in future, I recommend you get yourself along to it!

[And if you can't manage that, hound them to write a book about it]