Wednesday, August 6, 2014

Let's count test cases shall we?

Test cases can be as long as short as they need to be.  They can cover a lot of requirements in one go, or maybe just even one.

And yet many metrics for covering test progress seem to fall back on "what percentage of test cases are complete".

Think for a while about this fruit bowl ...

If I told you that today my team had finished off 5 pieces of fruit, how much would be left for the next few days?  I bet you'd hope (if you're the business owner) that one of those pieces of fruit was the watermelon.  But what if it was just 5 grapes?

That's what happens when you count test cases, you're saying a grape is the same amount of fruit as an apple ... or a watermelon.

If you can find a better way to track when we're out of fruit, please comment below ...

Thursday, July 17, 2014

A new defect severity hierarchy ...

Defect severity - we know them, they're "critical", "high", "medium", "low" aren't they?

Well, just for fun, I had a new idea for that, which related the emotional impact much better - and it seems I touched a nerve, and a fair few people agreed on Twitter ...

This reply though encouraged me to play out the idea a little more ...

Sounds like a challenge!

Severity 1 – OMFG
Well, that’s not good. At all. Plus my insurance details are in the glovebox.

Severity 2 – WTF
Most of its there, but it won’t go.

Severity 3 – Uh?
We can kind of drive this, but it’s not as I’d expect

Severity 4 – Meh
Well, that’s annoying. Would love it to be repaired, but won’t stop me driving.

Wednesday, July 16, 2014

Schrodingers jokes ... and some thoughts on test cases ...

Another day, another Schrodinger meme ...

Everybody loves Schrodinger, or rather Schrodinger's Cat.  Heck, I've even talked about it in the past.  The thought experiment ripples into a lot of everyday life.  The core of the experiment is, if you put a cat into a closed environment where a death trap has a 50% chance of being sprung in an hour, you can theorise all you want.  But the only way to know if the cat is alive or dead is to open it up and look inside.

The same it seems is true for many things in life - including my recent stint at stand up.  I thought a lot about my 10 minute slot, and to be honest wasn't too concerned.  After all I'd done plenty of presentations before to sometimes to difficult rooms.  This one just had to be funny!

I spent a lot of time running through lines, and building enough of a set up.  Thankfully I had some great support from friends like Rob Deb and Alex Finch who are stand up comics themselves.

Now they'd warned me of the effect I was about to experience, but no amount of being told actually prepared me.  It was scary, and more than once I found I'd lost where I was.  But the main thing was, I would build up and build up to the killer line of humour.  That sure fire gem of wit.  And then I delivered it ... to silence.

That it must be said was intimidating.  Oh I did get laughs, so didn't die on stage.  But still the places where I thought the big laughs would be, they just weren't.  And both Rob and Alex had warned me of this - to try out my material by dropping in lines to conversation, and see if it got a reaction.  But even then they said, what gets a big laugh today, might not get the same laugh tomorrow.  The bottom line though, the only way to know if all your material works, try it.

Of course this leads back to our testing - and much like Schrodinger, trying to build up a robust suite of tests, just through pure theory.  How does that work out?

Over the years I've had to do more than my fair share of what I call "test archaeology".  Nod your heads if any of this seems familiar ... a few years ago there was a bespoke project that was put together, everyone on the project has moved on (to other places, not died, this isn't a cursed project after all), and it's been in production ever since.  Only now after a few years, the customer wants some changes.

It's usually at this point that the only survivor from the original project, we'll call them Old Jethro, remembers "our testers wrote lots of scripts, and we spent a lot of money while they were there ... writing their scripts".

Test archaeology can be a fascinating business - you look through the archives what was done, and more often than not what I see is a surprise.  There can be reference to a number of script, and even a high level view of a number of these scripts and their names.  At this point one of either two things happen,

Scenario One - Script detail TBD

You open the scripts.  The format is amazingly thought out, with details of the author, the date, the requirement references.  It looks incredible.  So you look at step one, and it says "TBD".

Scenario Two - Oops

The other oft frequently encountered thing is to find the scripts, and they have just the detail you need to know how to do some odd behaviours (as there's no user guide).  You go to use it, and "that's suspicious" the instructions don't work.

So you check the execution logs, the weekly reports, the exit report.  And soon it becomes apparent that a lot of scripts were written ... but none of them actually run.  I've actually managed to collar one of those script writers and ask "what happened?".  And typically the response is "well requirements and scope kept changing ... and the release date kept moving back ... so we were asked to look busy and keep writing scripts".

I think we've all been in that boat, and tried to be coerced to do something like that!

This is how many a project that looks from the outside to be well documented and mapped out, under scrutiny can turn out to be nothing of the sort.  The problem is the over reliance on the idea that "if we have test cases, testing is easy", just put testing on auto-pilot, all will be fine.

It's hard to choose the worst of the two scenarios - the test cases which look written but aren't (scenario one) means at least you have a clean sheet to work with.  The problem is from the outside it doesn't look like "TBD", and you'll have a battle to prove why you need to spend more time planning (and even experimenting) with your testing, when it all should be done for you.  "Surely it's as simple as Ctrl+C, Ctrl+V?"

Scenario Two is by far scarier though - there are whole reams of tests which you have to spend time investigating before you can reason with the project to say why you're not doing them.  Old Jethro will keep reminding the team how much money was spent writing them, so they "darn well ought to work".

At the end of the day, a test case is really a very formalised test idea.  With a lot of the effort going into "making it formal".  The problem is, a test idea is very much like Schrodingers Cat or my stand-up material.  You can't just keep it secret in a box, and trust it's all good.  It needs to be brought out, and attempt to be used with the system to find out if it's any good or not.

Even when I've used test cases for my testing, I've always either had a prototype of the software, or an old version, so I can try out most of the testing I can (sometimes baring the new feature that may be absent), but to make sure that what I write down makes sense.

In truth, test cases don't have to be a bad thing.  But an over reliance that "we have test cases written" means "we have testing sorted" is going to get you stung when you find that Scenario One or Scenario Two applies to your precious test cases.  It's like Schrodinger saying "have you met my cat Mr Tiddles ... he's been in this box for the last four hours ...".

Tuesday, July 15, 2014

Shakespeare's lost (test) scripts ...

Shakespeare - great scripter of plays, not so great at scripting tests it would seem.

This started as a bit of fun after watching As You Like It the other week, and through reading William Shakespeare's Star Wars (basically Star Wars as Shakespeare would have written it - fun book).  What you Shakespeare's test scripts look like?

Test Script The First


On approaching the steely sentinel, who lies in wait, ever watchful as Heimdall in ancient times stood in silent vigil over Asguard, render unto him both thine identity and a word most secret.  A word, which hath been arranged in private and ascertained that even in dire peril not to be diverged to man or beast that may roam this land.


Avast, avast!  Look forth as the heavens rend themselves, as a choir most angelic in voice chime "hallelujah" and bid you welcome indeed with great merriment and triumph within.

Or in other words ...

You provide correct details ... you're logged in.

Test Script The Second


Before you lurks sternly a being who in nature unto thee is as hostile as the Minotaur of those ancient tales unto Theseus,  and unto thee doth a grim challenge issue.  Unto him thou shouldst render your identity, and a word of secrecy most counterfeit.


The Minotaur in most dire countenance doth bellow and rage, and in words most stern warns that thou shalt not be permitted to pass.

Or in other words ...

You log in with false details.  You're not let in.

Test Script The Third


As intimidating as a three headed Cerberus stands the challenge betweenst thou and thy goal, to assail to the fortress beyond.  Unto him thou givest thy name and a password contrived to be most treacherously false.  Onward thy web of lies are repeated - once, twice, thrice.


Upon the third falsehood, barks the dog most foul.  For ahead dost a portcullis descend from the loft of heights to bar thy way, each ratchetty clank being the sound of thy doom and abandonment.  

Oh woe lament, as a voice from the darkest pit deep down, a voice wreathed in brimstones own stench doth call out, "thy art locked out for thy falsehoods in great number.  Lament, oh lament ... and contact the help desk".

Or in other words ...

You attempt too many times to log in with false details.  You're account is locked.

Saturday, June 28, 2014

Farewell to my beloved grandma

Grandma's surprise visit to us when we were on holiday in Wales

In a way, I'm a very lucky man to be here in my forties and only just now having to deal with the death of my beloved grandmother in the last month.  But to be honest, I was a fortunate person indeed to have her in my life, regardless of the duration.

She was such an important person in my life, that much has never been in doubt.  However, the flood of memories that her passing brings gives time to reflect on my time with her, and the influence she leaves still to this day.

As a kid, going to spend the weekend with my grandparents was always an exciting adventure.  It was only an hour in the car, but it seemed an epic journey.  She would always be there to great us on our arrival, always smiling and ready with a gigantic hug and kiss for each and everyone of us.

She's always have something for me.  This was where the excitement began.  There would be a couple of cardboard boxes, some scissors, some pens and all the tape I could ever need waiting for me in the kitchen/dining room.  Each time the boxes would be different ... I looked through them, and when I had some ideas, would get to work.

That pile of cardboard and tape could be fashioned into anything my imagination desired.  Sometime it was a helmet, sometimes a garage for my toy cars, or secret space base for my Star Wars figures.  The possibilities were endless.  It was exhausting work, and thankfully there would be a glass of milk and some Animal Crackers for me when I needed a break.

Grandma loved to watch me at work, and for me to show her all around my inventions.  As I was at work, so was she cooking usually the some stew for our tea in her pressure cooker.  Grandad usually worked Saturdays, and when he came in from the pit, I'd start to tidy up, as we got ready for dinner, taking out the folding table putting the cloth over and setting places.

Grandad liked to joke that Grandma wasn't a great cook, and I'm ashamed to say he was right.  The stew would often be overcooked and over-peppered.  It was years before I realised that brussel sprouts weren't supposed to be yellow!  There was all of us, and often my Uncle Melvin, occasionally with his "girlfriend" Sarah (now his wife).

After dinner, Grandad would put on a record, and get to work on the dishes.  Then we'd drive the car to a nearby garage, walking back to the house with him.  Then on to the evenings entertainment - usually first would be a slide show with Grandad manning the projector, and maybe even with a reel of film.  Without doubt my favourite would be one where we'd gone to see a Wombles carnivale (and the story would always go that I had tried to run away with Madame Cholet).

After that, Grandad would power up the electronic organ - usually we'd be allowed to take one of the books of music and have a go on it.  Afterwards, he'd play and sing for us.

More than anything, my grandparents house was one filled with music.  Neither of them did anything without it being worthy of a song.  Grandma sang when she was a cook (thankfully we was a better singer than a cook), Grandad sang as he did the dishes (and he has one of the most impressive tenor voices I've ever heard), there were records like The Great Carouso being played or the organ recital.

In many ways being at my grandparents felt like living inside a musical.  You know when people say "the problem with musicals is that no-one would just break into song like that" ... well they did at my grandparents house.

All too soon, it would be time for bed, and me and my brother would willingly go up, squeezing into a single bed.  We'd go willingly, as Grandma always came with us, and we weren't really going to bed.  It was story time.  We'd squeeze into bed with Grandma, and she'd read us a book (often one of the Caspar books).  Then it was our turn.  Me, my brother and grandma would take turns telling each other stories - she'd keep asking "and what happened next".

Sunday mornings would start with me and my brother joining Grandma and Grandad in bed for hugs.  I remember always being fascinated by Grandma's dresser which included a recess surrounded by mirrors.  If you looking inside, there was a corridor stretching out to infinity in each direction.

A little like this

[It moved me that when my son was three, he was as fascinated as me by this dresser, but insisted that the mirror boys on the left "were naughty"]

We'd go down to start on getting breakfast ready.  Grandad would shave in the kitchen, singing as always, and filling the house with the smell of Old Spice, before ringing work, to get the status of the pit.  We'd then sit down to tea and oatcakes with cheese.

I'd sometimes cry, because I never wanted to leave.  I just wanted them to live next door to us, and see them every day.  They never spoiled us, except with attention.  Sometimes the people you love the most are those that just give you the tools, the space and the support to be creative.

That's just a slice of who my Grandmother was to us, but an important one.  She was the one who always encouraged us, always loved us, always was wowed by every invention and story.  And that love was unconditional - my mother said of us that we could be found by the police with a bloody knife standing over someone, and she'd go "they've got the wrong person".  Her love and faith especially in me was absolute - and I hate to say it, but as a teen or a young adult, I was unworthy of it.  But here's the thing about a love like that, if it's rare and unusual, you know how precious a thing it is.  And though you are unworthy of it, when you realise just how rare it is, you spend the rest of your life trying to make yourself worthy of it, even beyond death.

The week before I married in 1997, she was diagnosed with the early stage of Alzheimer's.  My funny, always slightly ditsy Grandmother became a little bit more eccentric each year.  But I learned how fragile her life was, and how important in the scheme of things a simple "I love you" could be.  I treasured every moment, such as walking arm in arm with her after my wedding, or the time we took my son to first visit her.  She fought hard, but in 2003, she needed to go into a care home.

Piece by piece, the Alzheimer's took her away.  From 2005 it was impossible to hold a conversation, then she could only recognise Cameron (thinking he was me).  It was heart-wrenching to loose her slowly.  Even when I last saw her, she was unable to do much but hold my hand.  But as she did so she sang, it was the smallest sign that despite everything that Alzheimer's had taken, the smallest piece of her remained.

And through it all the time was Grandad, he visited her every day.  People will tell you the greatest love story you'll ever know was Romeo and Juliet or even Kanye West and Kim Kardashian.  But to me it's my Grandad and Grandma.  Whatever Alzheimer's took, he never forgot who she was, visiting her every day and always felt blessed that she was his, on their 60th Wedding Anniversary describing himself as "the luckiest man in the world".  On their 50th Wedding Anniversary he made this clear to all, singing to her The Anniversary Song in a crowded restaurant, forcing all into stunned silence by the power of his voice, and reducing many, including myself to tears (I told you he was good).

It is only appropriate that as she breathed her last, he was at her side.  Holding her hand.

For a couple for whom music was such an important piece of their life, the music chosen for her funeral was faultless ..

Nat King Cole - Unforgettable

The happy couple - at the beginning

My grandparents with my son

At their 50th wedding anniversary

Their 60th wedding anniversary

Wednesday, June 25, 2014

Dare to be different ... take risks

Back in 2009 we managed to get to see a William Shakespeare's Midsummer Night's Dream by final year students at Toi Whakaari drama school.  It was a seriously impressive production, which moved the action into a circus setting, with the faerie characters all imagined as unearthly carny folk, giving the actors the platform to learn and display various circus skills.

This year when we found out they were running a production of As You Like It, my 16 year old son was adamant we get tickets.  And let me just say, that's unusual for a teenage boy to go "oh, I really want to watch Shakespeare".

The action started in the basement theatre of Toi Whakaari dressed as the mighty halls of power of some great Dukedom - there is an estranged brother, starcrossed lovers, a comic relief fool and a tyrannical Duke.  Okay, so far so normal for both a Shakespeare production and your average Game of Thrones episode.

It doesn't take too long before everyone is banished from the kingdom on pain of death - it's a good thing Shakespeare never created this character (well it would have made for a shorter play) ...

It's at this point that things not only got odd, they got outright surreal.  It was time for a change of scenery (I know the play well enough to know that).  I have worked on enough amateur dramatic pieces to know this is usually the point where a load of burley folk such as myself, dressed in black, come on-stage to do the silent ballet that occurs when a scene change.

But it didn't happen.  Instead, silently, out came the fool from the story, pulling a curtain at the back of stage, then unbolted one door, and then another.  There was something behind the stage, an eerie lit corridor, leading to something uncertain.  At this point someone far beyond started singing (beautifully I should add), and we the audience were ushered from our seats, through and beyond.

We just didn't know what was going on, but we soon would.  Instead of a scene change, we joined the principles in exile, being moved into a second theatre behind where the rest of the play would take place.

Photos courtesy of Toi Whakaari, photographer Philip Merry (thanks guys)

Typically the rest of the play takes place deep in the forest, but here the scene and characters were imagined as some Kiwi hippie commune.  There was a lot of fun to be had seeing some of Shakespeare's characters re-imagined as a variation of the kind of people you'd bump into at the local hipster cafe, or on an Annabelle Langbein/Country Calendar episode.

Of course all these twists and customisations can be dismissed as gimmicks if the play itself isn't very good, or poorly acted.  As with Midsummer Night's Dream we walked away feeling we'd got a professionally produced play "on the cheap", and indeed there was the "hard to pull off" achievement of making a 400 year old play so exciting to a teen veteran of XBox that he wants to know if we can see again.  That's no small undertaking!

As per usual though, this isn't just a "review of a great play", but there are some really key themes here that echo with life in the office.  There are a lot of oddities and gambles at play here,
  • Switching theatres mid-play
  • The commune setting and Kiwi-isation of characters
  • As usual with Toi Whakaari, it's a very intimately performed play, with actors sometimes not just making direct eye contact as they deliver a line, but also instead of going off stage will sit in with the audience in character.  This always gives this odd, surreal experience that you are not so much watching a play as sitting within it.  Such things are not for the faint hearted!

If this was a full professional production, there would be the temptation to cut a lot of that out - a bigger, theatre means more bums on seats, and more money yes?  And while we're at it, maybe we should curb back a few of our ideas, and make it more "as per expectations" for Shakespeare, right?

What's so interesting about both productions we've seen at Toi Whakaari is that there is an awful lot of passion on show, but also an awful lot of gamble.  And that's risky - in fact from this theatre piece from our paper it seems there was reviewer who disliked it as much as we loved it "muddled ... there are too many distracting moments from the main scene" and "what is missing is the sense, made explicit in the play, 'of the golden world', whether it was in classical Greece or Robin Hood's merrie England".

Unfortunately that reads like "why isn't it like every other production of As You Like It".  That gives a sense that the play that's performed today should be exactly as performed 400 years ago, with no new-fangled electric lights, or seating, and with all parts played by men.  [By the way, Rosalind in the story is a girl who disguises herself as a man, which means in Shakespeare's time there was a male actor, pretending to be a girl ... who was pretending to be a man]

Innovation and change of course comes from taking those gambles, and chancing/limiting when they don't quite pay off.  But it's worth it because there are no immutable rules in any field, be it theatre or testing.

Which nicely brings me to the Test Sheep's gamble of yesterday - for which you may have seen the Tweet ...

Okay - excuse the grammar (written on a phone), but you get the idea!  Yes indeed, I'm moving on for a few months to help with another project.  Unfortunately we have a very small project where work comes through somewhat ad-hoc, and when it does come through, is typically a one-person task.

The problem is that there's no assurance when work comes through that there will be the same person available to test, and only a few people have experience of the system.  For one of our key projects as discussed previously, we not only have a team culture, but a supporting handbook for the project.

The best way to onboard someone onto a new system is to talk them through it, and "let them explore and experiment" to get to know the system.  But you can't just explore without some guidance (unless the pages have help or are obvious).  Some guide is useful.

Sadly even at this point there wasn't time to write and screenshot the instructions.  This is where that XBox-generation son of mine came in, he was looking up some how-to-guides on YouTube, and I had a cunning plan!

It's true - turn to Google, and ask "how do I...", and more than likely you'll end up with a whole load of YouTube links.  Video is a method we're increasingly getting experience of using to learn.

I figured I could do a screen recording with a headset talking through the important attributes (both business and technical) as I performed several key tasks on our system (especially those non-intuitive tasks). I have to say everything on our system is documented, but that doesn't really mean if I gave all that documentation to a new tester that they'd know where to get started.

My reasoning for recording a series of screencasts was I could achieve in about a day what it would take a week and more to achieve in writing a handbook.  It sounded too good to be true.

And it was!  There were some difficulties namely,

  • Screencasting software isn't "just available to download", it costs.  I tried some software out, and hated it.  Fortunately from my stop motion work I had a copy of AVS Video Editor.  The problem was it ran (and was locked to) my very old Vista machine.  Vista!

  • Thankfully my target audience was other peer testers, because the videos are not slick, and involve some fumbling "oh wait ... I'm not on the right page".  To get something good enough to show-and-tell a customer, you do really need thorough knowledge of the system, and (shock horror) be working to a script.  So closer to a week than a day there!
  • Vista.  Well to be honest it's an old laptop, and I was running a lot of applications on it, so it's 2GB memory struggled, and occasionally the IE browser I needed to use just "stuck", but even that pailed into comparison next to everyone's favourite, the blue screen of death that would happen if my video was too long ...

This path had risks to it - mainly that by the end of it, I'd have used up a day or more, with no real gain, and be stuck trying to do the handbook again (and being a day behind on that).  To that it became important to know "how long to spend on this before I need to see results?".  Thankfully my technical problems only wasted a few hours, but I needed to be mindful if I kept having problems on when to just abandon the project and do things "more traditionally".

It was also not only untried, but even when finished, I had the feeling of "cheating".  Somehow onboarding instructions should be a Word document, we're just so trained to it like Pavlov's dogs.  Videos felt "wrong" somehow.

Yet all the same, the purpose of the material, whether video or Word document is to educate new testers.  The media through which that is achieved is not as important as the acid test of if people can learn from what you've produced.  Certainly there was a lot of Twitter support last night for the concept ....

What I realised last night is that just like the As You Like It production last night, I won't "be proved right", I'm just enjoying as we all should, and opportunity to experiment, try something new, and if it works, maybe evolve our approach.  It's not about being proven right, it's taking the leap and freedom to try something new, and by doing so not feeling trapped by convention we feel we have no control of.

In our own way, every week we should be taking risks, trying something new, but knowing when somethings "not working" and to cut our losses.  Thats how we evolve our approach to what we do, much like that first theatre that used (shock horror) female actors to play female characters!

[Of course all that got put "on drugs" by the British convention of pantomime where typically the hero is a female playing a male, and there's a pantomime dame, played by a guy.  Typically most pantomimes end with a double wedding of the (female hero) and damsel, as well as the pantomime dame and their (male interest).  Same sex marriage, and in front of the kids!  And yet it's only lately that some people are getting horrified by the concept!]

Tuesday, June 17, 2014

Testing is just "as per requirements" right?

A couple of weeks back, I was witnessed some brilliant observations by testers on Twitter that "testing is about more than as per requirement".  [I believe this was started by Michael Bolton, but I can't now find the originating Tweet].

This obviously at first is a very difficult idea to get to get your head around for some testers.  Lets face it most of our test ideas, especially scripts, are derived from these requirements.  And when we have a project without much in the way of requirement, it's quite unnerving, because we feel we're operating in a vacuum, and don't know exactly how to start.

But of course, requirements are fallible and prone to a certain type of error themselves if we take them as the only source of gospel truth.

There is an example of this which I really hope people are familiar with - if you've not watched it yet, can I please encourage you to watch this video of Spinal Tap's Stonehenge.

The real Stonehenge

If you're not familiar with it, here's the story ... Spinal Tap are a controversial (and fictional) British rock band.  For one of their songs, dedicated to the monument and druids of Stonehenge, they decide to have a prop for their act the represents the ancient stones.  So they draw a design, which their manager passes to a production company.

Stonehenge Design - your requirements, if you will

So off the design goes, until the production company comes back to the manager with what they've built.

Stonehenge Prop - are you telling me this is it?

The manager thinks it's wonderful, until he says "this is just the prop yeah"?  The production company says that no, this is it.  They built to design, 18 inches by 18 inches.  Someone it seems meant it to be 18 feet by 18 feet.  Yes, there was a fundamental flaw in those requirements.

The manager (god bless him) tries to pass off the prop, and the band end up seeing it for the first time during their song, bewildered about just why they have a version of Stonehenge that's in peril of being trambled by a dwarf ...

Even next to a dwarf, this monument does not impress

There's a serious point to this, if you hand someone a design, and tell them "just build this", you're going to probably get exactly what you asked for.  But it may not always be what you wanted.

Short of having accurate requirements or being a mind reader, how can the production team really achieve this?  This is why although requirements are an important oracle, they're not the only one.  This is why many disciplines of testing, talk about needing to understand not just what the client wants (which really is the requirements), but why they want it.

Had a discussion with the band taken place, I'm hoping that the production company would have heard, "this is for our song Stonehenge, we want it to dominate the stage, and look really impressive as these dwarves dance around it".  It's at this point, handing over the design, the company might have gone "dominate the stage ... at 18 inches?", to query and probe what they've been asked for, and hopefully deliver exactly what the customer wanted.

Of course, had that happened, we'd have lost what for me is one of the funniest comedy moments of all time ...