Friday, September 5, 2014

Reputation and engagement


I have a friend Lotz who owns and runs a record company and band in Wellington.  I've managed to catch his group, HMR Records a few times.

Lotz is a proper professional musician, and has built up HMR over several years.  They specialise is reggae, but have a stable of musicians which include a lot of genres.  As always I there's a lot you can learn in life by having around you people who have a genuine passion in life, and just listening.

A few months ago I ended up getting a lift back with him from a gig, and he told me a bit about this mixer he uses.  Sound checks for gigs are always a bit of a nightmare (heck, they're a kind of test, aren't they?).  Typically most time goes for the headline act, and that can leave very little for the other acts to tune up, especially if there are problems, when things tend to overrun.  A core part of his business model wasn't just the diversity of his group (something for everyone, and no two songs too much the same), but also that he could pre-program his mixer, so they needed minimal sound check time.

This was really important because he wanted to build up the reputation that HMR was "easy to work with" and no egos.  They wanted to build up that kind of reputation because it was important to get new work to be seen as someone that was no hassle, and could "play well with others".  No one wants to be an act after all that people go "uh, not them".  And Lotz is about one of the most relaxed people to deal with you could imagine.

Today I ended up meeting with several senior managers, and found that conversation with Lotz came flooding back.  Of course in our own way in the IT vendor industry, we're striving as individuals to get the same reputation of "easy to work with" for our companys' sake.

But what does that mean?  Does"easy to work with" mean we obey every customer request?  Even the ones which we know will cause problems down the line?  Of course not.

The trick of course is this mysterious thing called "engagement".  And there's no real absolute recipe - different people do this differently according to their personality.  Engaging means listening to what the customer wants, and potentially asking for clarification.  It's then guiding them through your experience on other similar projects, highlighting pitfalls you've experienced before, to attempt not only to mentor and enlighten them, but also to try and give counteroffers of what you can do (if you can't just say yes to their requests).

Engagement with a customer is a complex thing, because in essence it's a relationship.  And any relationship is a tightrope.  You are trying to give them the benefit of your experience, hopefully without belittling them or playing the "I'm right, you're wrong".  I know with the relationship with my wife, I've occasionally tried to use my advanced experience in science to tell her she's wrong, and as I settled down to bed on the couch that night, wondered to myself "well, that could have gone better".  No-one likes to be made to feel stupid, no matter your qualifications - and we tend to shut such people out.  The greatest teachers we know are not the ones who made us feel ignorant, but the ones who made us go "ah!", as we came to a realisation.

A comment I loved from KWST3 is a great relationship with a customer is like the movie Inception, where you're planting the seeds of powerful ideas about testing in the minds of your clients.  But the clients need to feel these ideas are their own, not ones "mandated" and forced upon them.  As I've often said, we tend to value ideas we understand more than ideas we're forced to take up, and trying to make our actions and activities understood (with clients and team) is a big part of my role.

If you're in New Zealand and reading this, I really recommend if you ever get the chance to go to a HMR Records gig, you go along - they're really great guys, and well worth seeing live!


By the way the Time Will Tell YOLO - before you think the YOLO is too gangster - it was actually a charity event run by a Les Mills PT to thank the hospice who'd looked after her father.

Thursday, September 4, 2014

Metrics and the science of measurement ...


"Physics is the science of measuring things".  These words as uttered by Mr Botham on an autumn day of my first week at Abbot Beyne Secondary School changed my life.  From that moment on I was both fascinated and hooked on the subject.

I had great teachers for physics at "The Beyne" as we called it.  Greatest of all was the friendship I developed with John Sneyd who taught me for 5 years, and put up with my many questions.  Physics held a fascination for me, and one element never left me, the fascination in measuring and observing.  It's probably the reason that testing eventually ended up feeling "the right profession" for me.

It's not a surprise then that I continued in the subject at The University Of Sheffield.  Looking back though, there were subtle things we learned year on year.  To be a good scientist you needed to know how to arrange a good experiment (preferably repeatble tests), use the appropriate equipment to measure but most of all, to understand your data.


As years went on, "understanding your data" evolved.  If any of my old log books remained, you'd notice that I always recorded answers to ridiculous levels of decimal points, "to be careful".  But any tool you measure is only as good as it's accuracy and suitability for the job at hand.  I may hand you a ruler, and get you to acknowledge it measures distance, and then go "good, so measure me out accurately 1 km".  Likewise, you'd be equally baffled if I gave it to you to measure the size of a grain of sand.

With a ruler, the best you can measure with is probably 15cm, and to an accuracy of about 0.05cm (if you've got a keen eye and steady hand).  If you're measuring a length with it can come out with a value like 16.07cm, I have to figure you're measuring less accurately than you think.

At University, this hands-on and understanding of data was crucial.  One exercise I remember was at Astronomy Lab.  I was really excited to do our first Astro Lab, thinking it's be telescopes and stargazing.  It wasn't.  We were given about 500 data points to draw in a graph.


It was boring.  And I mean really boring.  Little did I know, but we were charting our own Hertzsprung-Russell diagram (a key graph in Astronomy, as it identifies classifications of stars).

It took a few labs to put the graph together.  And I'm afraid to say at week two, I just flipped and asked our tutor (Prof David Hughes, pictured) why, "this being the far flung future of 1989" we weren't just putting all this data into a computer and have it process it for us.  I no doubt seemed impatient and dumb to my classmates, but I'm glad I did, because his answer has stuck with me to this day.  After 25 years, you'll forgive me if I don't get the words quite right, but it went a little like this ...

"You can feed a computer a string of numbers, and it can add them and divide them, and multiply them faster and more accurately than any human will.  It will even draw a mean graph line through them.  But it will never go "uh-oh, that piece of data looks out of place".  Only human beings can look at, and if need be, ignore data that could be erroneous.  If you just feed numbers into a computer without an intrinsic understanding of the data and the measurements you're using, you're essentially cutting out human judgement and intuition.  You're not here to learn how to enter numbers into a machine, but how to see those patterns for yourself, and trust to your own judgement over that of a computer."

It's actually a good point, when you're measuring, recording and handling the data yourself you get a feel for it, and intuition if you like.  If you try and measure out 1 km using a 15 cm ruler, something in your brain goes "this isn't quite right".  Give that task to a machine, and it will never figure that out, it will just carry out the task.

That intuition in good scientists develops as an understanding of error.  This idea says that between an actual value of something and the value I measure, there is always going to be a discrepancy because of the tool I use to measure.  If I go back to the simple problem of measuring a short line with my ruler - the actual line might truly be 5.648cm long, but the best I can measure to is 0.05 cm (and then only with a steady hand and keen eye).  Most of the time that's accurate enough.

If I think of the "measure out 1 km using a 15 cm ruler" scenario, you can lay the ruler out, end on end and measure out 6666.7 ruler lengths.  But you have to ensure each length is perfectly in alignment with the previous one, and starts exactly when the previous one finished.  In reality, it's going to be a small amount off, which multiplied by 6666.7 times, is going to magnify that error.



And what if I use it to measure a grain of sand?  The best I can measure with a ruler is to an accuracy of about 0.05cm, but the grain doesn't show very well against the ruler (sand from my local beach is finer than that shown).  And am I measuring length wise or width wise?  In this case my ruler, although a method of measuring distance, is once again completely inaccurate.

This of course links back to my counting fruit challenge a few weeks ago.  I asked you to imagine items of fruit in a basket as if there were test cases, some were big melons, and some small grapes.  How could just counting them help you really know how you were progressing through your bowl of fruit?  If you said that this morning there were 8 items of fruit, and at the end of the day there were 6 left, I could tell you with some accuracy that you have less fruit left that this morning.  What you can't say with accuracy is that in 3 days you'll have finished your last piece of fruit.


The problem if of course there are a good deal of test management tools out there who collect all forms of statistics on your project, and that's exactly what they DO say.  Often in pretty graphical form, and (what can kind of annoy me) to several decimal points.  The decimal points can really get my goat, because by stating it to such a level, it's implying a level of accuracy that as we've said, just isn't there.


To me with an understanding of the science of measurement and error, it feels like Prof Hughes comments come to bite us in the ass, and hard.

Of course if you're a tester, you may well be contractually obliged to use these tools, and even provide this data on a regular basis.  What I would encourage you to do is to think and learn about the error in what your tool is measuring and quoting to a high level of detail, and advise caution.  Like Prof Hughes said, that computer tool is taking those numbers, counting daily, dividing, but with totally no understanding of the data it's using.  It's model expects a single line script to run the same as a one with dozens of items.  It won't expect a "test the one hour automatic logout" to take any longer than "log in with user name and password.  You are now logged in".

If you have to use test case counts, I do encourage you to group together similar scenarios and create a form of dashboard.  A total test count of 94% of passes might be encouraging for someone to at least consider to release into production (obviously depending on the defects), however there might be a clustering of tests which have failed or simply not been run, look below ...


Well that's worrying - why have no login scenarios been passed?  Turns out under inquiry that the team is using some form of stub to enable login, because the login page is still under development.  Do you even have a product without a login page?

A dashboard at least enables people to ask why you've focused on one certain scenario, and not on another.  It allows people to look at what you're focusing on.  The "total test case percentage" doesn't encourage that kind of engagement.  Whenever possible I use a dashboard to talk with management and business owners.  As ever, James Bach has a useful set of resources and slides on them here.

But as I've mentioned, I do encourage you to think about the data you use in your reporting, and how accurate it really is.  We're often under duress to use it as "it may be inaccurate, but it's the best data we have", but I do encourage you to understand the error in those numbers, and use methods like the dashboards mentioned to give something more meaningful.

I'll just leave this here, because it's doing the rounds in my head ...


Wednesday, September 3, 2014

Failure - the secret of my success ...

As you may have noticed reading through this blog, I have a few interests outside of just testing.  I'm often trying my hand at something different, but no matter what, I try and apply a certain level of critical thinking to make it a learning and growing experience.

My mother told me something on the phone a few months ago, and it was a comment which I found really quite insightful, "you have no fear of trying things, and you're not really afraid to fail.  And that means you often can get quite good at things quite quickly, which can be difficult for people around you to take".  It's an interesting point - we publicise our success and of course are a bit quiet about the failures.

I recently went to lunch with a good friend from a previous company, where he worked on the helpdesk.  Dave is a very smart guy, and has worked as a lawyer, but thinking of going into programming.  He wanted to ask me about my experience, as I'd been fairly successful.  And it hit me like a brick, because I don't really think of myself as a successful person.  But that said, as time ticks on, we tend to leave our failures behind, and we don't really talk about them.

I got into development originally because I was a bit of a failure.  I'd trained as a teacher, and there were elements to the job I loved, and some I found emotionally crushing.  I actually spent six months unemployed during a recession in the 90s, and without the support of my in-laws, I'd have probably had to move away from the girl who'd end up becoming my wife (things like that stick with you, and why I've never had anything but love for my parents-in-law, or second-parents).


It was a tough time of uncertainty, lots of job interviews, and waiting desperately for something to come through.  I wanted to work in computers, and read as many books on programming as I could.  I have a lot of University degrees, but actually was self taught in the area I actually ended up in a career with!

I still had access to a University computer lab, so would practice UNIX and C programs there.  Life was a cycle of printing CVs and application letters, and rushing to the letterbox in a morning to see if there was any fruit from my interviews.  There were a lot of rejection letters, an awful lot.  Thankfully, eventually Thompson Marconi Sonar Limited took a chance on me, and the rest is history!  Well it certainly got a heap easier from that point on, but still an occasional rocky road.


The point is, every success story has it's difficult moments somewhere.  When Rocky finally gets to the steps, it's a moment of triumph that symbolises all the struggle he's taken to get match fit.  When we ourselves mirror that achievement, the disappointment of the first time we tried and got half way and gave up gets dissolved in the elation.  Some people, many people in fact, never return to get to the summit after their first failure.

Let's face it, we just don't like it when we fail!

Back in 2011, I took part in what was to be an instrumental training course at Kiwibank.  It was a one day workshop "for reward and motivation" - our speaker came in and promised us by the days end we would have achieved at least four feats which right now seem impossible.  And he was right!


One of the tasks was learning to juggle, and he really encouraged us to embrace the failure.  First off we took a single ball, threw it from one hand to the other, and let it hit the floor.  And then, we clapped.  We clapped because right now we were going to embrace the fact that we were going to fail, in fact we were deliberately going to fail at this point.  But we were going to keep on.  Piece by piece we tried it more and more with one ball, then two, then eventually being able to do three for a limited time.  Each time applauding the balls if they fell.  Learning started with failure, and we worked on it, just a bit at a time.

Most instructional to me was the minefield game.  There was an invisible route through a grid, and we had to take turns at it.  If we hit a bad square, we got a warning "beep" and had to back out down the safe path and let another team mate have a go.  At the end, we had an analysis, which was a bit of an eye opener.

The board looked a little like this, "Gronda!  Gronda!"

What we found was typically people would stand on a safe square and look for the next "safe" square.  And they were really hesitant.  People would often take a minute or two to decide.  If they hit an unsafe square there would be a "beep" noise, immediately that person would sigh, shrug their shoulders, and leave the board dejectedly.*

As was mentioned - there was no skill or insight needed for working out the next safe square.  There was no way you could know, so taking time to determine your next step didn't really help.  And feeling dejected because you'd gone onto an unsafe square, again, you couldn't know.  Finding an unsafe square wasn't due to lack of ability at all, and yet people still took it very personally.  In fact it could be said you'd helped, because you'd found an unsafe square for the rest of the team, making it easier for the person who followed you.

And this is how we are - we like the idea of being a natural, as if you're born with skills, being good at something is a lot more easy.  Starting off making a mess and not doing so well is really scary, because the journey is longer, and a good deal tougher.  But in truth it's the reverse, if you have some ability, it's harder to accept you STILL need to work on elements to advance!

This is why, I guess, I like to try new things.  To keep that learning part of my mind fairly elastic, lest it otherwise gets too proud and dislikes dealing with failure.  As long as it's elastic it means I get to keep trying things, and hopefully getting better at it!

My most recent challenge has been my directorial début - some friends and I have got together to make a short film called "Under The Carpet".  We're hoping to be good enough to do the 48 hour film challenge next year, and of course I want to bring a certain level of Agile mentality to our filming, most importantly "getting the project finished", over just being too obsessed with details we never make anything.

Directing was fun, educational and very very tiring.  For all involved it was their first time doing anything like it, and I'm pleased with the end result.  Though at the same time I look at some elements and go, "I wish I changed that line" or "wish we got the lighting better".  I'm avoiding the temptation to keep going back to this film and refilming and re-edit pieces.  Instead, we'll just try and learn our lessons, and make the next film that little bit slicker ...


 

Wednesday, August 6, 2014

Let's count test cases shall we?

Test cases can be as long as short as they need to be.  They can cover a lot of requirements in one go, or maybe just even one.

And yet many metrics for covering test progress seem to fall back on "what percentage of test cases are complete".

Think for a while about this fruit bowl ...


If I told you that today my team had finished off 5 pieces of fruit, how much would be left for the next few days?  I bet you'd hope (if you're the business owner) that one of those pieces of fruit was the watermelon.  But what if it was just 5 grapes?

That's what happens when you count test cases, you're saying a grape is the same amount of fruit as an apple ... or a watermelon.

If you can find a better way to track when we're out of fruit, please comment below ...

Thursday, July 17, 2014

A new defect severity hierarchy ...

Defect severity - we know them, they're "critical", "high", "medium", "low" aren't they?

Well, just for fun, I had a new idea for that, which related the emotional impact much better - and it seems I touched a nerve, and a fair few people agreed on Twitter ...


This reply though encouraged me to play out the idea a little more ...


Sounds like a challenge!

Severity 1 – OMFG
Well, that’s not good. At all. Plus my insurance details are in the glovebox.


Severity 2 – WTF
Most of its there, but it won’t go.


Severity 3 – Uh?
We can kind of drive this, but it’s not as I’d expect


Severity 4 – Meh
Well, that’s annoying. Would love it to be repaired, but won’t stop me driving.



Wednesday, July 16, 2014

Schrodingers jokes ... and some thoughts on test cases ...

Another day, another Schrodinger meme ...


Everybody loves Schrodinger, or rather Schrodinger's Cat.  Heck, I've even talked about it in the past.  The thought experiment ripples into a lot of everyday life.  The core of the experiment is, if you put a cat into a closed environment where a death trap has a 50% chance of being sprung in an hour, you can theorise all you want.  But the only way to know if the cat is alive or dead is to open it up and look inside.


The same it seems is true for many things in life - including my recent stint at stand up.  I thought a lot about my 10 minute slot, and to be honest wasn't too concerned.  After all I'd done plenty of presentations before to sometimes to difficult rooms.  This one just had to be funny!

I spent a lot of time running through lines, and building enough of a set up.  Thankfully I had some great support from friends like Rob Deb and Alex Finch who are stand up comics themselves.

Now they'd warned me of the effect I was about to experience, but no amount of being told actually prepared me.  It was scary, and more than once I found I'd lost where I was.  But the main thing was, I would build up and build up to the killer line of humour.  That sure fire gem of wit.  And then I delivered it ... to silence.

That it must be said was intimidating.  Oh I did get laughs, so didn't die on stage.  But still the places where I thought the big laughs would be, they just weren't.  And both Rob and Alex had warned me of this - to try out my material by dropping in lines to conversation, and see if it got a reaction.  But even then they said, what gets a big laugh today, might not get the same laugh tomorrow.  The bottom line though, the only way to know if all your material works, try it.

Of course this leads back to our testing - and much like Schrodinger, trying to build up a robust suite of tests, just through pure theory.  How does that work out?


Over the years I've had to do more than my fair share of what I call "test archaeology".  Nod your heads if any of this seems familiar ... a few years ago there was a bespoke project that was put together, everyone on the project has moved on (to other places, not died, this isn't a cursed project after all), and it's been in production ever since.  Only now after a few years, the customer wants some changes.


It's usually at this point that the only survivor from the original project, we'll call them Old Jethro, remembers "our testers wrote lots of scripts, and we spent a lot of money while they were there ... writing their scripts".

Test archaeology can be a fascinating business - you look through the archives what was done, and more often than not what I see is a surprise.  There can be reference to a number of script, and even a high level view of a number of these scripts and their names.  At this point one of either two things happen,

Scenario One - Script detail TBD

You open the scripts.  The format is amazingly thought out, with details of the author, the date, the requirement references.  It looks incredible.  So you look at step one, and it says "TBD".

Scenario Two - Oops

The other oft frequently encountered thing is to find the scripts, and they have just the detail you need to know how to do some odd behaviours (as there's no user guide).  You go to use it, and "that's suspicious" the instructions don't work.

So you check the execution logs, the weekly reports, the exit report.  And soon it becomes apparent that a lot of scripts were written ... but none of them actually run.  I've actually managed to collar one of those script writers and ask "what happened?".  And typically the response is "well requirements and scope kept changing ... and the release date kept moving back ... so we were asked to look busy and keep writing scripts".

I think we've all been in that boat, and tried to be coerced to do something like that!


This is how many a project that looks from the outside to be well documented and mapped out, under scrutiny can turn out to be nothing of the sort.  The problem is the over reliance on the idea that "if we have test cases, testing is easy", just put testing on auto-pilot, all will be fine.

It's hard to choose the worst of the two scenarios - the test cases which look written but aren't (scenario one) means at least you have a clean sheet to work with.  The problem is from the outside it doesn't look like "TBD", and you'll have a battle to prove why you need to spend more time planning (and even experimenting) with your testing, when it all should be done for you.  "Surely it's as simple as Ctrl+C, Ctrl+V?"

Scenario Two is by far scarier though - there are whole reams of tests which you have to spend time investigating before you can reason with the project to say why you're not doing them.  Old Jethro will keep reminding the team how much money was spent writing them, so they "darn well ought to work".

At the end of the day, a test case is really a very formalised test idea.  With a lot of the effort going into "making it formal".  The problem is, a test idea is very much like Schrodingers Cat or my stand-up material.  You can't just keep it secret in a box, and trust it's all good.  It needs to be brought out, and attempt to be used with the system to find out if it's any good or not.

Even when I've used test cases for my testing, I've always either had a prototype of the software, or an old version, so I can try out most of the testing I can (sometimes baring the new feature that may be absent), but to make sure that what I write down makes sense.

In truth, test cases don't have to be a bad thing.  But an over reliance that "we have test cases written" means "we have testing sorted" is going to get you stung when you find that Scenario One or Scenario Two applies to your precious test cases.  It's like Schrodinger saying "have you met my cat Mr Tiddles ... he's been in this box for the last four hours ...".


Tuesday, July 15, 2014

Shakespeare's lost (test) scripts ...


Shakespeare - great scripter of plays, not so great at scripting tests it would seem.

This started as a bit of fun after watching As You Like It the other week, and through reading William Shakespeare's Star Wars (basically Star Wars as Shakespeare would have written it - fun book).  What you Shakespeare's test scripts look like?



Test Script The First

Action

On approaching the steely sentinel, who lies in wait, ever watchful as Heimdall in ancient times stood in silent vigil over Asguard, render unto him both thine identity and a word most secret.  A word, which hath been arranged in private and ascertained that even in dire peril not to be diverged to man or beast that may roam this land.

Response

Avast, avast!  Look forth as the heavens rend themselves, as a choir most angelic in voice chime "hallelujah" and bid you welcome indeed with great merriment and triumph within.

Or in other words ...

You provide correct details ... you're logged in.


Test Script The Second

Action

Before you lurks sternly a being who in nature unto thee is as hostile as the Minotaur of those ancient tales unto Theseus,  and unto thee doth a grim challenge issue.  Unto him thou shouldst render your identity, and a word of secrecy most counterfeit.

Response

The Minotaur in most dire countenance doth bellow and rage, and in words most stern warns that thou shalt not be permitted to pass.

Or in other words ...

You log in with false details.  You're not let in.

Test Script The Third

Action

As intimidating as a three headed Cerberus stands the challenge betweenst thou and thy goal, to assail to the fortress beyond.  Unto him thou givest thy name and a password contrived to be most treacherously false.  Onward thy web of lies are repeated - once, twice, thrice.

Response

Upon the third falsehood, barks the dog most foul.  For ahead dost a portcullis descend from the loft of heights to bar thy way, each ratchetty clank being the sound of thy doom and abandonment.  

Oh woe lament, as a voice from the darkest pit deep down, a voice wreathed in brimstones own stench doth call out, "thy art locked out for thy falsehoods in great number.  Lament, oh lament ... and contact the help desk".

Or in other words ...

You attempt too many times to log in with false details.  You're account is locked.