Alice took a quizzical look at the Cheshire Cat, “so you’re
a tester? But you’re a cat!”.
“Oh,” the Cheshire Cat beamed, his almost rictus grin fixed permanently and inflexibly across his face, “in your world that might be a problem, but not in
this one. Did you have a better candidate in mind?
Would YOU want to sit in a meeting with the Mad Hatter, changing his seat
every few minutes, or tell the Queen of Hearts that the project might have to
be delayed?” He purred a very
self-satisfied purr to himself. “No … I
think you’ll find in reality, or as close as it gets in this place, I’m the
perfect person for the job.”
Alice looked at the heap of worn leather bound books that the cat had seemed, almost impossibly, to leasurely recline on top of. They came up to Alice's shoulders, and she ran her eyes down the spines ... “What are all those books?”, she
inquired.
“Oh THESE? This is
our documentation … we’re all ISO 29119 compliant over here you know. You try telling the Queen of Hearts it's not needed. Actually my last few predecessors
did just that, and that’s how I came
into the role, if you get my drift,” at which the cat extended his claws and
made a mimed guillotine motion against his own throat, “but then of course business can be so cut throat, especially,” and he giggled to himself, "when it comes to the subject of test execution."
Alice took a book at random from the pile that said “test
plan”, and started to read it. “But this
is all a nonsense, you’ve just written ‘all work and no play makes Jack a dull
boy’ repeatedly with a few headings!”
“And yet it is to the standard”, beamed the Cheshire Cat
ever smugly, “the standard says we must have a test plan … so we got a book and called it the test plan. It mandates we
should have certain titles within that book as well, and being compliant, we have those as well. But then …”, and the cat began a very
self-satisfied purr, “nowhere does it say that it has to be based on any kind
of reality, which as you know in this place is the hardest type to find. And as the regulation says, we are allowed to tailor the actual contents as we most see fit ... as long as it follows a template.”
The cat stood up, gave a little stretch, and started to do
that annoying vanish from the tip of his tail, until all that remains was his
smile. That smile still beamed as he said, “and do you know, I think this template suits me just
fine …” And with that he was gone.
My family are a fairly spiritual one - my parents were after all hippies when I was born, and hence an awareness of ourselves, and our role within the world to make it a better place was an important part of our education.
My parents tried out a few churches as they grew older to find one which felt right - spiritual, and yet we were an engineering family. Somehow religion and science really needed to be good bedfellows - so we ended up joining Winshill Methodist Church.
The Reverend there were a very passionate man named WH Pittam, who just had a great way with the congregation. There are few men of the cloth who'd so relish playing Barabbas (the man who's freed instead of Jesus) in the Easter Story drama.
He could also do a mean sermon, and aim it at children. One of the ones I remember being a favourite was the one about a lamb who'd had enough of being a sheep, and decided it wanted to be a lion. And so it went through a transformation to make itself more of a lion - it got a wig to make a shiny mane of hair, it got a pair of vampire teeth to look more fierce, if tried to practice it's roar, and even went to far as to put a sign around it's neck to say "LION". But it didn't really work. It could come close to making itself appear more like a lion, but in truth it was no closer to being an actual lion.
Certifications for all!
There is something quite close to this in the Wizard Of Oz - the Scarecrow, Tin Man and Lion go to the Wizard to get some trait they think is missing. But their encounter and trials with Dorothy brings these abilities out of themselves naturally. In the end, the Wizard is exposed to be a snake oil salesman/con man. And he still has a trick up his sleeve - yes the trio have found their own brains, heart and courage, but he gives them trinkets - the Scarecrow a paper certification (to show he has knowledge), the Tin Man a token of appreciation (to show he's a philanthropist) and the Lion a medal (to show he has courage). The odd thing is the Wizard has done nothing really, all he's given each individual is a token to show others they have a trait that they already know they had. Yet in the movie, everyone's most thankful (even though the Wizards real plan was to send them on a suicide mission).
So why does any of this matter?
Because it's happening right now, in testing. I've spoken before about the potential pitfalls with ISTQB qualifications being a way to measure if someone is a "tester". In the Wizard Of Oz, when the Wizard produce a paper certification, did the Scarecrow really get any smarted from it? The Scarecrow showed his smarts through his trial with the Wicked Witch Of The West. The Wizard didn't help or aid or mentor, be just pulled out a certificate.
The Wizard though said the certificate was important because it would show other people that the Scarecrow was smart. But the truth is the Scarecrow had proved that smarts himself, and he owned that intelligence as his own - why should someone unconnected and unqualified essentially steal that with a certificate? Someone really should have taken the Wizard to task.
But now we have a new issue on the testing horizon - ISO 29119, the standard on "software testing", and it has a lot of people in an uproar. It's been doing the rounds for a while, and supposedly "been out for feedback". People who have traced the standard have noticed that in all that time there doesn't seem to be much change occurred at all, although there has certainly been no shortage of comment.
The issue is it attempts to implement an idealised "best practice" to software testing as a mandatory standard, and encourages customers to make sure their vendors are compliant. This is a great idea - we can follow an idealised model if the whole project delivers to testing following the strict idealised model for requirements and developments. The harsh fact is though that projects are tailored around objectives, constraints and contexts which will take them in different ways off that "idealised path", and testing needs to flex with the rest of the team to find the best pragmatic method to match the project as it's being delivered. To take on a rigid standard and sit in our ivory tower complaining that the project hasn't been delivered correctly for testing (as the ISO standard seems to encourage) reeks of unprofessionalism, and it really does not serve our customers.
It has potential to be detrimental to our industry, to the minds who make up it, and as said to the teams and customers we have skin in the game with. The only people who seem to benefit from this are those who will be in charge of selling and auditing the standard.
There is a petition to force a rethink of the ISO 29119 standard, and I really encourage you to read through and make your own mind about this. As before all I can encourage you to do is read the facts and make up your own mind. As a tester, this standard will impact you. Much like our lamb, having a sign saying "lion" won't make it a lion. Having a certificate with a tick won't ensure your testing is really the "best practice" being followed for your customers.
I have a friend Lotz who owns and runs a record company and band in Wellington. I've managed to catch his group, HMR Records a few times.
Lotz is a proper professional musician, and has built up HMR over several years. They specialise is reggae, but have a stable of musicians which include a lot of genres. As always I there's a lot you can learn in life by having around you people who have a genuine passion in life, and just listening.
A few months ago I ended up getting a lift back with him from a gig, and he told me a bit about this mixer he uses. Sound checks for gigs are always a bit of a nightmare (heck, they're a kind of test, aren't they?). Typically most time goes for the headline act, and that can leave very little for the other acts to tune up, especially if there are problems, when things tend to overrun. A core part of his business model wasn't just the diversity of his group (something for everyone, and no two songs too much the same), but also that he could pre-program his mixer, so they needed minimal sound check time.
This was really important because he wanted to build up the reputation that HMR was "easy to work with" and no egos. They wanted to build up that kind of reputation because it was important to get new work to be seen as someone that was no hassle, and could "play well with others". No one wants to be an act after all that people go "uh, not them". And Lotz is about one of the most relaxed people to deal with you could imagine.
Today I ended up meeting with several senior managers, and found that conversation with Lotz came flooding back. Of course in our own way in the IT vendor industry, we're striving as individuals to get the same reputation of "easy to work with" for our companys' sake.
But what does that mean? Does"easy to work with" mean we obey every customer request? Even the ones which we know will cause problems down the line? Of course not.
The trick of course is this mysterious thing called "engagement". And there's no real absolute recipe - different people do this differently according to their personality. Engaging means listening to what the customer wants, and potentially asking for clarification. It's then guiding them through your experience on other similar projects, highlighting pitfalls you've experienced before, to attempt not only to mentor and enlighten them, but also to try and give counteroffers of what you can do (if you can't just say yes to their requests).
Engagement with a customer is a complex thing, because in essence it's a relationship. And any relationship is a tightrope. You are trying to give them the benefit of your experience, hopefully without belittling them or playing the "I'm right, you're wrong". I know with the relationship with my wife, I've occasionally tried to use my advanced experience in science to tell her she's wrong, and as I settled down to bed on the couch that night, wondered to myself "well, that could have gone better". No-one likes to be made to feel stupid, no matter your qualifications - and we tend to shut such people out. The greatest teachers we know are not the ones who made us feel ignorant, but the ones who made us go "ah!", as we came to a realisation.
A comment I loved from KWST3 is a great relationship with a customer is like the movie Inception, where you're planting the seeds of powerful ideas about testing in the minds of your clients. But the clients need to feel these ideas are their own, not ones "mandated" and forced upon them. As I've often said, we tend to value ideas we understand more than ideas we're forced to take up, and trying to make our actions and activities understood (with clients and team) is a big part of my role.
If you're in New Zealand and reading this, I really recommend if you ever get the chance to go to a HMR Records gig, you go along - they're really great guys, and well worth seeing live!
By the way the Time Will Tell YOLO - before you think the YOLO is too gangster - it was actually a charity event run by a Les Mills PT to thank the hospice who'd looked after her father.
"Physics is the science of measuring things". These words as uttered by Mr Botham on an autumn day of my first week at Abbot Beyne Secondary School changed my life. From that moment on I was both fascinated and hooked on the subject.
I had great teachers for physics at "The Beyne" as we called it. Greatest of all was the friendship I developed with John Sneyd who taught me for 5 years, and put up with my many questions. Physics held a fascination for me, and one element never left me, the fascination in measuring and observing. It's probably the reason that testing eventually ended up feeling "the right profession" for me.
It's not a surprise then that I continued in the subject at The University Of Sheffield. Looking back though, there were subtle things we learned year on year. To be a good scientist you needed to know how to arrange a good experiment (preferably repeatble tests), use the appropriate equipment to measure but most of all, to understand your data.
As years went on, "understanding your data" evolved. If any of my old log books remained, you'd notice that I always recorded answers to ridiculous levels of decimal points, "to be careful". But any tool you measure is only as good as it's accuracy and suitability for the job at hand. I may hand you a ruler, and get you to acknowledge it measures distance, and then go "good, so measure me out accurately 1 km". Likewise, you'd be equally baffled if I gave it to you to measure the size of a grain of sand.
With a ruler, the best you can measure with is probably 15cm, and to an accuracy of about 0.05cm (if you've got a keen eye and steady hand). If you're measuring a length with it can come out with a value like 16.07cm, I have to figure you're measuring less accurately than you think.
At University, this hands-on and understanding of data was crucial. One exercise I remember was at Astronomy Lab. I was really excited to do our first Astro Lab, thinking it's be telescopes and stargazing. It wasn't. We were given about 500 data points to draw in a graph.
It was boring. And I mean really boring. Little did I know, but we were charting our own Hertzsprung-Russell diagram (a key graph in Astronomy, as it identifies classifications of stars).
It took a few labs to put the graph together. And I'm afraid to say at week two, I just flipped and asked our tutor (Prof David Hughes, pictured) why, "this being the far flung future of 1989" we weren't just putting all this data into a computer and have it process it for us. I no doubt seemed impatient and dumb to my classmates, but I'm glad I did, because his answer has stuck with me to this day. After 25 years, you'll forgive me if I don't get the words quite right, but it went a little like this ...
"You can feed a computer a string of numbers, and it can add them and divide them, and multiply them faster and more accurately than any human will. It will even draw a mean graph line through them. But it will never go "uh-oh, that piece of data looks out of place". Only human beings can look at, and if need be, ignore data that could be erroneous. If you just feed numbers into a computer without an intrinsic understanding of the data and the measurements you're using, you're essentially cutting out human judgement and intuition. You're not here to learn how to enter numbers into a machine, but how to see those patterns for yourself, and trust to your own judgement over that of a computer."
It's actually a good point, when you're measuring, recording and handling the data yourself you get a feel for it, and intuition if you like. If you try and measure out 1 km using a 15 cm ruler, something in your brain goes "this isn't quite right". Give that task to a machine, and it will never figure that out, it will just carry out the task.
That intuition in good scientists develops as an understanding of error. This idea says that between an actual value of something and the value I measure, there is always going to be a discrepancy because of the tool I use to measure. If I go back to the simple problem of measuring a short line with my ruler - the actual line might truly be 5.648cm long, but the best I can measure to is 0.05 cm (and then only with a steady hand and keen eye). Most of the time that's accurate enough.
If I think of the "measure out 1 km using a 15 cm ruler" scenario, you can lay the ruler out, end on end and measure out 6666.7 ruler lengths. But you have to ensure each length is perfectly in alignment with the previous one, and starts exactly when the previous one finished. In reality, it's going to be a small amount off, which multiplied by 6666.7 times, is going to magnify that error.
And what if I use it to measure a grain of sand? The best I can measure with a ruler is to an accuracy of about 0.05cm, but the grain doesn't show very well against the ruler (sand from my local beach is finer than that shown). And am I measuring length wise or width wise? In this case my ruler, although a method of measuring distance, is once again completely inaccurate.
This of course links back to my counting fruit challenge a few weeks ago. I asked you to imagine items of fruit in a basket as if there were test cases, some were big melons, and some small grapes. How could just counting them help you really know how you were progressing through your bowl of fruit? If you said that this morning there were 8 items of fruit, and at the end of the day there were 6 left, I could tell you with some accuracy that you have less fruit left that this morning. What you can't say with accuracy is that in 3 days you'll have finished your last piece of fruit.
The problem if of course there are a good deal of test management tools out there who collect all forms of statistics on your project, and that's exactly what they DO say. Often in pretty graphical form, and (what can kind of annoy me) to several decimal points. The decimal points can really get my goat, because by stating it to such a level, it's implying a level of accuracy that as we've said, just isn't there.
To me with an understanding of the science of measurement and error, it feels like Prof Hughes comments come to bite us in the ass, and hard.
Of course if you're a tester, you may well be contractually obliged to use these tools, and even provide this data on a regular basis. What I would encourage you to do is to think and learn about the error in what your tool is measuring and quoting to a high level of detail, and advise caution. Like Prof Hughes said, that computer tool is taking those numbers, counting daily, dividing, but with totally no understanding of the data it's using. It's model expects a single line script to run the same as a one with dozens of items. It won't expect a "test the one hour automatic logout" to take any longer than "log in with user name and password. You are now logged in".
If you have to use test case counts, I do encourage you to group together similar scenarios and create a form of dashboard. A total test count of 94% of passes might be encouraging for someone to at least consider to release into production (obviously depending on the defects), however there might be a clustering of tests which have failed or simply not been run, look below ...
Well that's worrying - why have no login scenarios been passed? Turns out under inquiry that the team is using some form of stub to enable login, because the login page is still under development. Do you even have a product without a login page?
A dashboard at least enables people to ask why you've focused on one certain scenario, and not on another. It allows people to look at what you're focusing on. The "total test case percentage" doesn't encourage that kind of engagement. Whenever possible I use a dashboard to talk with management and business owners. As ever, James Bach has a useful set of resources and slides on them here.
But as I've mentioned, I do encourage you to think about the data you use in your reporting, and how accurate it really is. We're often under duress to use it as "it may be inaccurate, but it's the best data we have", but I do encourage you to understand the error in those numbers, and use methods like the dashboards mentioned to give something more meaningful.
I'll just leave this here, because it's doing the rounds in my head ...
As you may have noticed reading through this blog, I have a few interests outside of just testing. I'm often trying my hand at something different, but no matter what, I try and apply a certain level of critical thinking to make it a learning and growing experience.
My mother told me something on the phone a few months ago, and it was a comment which I found really quite insightful, "you have no fear of trying things, and you're not really afraid to fail. And that means you often can get quite good at things quite quickly, which can be difficult for people around you to take". It's an interesting point - we publicise our success and of course are a bit quiet about the failures.
I recently went to lunch with a good friend from a previous company, where he worked on the helpdesk. Dave is a very smart guy, and has worked as a lawyer, but thinking of going into programming. He wanted to ask me about my experience, as I'd been fairly successful. And it hit me like a brick, because I don't really think of myself as a successful person. But that said, as time ticks on, we tend to leave our failures behind, and we don't really talk about them.
I got into development originally because I was a bit of a failure. I'd trained as a teacher, and there were elements to the job I loved, and some I found emotionally crushing. I actually spent six months unemployed during a recession in the 90s, and without the support of my in-laws, I'd have probably had to move away from the girl who'd end up becoming my wife (things like that stick with you, and why I've never had anything but love for my parents-in-law, or second-parents).
It was a tough time of uncertainty, lots of job interviews, and waiting desperately for something to come through. I wanted to work in computers, and read as many books on programming as I could. I have a lot of University degrees, but actually was self taught in the area I actually ended up in a career with!
I still had access to a University computer lab, so would practice UNIX and C programs there. Life was a cycle of printing CVs and application letters, and rushing to the letterbox in a morning to see if there was any fruit from my interviews. There were a lot of rejection letters, an awful lot. Thankfully, eventually Thompson Marconi Sonar Limited took a chance on me, and the rest is history! Well it certainly got a heap easier from that point on, but still an occasional rocky road.
The point is, every success story has it's difficult moments somewhere. When Rocky finally gets to the steps, it's a moment of triumph that symbolises all the struggle he's taken to get match fit. When we ourselves mirror that achievement, the disappointment of the first time we tried and got half way and gave up gets dissolved in the elation. Some people, many people in fact, never return to get to the summit after their first failure.
Let's face it, we just don't like it when we fail!
Back in 2011, I took part in what was to be an instrumental training course at Kiwibank. It was a one day workshop "for reward and motivation" - our speaker came in and promised us by the days end we would have achieved at least four feats which right now seem impossible. And he was right!
One of the tasks was learning to juggle, and he really encouraged us to embrace the failure. First off we took a single ball, threw it from one hand to the other, and let it hit the floor. And then, we clapped. We clapped because right now we were going to embrace the fact that we were going to fail, in fact we were deliberately going to fail at this point. But we were going to keep on. Piece by piece we tried it more and more with one ball, then two, then eventually being able to do three for a limited time. Each time applauding the balls if they fell. Learning started with failure, and we worked on it, just a bit at a time.
Most instructional to me was the minefield game. There was an invisible route through a grid, and we had to take turns at it. If we hit a bad square, we got a warning "beep" and had to back out down the safe path and let another team mate have a go. At the end, we had an analysis, which was a bit of an eye opener.
The board looked a little like this, "Gronda! Gronda!"
What we found was typically people would stand on a safe square and look for the next "safe" square. And they were really hesitant. People would often take a minute or two to decide. If they hit an unsafe square there would be a "beep" noise, immediately that person would sigh, shrug their shoulders, and leave the board dejectedly.*
As was mentioned - there was no skill or insight needed for working out the next safe square. There was no way you could know, so taking time to determine your next step didn't really help. And feeling dejected because you'd gone onto an unsafe square, again, you couldn't know. Finding an unsafe square wasn't due to lack of ability at all, and yet people still took it very personally. In fact it could be said you'd helped, because you'd found an unsafe square for the rest of the team, making it easier for the person who followed you.
And this is how we are - we like the idea of being a natural, as if you're born with skills, being good at something is a lot more easy. Starting off making a mess and not doing so well is really scary, because the journey is longer, and a good deal tougher. But in truth it's the reverse, if you have some ability, it's harder to accept you STILL need to work on elements to advance!
This is why, I guess, I like to try new things. To keep that learning part of my mind fairly elastic, lest it otherwise gets too proud and dislikes dealing with failure. As long as it's elastic it means I get to keep trying things, and hopefully getting better at it!
My most recent challenge has been my directorial début - some friends and I have got together to make a short film called "Under The Carpet". We're hoping to be good enough to do the 48 hour film challenge next year, and of course I want to bring a certain level of Agile mentality to our filming, most importantly "getting the project finished", over just being too obsessed with details we never make anything.
Directing was fun, educational and very very tiring. For all involved it was their first time doing anything like it, and I'm pleased with the end result. Though at the same time I look at some elements and go, "I wish I changed that line" or "wish we got the lighting better". I'm avoiding the temptation to keep going back to this film and refilming and re-edit pieces. Instead, we'll just try and learn our lessons, and make the next film that little bit slicker ...
Test cases can be as long as short as they need to be. They can cover a lot of requirements in one go, or maybe just even one.
And yet many metrics for covering test progress seem to fall back on "what percentage of test cases are complete".
Think for a while about this fruit bowl ...
If I told you that today my team had finished off 5 pieces of fruit, how much would be left for the next few days? I bet you'd hope (if you're the business owner) that one of those pieces of fruit was the watermelon. But what if it was just 5 grapes?
That's what happens when you count test cases, you're saying a grape is the same amount of fruit as an apple ... or a watermelon.
If you can find a better way to track when we're out of fruit, please comment below ...
Well, just for fun, I had a new idea for that, which related the emotional impact much better - and it seems I touched a nerve, and a fair few people agreed on Twitter ...
This reply though encouraged me to play out the idea a little more ...
Sounds like a challenge!
Severity 1 – OMFG
Well, that’s not good. At all. Plus my insurance details are in the glovebox.
Severity 2 – WTF
Most of its there, but it won’t go.
Severity 3 – Uh?
We can kind of drive this, but it’s not as I’d expect
Severity 4 – Meh
Well, that’s annoying. Would love it to be repaired, but won’t stop me driving.
