Quality Auditing - A bad review worth having ...

I was taking another look at Lessons Learned In Software Testing on Amazon today.  I've used the work copy extensively, and really want "my own" edition, but I'm torn between having a paper copy or an electronic version (I use my Kindle a lot).

I have to admit I've not really finished it yet - but I do find it full of useful insights.  I often read things in there that I'm afraid to say I agree with, because I've seen companies and projects learning the hard way.

What interested me was the reviews, or rather the profile of the reviews ...

Most people think it's an amazing and indispensable book ... except 2 people who think it's dreadful (out of 50, that's about 4%).  Fascinated, perhaps with a shade of rubber neck syndrome, I had to look at these reviews.  Maybe these "thought leaders" of testing had an interesting take on the book, that I should not be dismissive of.

This is by far the more interesting ...

Promises Much, Delivers [little]

This book was praised by several colleagues as THE way to work on testing methods and thinking. After reading it and talking with each of them, it was apparent they were excited based on false credentials about ideas that were easy and comfortable but ineffective long term. 

This book is VERY dangerous to a serious testing organization because it focuses on minimal documentation (which means in 6 months when you're asked if you tested X and you can't remember, you'll get 5mins to get out of the building), downplays automation in regression testing (what!!?), and admits openly that it is proposing ideas that are NOT proven (contrary to what the title states) but rather are ideas that "seem to be working" (see pg 176) but no formal nor long term studies support their claims. 

Well, long term studies that have already been done directly contradict their findings: process is driven by a need to be effective and if you don't know what you're doing before you do it, then you don't know what you did when you're done. ... This is a book for those who advocate ad hoc testing to their own discredit and need a means of justificating their apathy and laziness to those who actually know effective testing techniques.

Now that is a bad review worth having!  Sadly I am surprised that this form of opinion represents only 4% of people - though the figures are going to be skewed by the fact that people who feel as the above individual are (I seriously doubt) unlikely to buy this book.

But this view, represented by, let's call him for simplicity Rex, is one many software testers have to face.

The phrase that struck me with a level of absolute terror on Rex's thought leadership style was that piece here, "it focuses on minimal documentation (which means in 6 months when you're asked if you tested X and you can't remember, you'll get 5mins to get out of the building)".

That isn't a model of testing - that's a model of QA, in this case Quality Auditors.  In his model, QAs go around as almost accountants of software recording, noting, writing.  Constantly writing more and more reports and documents which just stack up.

I have to admit early in my career I worked for a project like that (to be honest we didn't know better).  When we ran a "formal test", we needed three people - one to read the script, one to perform the action, and a third person who "witnessed" our test,

• at the end of a successful step, the reader would tick a box in the printed hardcopy of the test script.  
• at the end of a successful test case, we'd each sign our relevant roles
• at the end of a successful test phase, all our signed test scripts would be filed in our office in case of audit

And sometimes we'd have the customer come along to witness one of our tests - so we'd have a special signature box for those occasions too!

Pretty soon of course, the office just couldn't keep up with all the paperwork we were generating.  In truth we were probably now a fire hazard.  We looked like something from the movie Brazil ...

What we eventually learned was that having up to four people witness every test step, was kind of inefficient.  Oh sometimes it was worth having two people testing together to challenge each other, but mainly it was a waste, slow, boring.  At the end of my two years there I heard two testers running one of my scripts and I was actually ashamed of how slow and boring I'd made testing on the project.

The problem with the Quality Audit model of testing is this,

• just because you're recording everything you're doing doesn't mean you're doing the right thing.  It just means you can defend what you did.  And if you missed doing something because (a) people wouldn't give you feedback or (b) you were committed to following your scripted planned process, even though you found issues, then too bad.
• you're to blame for software.  Notice in Rex's take on testing, if there's a problem with the software, it's not the project manager, the developer or the BA who is made to pack up their desk.  It's the tester.  The "QA tester" here is the fall guy - he owns quality, so if the product has issues it's because he didn't add quality at the end - fire him!  Why would anyone want to work in that role?  Personally I'd rather be a Wedding Planner in The Game of Thrones!
• it creates the idea of testing being "just a very bureaucratic layer of software development".  We had three people performing one test - and doing it by slowly reading instructions and performing them.  Did we get the best value from those three testers using that model?  Hell no!  But the documented process was to die for, and would keep our auditors happy!  This leads to the idea many managers have that testing is full of waste.  Yes we do need to document anything important (see my piece on exploratory testing for more), but to give the most value for our testing time, we need the most amount of time either with hands on the software, trying out different ideas and pathways.

Of course the problem with the last point on bureaucracy is inevitably testing finds itself squeezed, and it's very tempting to do the wrong thing in that position (you keep the level of documentation, but test less yes?).  As I've said, I'm a huge fan of using tools like QTrace which just make notes for me as I test, so I don't need to either follow a script or take time consuming screenshots as I go.  Sometimes I know I have a problem area, and I want to try a few things out, and having it recorded helps be go back and show someone.  But even so if a recording doesn't show something, I don't just save it "just because" I want to avoid building up a huge library of files if I don't think there's anything useful in that file.

Testing isn't about auditing software, it's about trying things on your software (the actual doing, not the documentation).  Probably the documentation that matters the most are bug reports - but even so I see test managers who have a hard time coping with teams who deal with defects that can be fixed easily "using sticky notes" because they can't use their tool to "see what defects were fixed 6 months ago".

I'm glad I see less and less "thought leaders" like Rex.  Their ideas to software and to testing are pure bureaucratic toxicity.  They imagine a career path that as long as you can generate a documented trail of evidence you can claim the high ground with any project stuff up.  I think somewhere down the line, Rex is in for a nasty shock ...

I'll wrap up with a superb and concise quotation from Scott Barber to one of the reviewers,

"This review makes it clear that what you really want is to not have to think. Testing is a thought-engaged activity. I suggest that if you want to be told the "only" way to do a thing, that testing might not be the best career path for you."

Your welcome - you're grammar challenge

For me, grammar is a bit of an odd thing.  I was a child of the 70s, and the thinking of the UK educational system I was a part of was that you taught people how to use the English language more through repeated use and examples, than applying terms and rules and assessing them on that.

It's interesting, because I do a good job with writing (not perfect, but hey, this blog and a couple of books isn't bad).  Grammar I'm not bad on, however I'm actually partly dyslexic, and my spelling is pretty terrible - thankfully I've learned a few tricks to hid it, such as a reliance on the red underline on Word documents to warn me I might have made a boo-boo.  [Typically I go ballistic when someone turns that feature off, as it's part of my coping mechanism]

I've been able to use the language quite well, to University academia standards and beyond.  But people are sometimes surprised to know I struggle to know what a verb and a noun is.  We were taught to use the English language without the labels.  In fact in many ways I've learned more recently about this through my son as he goes through education himself (and as per the norm, names and labels are back in vogue).

It's impossible to avoid some of the grammar memes which are floating about social media.  Mainly I find them quite fun, and educational.  Although I find some can be posted from grammar bores.

This though has to be one of my favourites,

Now this is a grammar rule that I definitely know - the difference between,

• your - meaning an item that belongs to you such as your ball, your vanity, your blog
• you're - a contraction of "you are", examples being you're vain, you're annoying, you're welcome

What's fascinating though is despite be knowing this rule quite well, I'll often reread something I've written on here or in one of my books, and notice despite knowing this rule, despite having reviewed my material (in the case of my books, externally), I will occasionally make the mistake of using the wrong form.

This fascinates me - indeed it's the basis of software testing.  Even when we thoroughly know a form like the English language, we'll occasionally make an "oops".  So any form of writing can be subject to these errors, and this obviously goes for coding as well.

If people were capable of never making these forms of mistakes, we'd need less testers, proof-readers and the like.  But of course, it's human nature to make these mistakes - and whilst we can take actions to minimise them, we can never take action to remove them altogether.