Opinion: Keith Klain and the ISTQB petition

For a while now, Keith Klain has asked me to sign his petition to the ISTQB.

His petition asks of the ISTQB, the following,

As a public service to the people who have taken the ISTQB Foundation level exam, I wish to add my name to those appealing to their board of directors for answers to the following questions: 

1) Have there ever been issues with the ISTQB Foundation exam reliability coefficient reviewed by your exam consultants Kryterion? 

2) Have the reliability co-efficients consistently shown, since the inception of the ISTQB's certification program, that results on the certification exams accurately measure the testers' knowledge of the syllabi? 

3) Have there ever been any other issues with the validity of the exams? 

4) How often do those external reviews take place? 

5) Are the results of Kryterion's (or a third party's) independent evaluations publicly available?

I have found the question of whether or not to sign this petition has split me like no other issue of late.  People who know me well also know that I have a bit of a radical past - signing petitions and even protesting are things I've a history of.  At University I petition, marched, sat in, occupied.  I would like to think I still have that level of passion even in middle age.

My dilemma though is that although I'm not a huge fan of the ISTQB, I'm not going to just sign anything that's "anti-ISTQB".  My personal issues with the ISTQB, their syllabus and how they measure "good enough" are specific points.

I strongly believe that an organisation like the ISTQB which claims to be non-profit and which also claims to be championing the testing profession should be accountable to queries from the testing community.  It's like asking a famine aid charity "how much of every dollar I give actually make it into food for Africa".  You should have a right to a certain level of transparency.

However my issue was with the questions themselves.  I know the areas I find myself not really seeing eye-to-eye with the ISTQB, and these questions didn't seem to cover that ground at all.  Truth is I didn't really understand what the questions were driving at.  I couldn't sign up to a petition "just because it was anti-ISTQB", and neither do I believe should you.

When at University I petitioned or marched or occupied, I knew why the principles being put down mattered (okay, occasionally it was to impress a girl ...).  If I can't explain and champion the principles on a petition, I really can't sign it, and that's why I didn't.  As someone who is sympathetic to the Context Driven School of Testing, I'm not going to sign a petition just to "flip the bird to the system", and neither am I going to do that because of peer pressure.

So thankfully this week, I managed via Twitter to really discuss why this petition was so important to Keith, and ultimately, to me and every tester on the planet.  It wasn't just about the questions in the petition, it was about the context of the questions, something you might miss from the questionnaire itself, but is more clear here in Keith's open letter to the ISQTB.

The root cause of the petition comes from Keith (who is a Context Driven Tester) discussing ISTQB exams with Rex Black, who is a member of the board, and past president of ISTQB.  Oh and he just happens to provide a lot of training courses on the many levels of ISTQB (which some might say is a vested interest - much like for example making the Chairman of BP the Secretary of State for the Environment).

Rex's comment around the ISTQB exams was that for the American board, the exams have professional exam consultants work with them, and the exams "though not perfect" were "constantly perfected" (whatever that means - but wouldn't elaborate).  He claimed that he could not comment beyond that due to non-disclosure agreements (which some might say is convenient).  He has declined further comment, saying pretty much that that is all we need to know.

Keith sees Rex's comments as a smokescreen - they don't really give much information or satisfactory answer.  And remember this is from a board which champions overly detailed measurement and metrics as the way to do any task.

Keith's petition therefore is about "digging for truth".  Rex is telling us that the exams are independently analysed, but that data is only shared secretly within the board.  But they can tell us on behalf of this independent group how great they are.  This reminds me of the kind of science and statistical analysis you see used to misrepresent and mislead in TV ads, especially (shudder) infomercials.

This is not how you see scientists behave in New Scientist, or at conference.  Galileo, Newton, Einstein showed their proofs and data to the world, and risked  ridicule and backlash.  They do not go "we have amazing data to prove this, which we can't tell you, but you have to just take our word on how awesome we are".

Discussing this with Keith, he convinced me how this was important to have this data, and the questions are about being able to peek into this (without infringing anyone's confidentiality).  The whole case of the ISTQB validity from Rex Black (who is a high up person in the ISTQB scheme of things), rests on these exams being independently validated and (in his roundabout way of razzle-dazzle) as perfect as a multi-choice exam can get.

But people who aren't in the inner circle aren't allowed to know details, they just have to have faith.  This reeks of the kind of leadership you get in a cult.  And this is why it's in everyone's interest - both CDT testers and testers who stand by ISTQB to sign this petition.  Not to have a go at ISTQB, but to support the level of transparency that an organisation like this needs to champion with the community it claims to represent.

Based on this, I had found my voice on this issue, and found myself agreeing with Keith's stance, signing his petition - but typical me, I added a mini essay,

The choice to sign this petition is an individual one - but needs to be made for the right reasons.  I'm not going to tell you to sign it - just to think about the issues at play here - and to give you my opinion.  If you think I've got it wrong ... that's what the comments box is for!

Discussion and disagreement isn't wrong - in fact me and Keith had to disagree and question for a whole while before we saw eye-to-eye.  We're not drones, and we shouldn't be expected to act as such.  That means not bowing down to Rex Black telling you "what you need to know, for your own good" or Keith telling you "sign my petition".  Your opinion counts - whether towards Keith or to Rex, make it count.