I was summoned to explain my test results. What happened next will surprise you ...

If someone follows me on Twitter, I always try and check their profile out.  If their profile says Test or even QA, I think it's worth a gamble following them.

That''s because sometimes they say something, strike up a conversation, and it leads to somewhere interesting.  This happened yesterday with a conversation that began with this, and you can follow it more on this thread.

Outside of our field, I've also been talking a lot to a couple of teacher friends who are likewise seeing an obsession with pass rates.

The logic goes something like this - X is an acceptable pass rate. If,

• Your pass rate < X.  Then this is bad.  Very bad. Reject
• Your pass rate > X.  This is all good.  Accept.

I've talked previously about not having had an easy ride through education, often failing a lot.  And also a bit about my tutor David Hughes, who I had an at times difficult relationship with.

I'd grown up reading every book on astronomy I could get my hands on, and really focusing on physics from the age of 12.  All with the aim of studying at University.

But my experiences in my first year were tough.  I went from being the brightest in my school, to being in a group of peers where I was below average.

At the end of the first term, we had a mock exam, and to be blunt, I bombed.  Physics and astronomy papers are notoriously tough, with a pass rate of only 40% needed.  I was one of the worst of my class in astronomy only getting 19% (the mark was far worse in physics).

To say I was heartbroken and felt stupid is an understatement.  I was seriously thinking of dropping out, because I felt that I just wasn't getting it.  I felt worse was to come when I was summoned to my tutor David Hughes office to talk about it.

I expected nothing less than humiliation, what happened next surprised me.  He had my paper with him ...

"Look, this isn't a great mark.  But I can see where you're having problems.  The important thing is we can work with this.  And you will find it easier."

Knowing what I did understand, and what I didn't, we did work through in tutorials, correcting some mistakes.  That failed test formed a map for him to guide me and make improvements.  Students who'd done better didn't get that, and didn't need it.

Through David Hughes, testing wasn't about rejecting or accepting a student.  It was about finding weaknesses and working on them.

Isn't that the case in software?  A test isn't something you simply pass or fail - a test is an activity you take to find out information about your application.  But a test is meaningless unless you have a feedback loop to apply what you've learned.

This is to me forms two of the most important questions in testing,

• What scenarios would we still like to exercise?  If they're important, why haven't we done them yet?
• Are there anything we've learned about the system that we really should address?

These aren't questions which revolve around numbers.  They revolve around specific answers which we can engage and converse about.

If we're not talking and thinking about the feedback from what we've learned about the product, then we're not moving the product forward.  It is not our job to choose what should be fixed though.  But we are possibly best placed to give an opinion.

Thanks Thomas Ponnet for his early review of this article.


As I explained last time, the blog is moving towards it's last post later this year.  I really want to enjoy these last posts together on what's been an amazing experience.  So I'd like to end each post now with a song, as we ride together into the sunset ...

Now playing:  Ellie Goulding