An equestrian project ...

Fellow tester Lisa Crispin gave me the idea for this ...

What the customer imagined ...

The early prototype ...

How it felt bringing the business owner on the journey ...

The speed with which the project progressed ...

Phase one testing ...

What was finally delivered ...

"Beam me up Scotty" requirements ...

Everyone's heard of the phrase “Beam me up Scotty” from Star Trek...

There's just one problem - that the phrase was never spoken in any TV series or film in the Star Trek franchise. We can trawl through them all and we'll find something very similar in a few occasions, "Scotty, beam us up" or "Beam them out of there, Scotty”.  But never properly, word-for-word “Beam me up Scotty”.

Never-the-less, the phrase has entered culture so much that we know it's associated with the series. We just have the inconvenience of not being able to be backed up by the facts!

I've recently learned that something similar with requirements. On my Waterfall project, we've been living and breathing the same set of requirements since Christmas. We've had meetings about them, we've had a dozen updated copies of the. We've discussed them informally.

Imagine our surprise then when we had software delivered this week, and it didn't quite meet expectations. We turned to our source of truth, the requirements, and found … erm that they weren't there. Not quite as we'd imagined them.

We'd kind of asked for a certain behaviour, and we'd definitely discussed them. But rereading them from a different angle now, maybe not.

This no doubt is the weakness of Waterfall projects. When you talk about the requirements so much, it's very easy to read them “in the same manner as how you've discussed them”.

The difference sadly is one between “what you think the requirements say” and “what they actually say”. Oops.

This in my mind is definitely the power of Agile, where it's the group discussion and consensus which is the living and breathing source of truth for the project, instead of the derived set of documentation (often delivered weeks after all those conversations).

The electric chair problem ...

This week I’ve been having a bit of fun talking with testers about implicit (unspoken) requirements we might have for products.  [Don't take the next bit too seriously]

One that seems quite obvious is,

“It’s a bad product if it ends up killing someone”

Of course testers know better, “but our product is an electric chair … so isn’t it the purpose of it?”.

How could we check this?  In a real project we’d ask a Business Analyst for their opinion.

I have a great relationship with Patrick, one of my Business Analysts.  We share a bay, and there’s a lot of good natured banter.  Like me he used to be a programmer, and he turned BA, whilst I turned Tester – so we’re kind of equal but opposite. 

So I put it to him,


To: Patrick, chief BA
From: Mike, chief Tester
Subject:  Electric chair requirements

We’re working on an electric chair, and we need your business analysis of it.  For our product, is the guy strapped into the chair the end-user, the customer or just an interested party?

When it comes to the reliability of the product, who would you consult most with, they guy throwing the switch or the guy sitting in the chair?

Where is the customer experience in this scenario?

Mike

---------------

His response was so good, I’ve just had to build a blog around it …

To: Mike, chief Tester

From: Patrick, chief BA

Subject:  RE: Electric chair requirements

The end-product is a chair that is capable of sending a high enough current discharged through the body of the person in the chair to end their life in (insert parameters here eg. 5 seconds). 

The Customer is the person who is paying for the product, together with all the people who work with them/support them to ensure that the product fulfils it’s set objective (“a chair that kills with electricity”).

The business rules for this product can be that,

• the power is not to ignite the body
• that it won’t prolong death
• that the current won’t blow the fuses

Consult with

• the Business owner,
• the chair manufacturer,
• the electrician looking after power supply,
• procurement,
• property,
• medical professionals

to determine best death time within the parameters.

So the deliverable is a chair that kills people, the business owner owns the relationship with the person in the chair, not the BA or the Project!

Patrick

---------------

Testing in pictures

How we testers see ourselves ...

How we view business owners ..

How we feel requirements are written ...

How we view developers ...

How we really need to behave with our teams ...

How what we deliver is viewed ...

How our jobs can feel like some days ...

What it feels like to encounter a high level defect ...

What it feels like when we miss a defect ...

How the project manager behaves when a deadline looms ...

How it feels when it goes into production ...

[Take with a pinch of salt - just a bit of fun]

To ISTBQ or not to ISTBQ, that is the question …

It is without doubt one of the most contentious points in software testing at the moment. “Do we need an official certification to be software testers”.

There are arguments all over the place, and I've found myself inspired by an article by my fellow Twitter collaborator Jari Laakso to write up my own opinion.  You can read his article here.

In favour of certification of course is the fact that “anyone can call themselves a software tester”. Any industry where you can just call yourself a role without needing some form of registration, means that the “cowboys” bring down the reputation of the industry. I might be able to call myself a plumber, but the almighty mess I'd leave your house in, coupled with a crippling bill for call-out would lead you to curse plumber-dom if I was allowed to behave in such a way.

However against certification, there is the fact that it very much becomes a whole “industry” in itself to sell you the course, the exam and the shining certificate. And how do you measure someone's ability? Frankly the way it's examined in a multiple choice exam is laughable – give any good tester options A, B, or C and they will always come up with a brand new answer D.

I myself qualifying for the ISEB exam back in 2005 – it was a compulsary course for my company, and I took it with someone who was new to testing. I found myself wrestling with the syllabus, because in many ways the material hooks into such an idealised form of software delivery that I've never seen it in the real world. I kept finding myself asking “but what if ...” and “but surely ...”.

That said, I really enjoyed the course, and I managed to learn from some formal ways and background theory of doing things which I'd previously just done intuitively (without really being able to say way). And despite being the slight heckler in the class, I formed a good relationship with Rob my tutor, and we kept in touch for a few years afterwards.

As a test manager myself I do find myself looking for someone with the foundation qualification, as I know we'll have a relatively similar framework of testing vocabulary to work from.

However I also agree with a lot of the criticism. The exam based on this certification is based on multiple choice and a rigid syllabus. It's the idea that there's a “right way” and a “wrong way” to test, like everything is black-and-white.

Sure there are many “wrong ways” to test. But there is also no single “right way” to do it either. Testing is taking the pure theory of software delivery model, and then using imagination to find a best model for the project in front of you, the way it demands updates, the way software is developed, the methods used to update your test environment. You don't put together a test plan using multi-choice …

I had a great boss (Stephen Pedrick) in 2005 when I took my course. Not only did he book me on the ISEB course, but he booked me on a following course “Testing – Putting Theory Into Practice”. This dealt with really looking at the theory and making practical test plans, evaluating sample projects. This was almost an antidote to the ISEB course, taking the theory we'd learned (which I did enjoy) and really thrashing it in real work scenarios.

The ISEB course was good, the Putting Theory Into Practice was invaluable. Sadly though, where 30 people took the ISEB certification, only 4 took the one follow on (for which you didn't get a shiny certificate). And this indeed is one of the many potential traps of certification, believing that certification is “all you need” to train as a tester.

It's not, as Obi Wan said to Luke Skywalker, “You've just taken your first step into a larger world” ...

Rapid exploratory website testing ...

I was faced with an interesting challenge today at 12:50pm.  We have a minor project which I'm not involved directly with, and hasn't needed any testing staff.  However they've had a new supporting website produced to provide information … this had been delivered that morning - could I spend half an hour checking it out for anything "obvious".

This was an ideal opportunity to really stretch my muscles and give it a going over exploratory testing style.  I knew nothing of the website, though maybe that wouldn’t be a disadvantage, I’d evaluate it much as I could within the time allowed.

1:00pm, the link came through.

Opening the page, I began my analysis.  The site was essentially a marketing toy, telling prospective customers about a new service which was being provided, and would allow them to register an interest.  It detailed all the features which were coming, why it would make life a lot easier, as well as links to both supporting partners and associated stories about the initiative.  It also had a top menu bar which allowed you to jump to parts of the page you were interested, which dealt with a particular aspect of the new service.

Several areas had icons, which if you held your mouse over, would allow bubbles to expand, giving a more detailed graphical explanation.

Starting with the basics, I attempted to click every link, every button on the page, making sure it went to the right target.  Two of the links to supporting partners could not be selected with the left mouse button, but could with a right button click menu [Defect 1].

I tried out the webpage in Internet Explorer.  The menu bar buttons did not take you to the right part of the page at all, which was most curious [Defect 2].

I opened the website using Chrome and Firefox (the browsers I had available).  The page looked identical in all browsers.  However in these two browsers the menu bar buttons DID work as expected [Revised defect 2 with this information].

Dragged my mouse over the icons that opened graphical explanations, confirmed they made sense and didn't behave oddly if close to the browser view edge.  I did wonder why one story had two links to website when others only had one (inconsistent) [Defect 3].

Read through the website – did it make sense?  I noticed that two sentences in the same paragraph were virtually identical, and referred to supporting partners in an inconsistent manner [Defect 4].

There was a field to register interest by adding your email address.  Tried a valid email, it was accepted (good).  Tried one junk email, and got told it was invalid (good).  Tried a couple of variations of invalid emails, not all were rejected.  Noted it as a possible problem  [Defect 5].

The bottom of the page had a huge, but empty grey box which just looked messy [Defect 6].

At this point I thought I was done.  Then I had a moment of slyness.  Thinking about how the menu worked for Firefox and Chrome was a little suspicious - I know web developers tend to love working and testing out on these browsers.  Likewise, I also know how web developers love their big, high resolution screens.  So I went into my personalised settings, and dropped my screen resolution to 600 x 800.  The page now no longer fitted to the screen, and some buttons on the menu bar became mangled with some icons missing altogether [Defect 7].

I emailed my list of discoveries to the project manager.  It was 1:30pm and time for lunch.



That was testing at it’s most raw, and a lot of fun (and a nice break from the meetings and documentation of the rest of the day).  For the product, it was a perfectly pitched piece of ah-hoc testing.  I defected everything I thought could be an issue - of those, there are about 3 things which need doing (the non-selectable links, menu not working in IE and probably the behaviour in low screen resolutions), the rest are more about consistency and look, which might be equally important if there's time.

The issue with the menu bars was discovered by a BA during the same time.  But where they reported it, I managed to define it was an Internet Explorer issue, and not one on Chrome and Firefox.  This made me realise that testers are more than people who "find problems" (their BA, a very talented and smart woman did that), however being a tester, it was my nature to go further than just find the problem, but "define the problem".

A most interesting exercise for sure ...

Roadrunner solutions ... why they need a tester

Last night I was reading Anne-Marie Charrett 's article “I am the Queen of Defocus”.

It's an interesting piece – the idea being really “what is it that some testers bring to the project table”.  Almost any developers understands the technology better.  Many BAs have better understanding of what's written in requirements.

Anne-Marie in evaluating her role has said her skill lies in putting together the “big picture”, or what she calls "defocus".  Many developers can tell you how “their bit” will work, but as long as their piece works as they think, they're not really so intrigued by what's “outside the box”, because from their perspective it's not something they have to work on.  They focus on the areas they're in charge of changing.

A lot of this is understandable – you want people to pay attention to what they're doing.  But at the same time, you need a layer of security that's keen to take over from them, and look at the bigger picture, where the components delivered act in a holistic way.

From this I found myself spinning a fun analogy (sometimes I can't help myself) … The Roadrunner project.

Wile E. Coyote is in no doubt an engineering genius and shows great imagination.  His business case is really rather simple “get that bird”.

To this end, he's supplied with everything her could need by the Acme Corperation, and gets to use their pre-tested commercial-off-the-shelf products to scale a suitable roadrunner-catching solution.

The problem is, Wile E. is so focused on how his solution should catch that bird, he's never able (until too late) to see the 1001 ways his solution will go wrong.  Actually watching Roadrunner cartoons is great tester-training, because we'll almost always notice things Wile E. misses, such as

• The rock he's tied himself to is on the edge of a very dodgy precipice.

• The trigger to chop his prey is set for Coyote and not Roadrunner weights

• Although the firework he's strapped to will give him tremendous speed, ultimately it’s designed to go “BOOM!”.

• His "unit testing" of his trap failed to return the trap to it's original start-up conditions.

Maybe Wile E. Coyote needs a defocused tester …