The Mission Of Testing

Way back when I started this blog, I wrote my definition of testing.  I've revisited it many times, and sometimes expanded upon it.

It's an important thing for testers to revisit their definition of "what testing is about".  Throughout this blog I have explored and revisited many facets of testing, and on the train today, I felt compelled to revisit this once again.

This is my current take on testing - it might change tomorrow, or next year, but it will change.  It's not fallible, but not meant to be.  It's designed to be a vision, that is readable and engaging.

The Mission Of Testing

The mission of testing is to seek out unexpected behaviour when a system is placed within differing scenarios, and to make both the quest and it's findings known to those who make decisions.

The prudent tester knows that as they are exploring the unknown, they can never know how much "unexpected behaviour" is left to discover, but can only estimate the risk based on what they know to work (or otherwise).

The Determination Of Scenario

The imaginative tester uses their skill to derive scenarios for testing from their understanding of the system through methods including - but not limited to - requirement, business value, standard expected behaviour or simple curiosity.

The Management Of Testing

The experienced tester knows that the time allocated for testing is finite whilst variations and imagination are almost infinite, and thus there are always more scenarios they can imagine than the time available.

Hence, they will always prioritise first the most compelling scenarios which are more representative or where there is great expected risk.

But the wisest of testers will make these choices transparent to all, both peer and those with the power to make decisions, and will encourage ownership and debate beyond those who call themselves testers.  They will seek to incorporate such direction where it is valid, but champion and mentor when they feel it is inappropriate.

The Execution Of Testing

As the mission of testing is seeking the unexpected, and as they are trying to fit the infinite variation of scenario into a finite time-box, the visionary tester knows it is more important to use their time to explore as many scenarios as possible, and will seek to use any documentation for this activity which is as lightweight and unobtrusive in this pursuit as possible.

Insecurity 101 - The Phil Cool Factor

As a kid growing up in England in the 80s, we all were entranced by comedian Phil Cool.  He was a an impressionist with an amazing ability to contort his face into different shapes.  I remember how excitedly we'd talk about the last nights show in the corridors of Abbot Beyne High School in between classes.

Here's a few examples of Phil at his best ...

You might have noticed this blog has a lot of comic tones, and maybe you think I'm making a joke when I say that I do actually take comedy really seriously.  But I love comedy which is clever and unexpected, I despise comedy that is cheap.  So you can imagine how much I was in awe when in 2009 I actually got to meet my comedic legend quite by accident at a Fairport Convention concert.

He was actually part of a folk support act that played before the main band.  Some of his songs were typically comic, but some really very straight.  At then end of the show I bumped into him whilst waiting for a friend.

What happened was a humbling eye opener for me.  Phil Cool, quite literally the coolest comedian on the TV in my childhood - a man so very loud and funny on stage - but in person ... quite a shock.  I got speaking to him, and said I'd really enjoyed his set, and what an interesting (but different) venture.  Although really polite, be seemed quite nervous, and blushing said it was nice to hear that.  The man before me wasn't a loud and bombastic comedian, but a humble, somewhat uncertain human being who struck me as doubtful of his own (immense in my opinion) talents.

Rather than shatter my perceptions of him, I came away fascinated that someone I looked up to was not the stuff of legend, but a flesh-and-blood human being not so different from myself.

You see we all have doubts.  It's a very human thing to have internal fears and insecurities.

During KWST3, talking about the same cloud of doubt, there was a point where Aaron Hodder asked people to put up their hands to say if they ever felt a fraud as a tester ...

Most of the room put theirs up, but I have to admit I didn't put my hand up.  Commentary from Michael Bolton suggested that was because of the Dunning-Kruger effect.  This is the "grad phenomemon" where someone just out of University thinks they know everything, whilst someone with 20 years experience seems to show less superiority because they're more aware of their limitations.  People who put their hand up were experienced testers who knew their limits, whilst those like myself who kept them down were "deluded newbies" who thought we knew it all.

However for myself, despite Michael's criticism, I've been in the industry, and in teaching and research science enough to know I am a tester.  In my time, I've thought enough to get feedback and to ask around both my superiors and my peers, and it turns out that much like Phil Cool, insecurity is a common thing.  We're often hiding it beneath bravado.

I never think of myself as a fraud as a tester, simply because I've had a couple of non-testing careers.  I would not have spent 16+ years in the IT industry if I was plagued by doubts about me belonging here or having worth here.  Simply put, life is too short, and I would have abandoned ship long ago.

But it doesn't come easy.  I've get to know others, and be open, and sometimes talk about how I feel out of my comfort zone with peers that I trust.  My wingman on RAF projects 10 years ago was John Preston, and we still exchange emails talking about problems and challenges we face in IT.  Through both my career and through the internet, I've built up a wide peer network, many of them people I trust to ask big questions.

I've also looked up and followed some industry leaders in IT, and got to know a few.  They can seem to tower over us as Titans, and we can look at our projects and go "I bet Heracles Tester doesn't have these problems".  What has surprised me as I've got to know a few better, is that the Phil Cool effect takes hold - even in IT.  People who seem superhuman figures, powerhouses who can get things done, the more you get to know them up close you realise they are not Testing Titans who have been empowered with mighty and mystical powers from the Gods Of Testing themselves.  They are human beings, vulnerable and at times caught and ensnared in their own emotions.  Just like you or me.

This revelation has not diminished them in my eyes - it moves them from being something iconic (but also people I can't relate to), to being more peer in nature.  But at the same time, they become people I can relate to more, and who inspire me more in daily life.  Because if they do not have to "skip the hard yards" because of their status, if they have to go through the same assault course, then maybe I can too.

I know being a "famous" tester doesn't mean the learning curve of a new project is any easier, it doesn't exempt you from the occasional tussle at work, having to explain your position as a tester or having to do the (occasional) extra hours to get a project launched.

With that knowledge I've gained through networking that there are no magic solutions or attributes, there was no purpose putting my hand up to feeling like a fraud.  It serves no-one to be modest.  And as I said, if I really felt like that, I would have jumped to a new profession by now if I genuinely did.

That said, I do have doubts and even insecurities.  To steal the words of Douglas Adams, I'd like to call them "rigidly defined areas of doubt and uncertainty".  I worry sometimes there are things when planning testing that I don't know, areas I'll not think to test enough, bad assumptions I'll make.  

These doubts are not about me as an individual, but about whether I know or have thought about enough things about what I'm testing.  But then again, isn't the first law of testing that the profession exists because people are fallible and make mistakes (in code or design).  Given that human frailty why would testers be exempt?

The answer to that is just as we do with testing, just as I mentioned about personal insecurity.  We put it out to a network and get feedback.  We put it in front of people we trust (hopefully who you work for), and ask their opinion, critique and questions.  Then the planned testing isn't a one-man-band, but the sum ideas of everyone on the team, some of whom might have different ideas to how things work to you.

Getting people to positively critique to make something better is a superb strategy for fighting neurosis and feeling more confident about what we do.  At the end of the day, giving critique and feedback to produce a positive outcome pretty much sums up what we do as testers.

Returning To Test Estimation ...

A couple of years ago I touched on the subject of test estimation in my article Those Darn Test Estimates.  Following a recent conversation with Jari Laakso I found myself wanting to return to the subject.  Why?  Because although at the time I was happy with that article, today I find the article isn't good enough, the reason being that in the time that's passed, I've learned a few things ...

We are told that as testers the most important relationship we have is with developers, and whilst that's true for junior testers, as we take on more leadership of our team, things change.  Increasingly as we get senior it's our relationship with the project manager that becomes more and more key.  Oh, the rest of the team need to keep networking with those developers, but you become the key relationship point between the project manager and the test team.

For that relationship to work, it's important to know what matters to project managers.  Many people when they start in software testing think project managers are only interested in two things "hours booked" and "percentage done".  Do not be fooled!

Over the last two years at Kiwibank I have been very lucky to work in a business unit of project managers, business analysts, market managers and business owners.  As Dian Fossey did with gorillas, I have managed (through the miracle of co-location) to observe business people in their natural habitat, and in the process learn more about them.

But even that hasn't been quite enough.  To understand more of what I heard and what I saw, I used Johanna Rothman's Manage IT guide for practical project managers.  Not because I wanted to become a project manager, but because I wanted to understand them and their processes.

And so this, in a basic nutshell, is what I learned.  Project managers have two main focuses, which although they link in with "hours booked" and "percentage done" they are not "hours booked" and "percentage done".  They are,

• To manage the budget and timeline of the project.
• To manage the risks linked with the project.

The risks are important - in fact, project managers keep a register of risks much like we keep registers of defects.  There they list potential pitfalls, how likely they are, and how significantly they could impact the project if they were to occur.  Multiplying the likelihoods with the impact gives a list of the top risks, which the project managers focus on.

With that in mind, lets return to the subject of test estimates, and see how we can work more on the same page with project managers in the domain they understand.  Of late I've worked with a project manager named Annabelle, and we've thrashed around our recent test plans until they've worked as below.  It's been a positive relationship, and has really allowed me to see much better how elements of my plan are used,

Estimates - the good, the bad, the ugly

The starting point it to look at the project, the general scope, and just trying to size it up.  There is probably some absolutely fabulous formula out there to do this.  But I think there's just no substitute for past experience.

What past project does this most likely feel like?  Don't be afraid to ask around people to get their experiences as well.  Listen to their stories of how their past projects took more or less time that you experienced in any parallel projects of yours.  Where is the complexity?  What will slow us down?  Are processes we're testing instantaneous, overnight, monthly?  How will that affect how we test?

By now you should feel split three ways on three possible scenarios; "well if everything went perfect", "I guess we might have a few bugs and tests to redo, we normally do" and "OMG, we're screwed".  From Those Darn Test Estimates, that's what we'd normally call Best Case, Probable Case, Worst Case.

But wait, we're not finished ...

Now here's where we can have a meaningful dialogue with our project manager.  I'd always put into any plan a list of my assumptions, but the genius suggestion from Annabelle was that I add a column which said "Impact if wrong".  If my assumption was wrong, what would it mean for my plan and my timescales?  What would have to be redone?

To me this was a game changer, and suddenly I realised I was helping to communicate more to my project manager in a much more direct manner than asking Annabelle to "read between the lines".  Suddenly from a dry list of assumptions, a few of them would stand clearly out, because anyone reading can clearly see "well if they're wrong, this is going to hurt".  This inevitably leads to a conversation around "well how can we get more information to check this assumption or monitor it?".  Suddenly people are being proactive, over waiting for issues to happen.

This of course leads to the issue of risks.  Actually the word is such an important one in software development, it needs a capital, exclamation marks and all kinds of funky formatting ... Risks!!!

It was only recently I had the revelation that when I'm giving a list of risks (with likelihoods and impact of course), what I'm telling to my project manager is this 'these are the factors which will push testing onto the "Worst Case" time estimates over the "Probable Case" ones'.  If you think your project manager doesn't understand this, make it clear by writing that very statement near your table of risks.  It's something we sometimes take the danger of implying, but it needs to come out on the table and be made clear to all involved.

Now, us those conversations you had with other people to help you fill in these risks.  On a first draft, you should really throw to a larger group all the risks people have thought of (if your project if big enough, maybe workshop it), sometimes something that sounds innocent gets someone else thinking "wait a minute ...".  It's important to capture as many as possible, and then truncate and collect into themes.

Like with assumptions, once collected, a few will really just stand out with an "ouch factor", and no doubt make it into the project manager's risk register.  But they're not just there to be something you fall back on when things go bad "look, said it might be a risk, turned out it was".  They, particular the "ouch" ones, are something to be proactive about.  How can you tell early if this is happening?  What can we do about it?

Lets look back at some of the risks I identified first time around, and see if we can be more proactive ...

Software Delivered Late / Software Delivered Is Of Poor Quality

What testing is development planning to do?  Can testing get early copies of software to get a feel for what's being built?  Can testing talk to development about the testing planned to help development in their unit testing?  Can testing get visibility of the testing developers plan to perform for unit testing?

Notice all of these are about dropping the silo mentality between testing and development and working closer together over a passing of baton between development and testing phases.

The Delivery Chain

How can you get developers and testers working closer together, especially if you're not co-located.  How about a daily catchup even in phone conference?  Throw in video if you can (it's the 21st Century after all, and we work in IT).

If the software development is done out of country and you only get weekly builds, can you get the development to try out scenarios on your behalf?

If you are getting weekly builds and the software prevents basic testing because it's so buggy, can development prioritise on those key bugs and get a fix over asap?

Time Erosion

Keep a log of all the meetings needed.  Does everyone need to be at these meetings?  Are you getting value when everyone attends?  Some form of daily meeting can be invaluable, but most people will only need that one.  Leaders can attend other meetings on the teams behalf, and feedback anything key to the rest of the group, likewise providing information up the chain from testers on the ground floor.

It goes without saying - don't try and to major testing sprints over major holiday seasons, and try and work on personal appraisals before/after scheduled busy times.

Requirements

James Bach in his Rapid Software Testing course calls requirements a form of Oracle, or "model of how software's supposed to work".  An Oracle sets up an expectation of behaviour, so when we do something with software and it goes against our Oracle we know "well that's not right".

Requirements are the most obvious "model of how software's supposed to work" that we usually encounter but there are other kinds of Oracle as well.  Can your team,

• Talk to the business owners who put forward this project?
• Talk to end users about what they'll be doing with the system?
• Talk to a subject matter expert?
• Try out similar products which are already out there on the market?

All these things help testers build up a sense of "what the product should do, and what behaviour matters".

Novopay - the tale of a compelling event in the New Zealand IT industry

Open almost any book on testing, and it will start with a "cautionary tale" about software testing, where something was released into production, and it went wrong with devastating results.  They are the ghoulish "tales around the campfire" of our IT industry.

One problem I have found within the New Zealand industry is that as testers we love to focus on anything that goes "bang" and crashes.  In 2011, I was in one a talk about software testing to a group of new developers as part of the Summer of Tech, where our introductory speaker spoke of an Ariane 5 rocket where the parts had been individually tested, but put together into a new rocket.  As you can imagine, this rocket barely cleared the launchpad before it exploded. Another cautionary tale of "you didn't test enough".

What's interested is what happened next.  One of the audience interrupted with "but we're creating applications ... not rockets.  Our stuff is hardly critical like that".

To me, this reinforced how vital it is to have stories and tales of failure which are relevant to the audience.  I thought the tale was a wonderful one, but looking around the room, I saw it failed, because the audience simply did not believe it applied to them.

New Zealand is a small and very pragmatic country.  A lot of processes here are still fairly manual compared to my home country of England, and overall that's not a bad thing.  Here in New Zealand if something isn't broken, New Zealand attitude is "why try and fix it", so,

• In the UK for trains we have automated ticket vending machines (usually vandalised), automated turnstiles to get onto platforms etc.  We used to get told our ticket prices would have to go up above inflation to cover this automation and it's continual repair.  In New Zealand they just have an old fashioned conductor on the train who checks and clips tickets, and can sell you one for cash if you need one (without charging a fee).  Simple, but y'know it works.
• In the UK there were plans to build a so called "chip and bin" system for collecting rubbish.  The UK rubbish collection vehicles would scan your bin, which would then be weighed as it was disposed of.  Back at the main office this data would be collected, and an itemised bill created (your bill would have to be more to cover all this new technology which would have to be developed and maintained).  Again being super pragmatic, New Zealand councils just sell you council marked rubbish bags which have a charge on them for collection.  Only rubbish in council bags is disposed of.  The more you throw away, the more bags you need.  Elegantly simple yes?

An unfortunate by-product of this pragmatic way of doing things is the attitude of "she'll be right", which means if something goes wrong, don't worry, we'll be able to fix it.  This means on the whole New Zealanders can be a little bit more risk takers than their European or American counterparts.

In a testing consultancy I previously worked for, my test manager would talk about how New Zealand would have to face an inevitable "compelling event" for software testing.  Most other countries have had one, but so far although there had been failed there hadn't been one in New Zealand.

A "compelling event" is something that forces you to take action - a wake up call the the importance of the value of testing.  A compelling event isn't just software going bad when it hits production, it's software causing a very large and public pain that it unnerves the local IT industry as a motivation to not repeat that mistake, often with a gasp of "that so easily could have been us".  It's not a rocket blowing up half way around the world, it's software that fails but feels too close for comfort compared to what you're doing right now.

It's a compelling event because once it's happened it compels you to take a good hard look at your strategies around testing and quality and ask "are we (and not just testers here) doing enough?".

Sadly, we've finally had our big compelling event here - the Novopay saga.  Novopay was an online system for managing the payment of teacher and school staff salaries.  But sadly it has gone into production to go horribly wrong, with payments to these people missing in the system.  There have been schools where teachers are missing months of pay (and teachers aren't exactly in the most affluence of careers).  More than just being "missed payments", there have also been erroneous payments gone out from schools - some teachers who've never worked for a school, are receiving payments from that school - and schools are having to bring in extra staff to go through the Novopay payments with a fine tooth comb to work out what's payments are good, what payments are errors and what payments are missing.

The media have been in a frenzy over it, hounding senior staff at the Ministry Of Education (who are behind Novopay) saying they have found a report of "200 known bugs" in the production system when it went live.  It is terrifying to me as a senior tester is how bad that sounds when the media put it like that.

Of course when I worked on an avionic computer, it'd surprise people to know it flew under the strictest of safety conditions, but still there were 1,500 known bugs in the software.  The important message though isn't the bug count, but the severity.  But this is a very difficult dialogue to have with a set of press hounding for a "big story" and "potential coverup".

In an unusual move, the Ministry of Education has released the test plans for the Novopay project into the public domain.  I took a look through, and they show a fairly thorough planned approach was taken to testing, not the slap-dash "rush into production" that many in the press are claiming.  That was pretty unnerving - most of our "testing horror stories" like the Ariane rocket involve people simply not testing, thinking it would be "alright mate".

The documentation they have also shows a decent approach to several forms of functional testing, indeed going beyond what I would have initially thought to do,

http://www.minedu.govt.nz/theMinistry/NovopayProject/NovopayTestPlans.aspx

However for all it's success in testing, something has gone very wrong in production, there can be no doubt about that.  And it is causing real grief to people whose life it should be easing.

What went wrong?  Each of us will have our opinions about this, and no doubt all of us will be right to some extent, whilst at the same time knowing in our hearts how easily something like this could happen to us.

Testing is about identifying and helping to remove risks, but it does not guarantee a bug-free product.  We can test in the many different ways we expect our software to be used, we can check for all the problems we can imagine could happen.  But that will never cover everything, there will always be issues beyond our imagining.

But the imagination of any individual has it's limits.  We have to be careful not to find ourselves screaming "inconceivable" at every bug found in production.  In my opinion, this is why testing and quality is not just a "test team" ticket.  We need to be engaged with developers, business analysts, market managers, usability experts, end users to work out a whole scope of things to test (from areas of functionality to ways of using) that is beyond the imagination we as testers can summon when looking at requirements.  But by then our scope of testing has probably increased by several orders of magnitude, and we don't have unlimited time.

That's when we need to do some risk based analysis, and cover as much of it as possible, touching on all the "most likely" cases, then touching on samples in other areas.  If you find problems, then odds are there'll be others, so keep looking in that area.

For myself then , if there is a compelling lesson to be had from Novopay it is that we need to draw up our test plans as we always have done, and then ask of others "what could I have missed".

Project Christmas: The Elf Who Learned How To Test

I have had a busy year, no doubt about it, with a lot published in magazines like Testing Planet, Testing Circus and Tea Time With Testers.  There has also been my book, The Software Minefield.

I've recently put together another much shorter book, which is free to download called The Elf Who Learned How To Test, of which I'm particularly proud (great I now sound like Q from James Bond).

The idea started a long time ago with a conversation with Rosie Sherry about the idea of "Imagine there's no testing" back in August.  And that's just what I did with the leap of imagination required.  I imagined Santa's Workshop where no-one tested, and children received substandard presents, and an elf who discovered how testing could add value.

I've been told by others in the testing community I'm a great storyteller.  Indeed in The Software Minefield, I mentioned I'm always telling war stories or parables.  What pleases me about The Elf Who Learned How To Test is that it's a tale not just for testers, but for their children as well.

I think as testers we sometimes are great at joining together as a community and sharing our stories.  But perhaps where we fail is sharing what we do, not just with others who aren't testing, but especially our children.  And my book works as essentially as a Testing Fairytale which can be shared with children, with some thinking activities at the back which I feel the children are likely to score as well in as the adults!

This year has seen me peeling back the mystique around testing for my 14 year old son, who has come in to see what we do.  I keep trying to talk to him about what I do and why I do it.  I try and develop him a sense of analytical thinking, especially in our common area-of-interest which is history.  Unsurprisingly he did well with the activities in the back (which do not have any "right answers", but as more about seeing how you can expand on the story, and how you interpret some things which are not said).

You can download the book here,

https://leanpub.com/TheElfWhoLearnedHowToTest

In addition, I did a video of me reading it for YouTube, but it turned out too big, so I've decided to put it up as a podcast, which can be accessed below,

http://testsheepnz.podbean.com/2012/12/22/the-elf-who-learned-how-to-test/

Podcast Powered By Podbean

This is all aimed at encouraging donations to a very worthwhile charity, Starship, which supports sick New Zealand children and their families.  I have been helping to support this charity through work, and if you'd like to support them as well, please give a one-off-donation below,

https://www.starship.org.nz/foundation/how-i-can-help/making-a-donation-now/

Testing the process of change ... lessons from HMS Invincible

Back in 2003 I worked for a company who provided the software used on the shipboard servers of the Royal Navy. The servers were used as an information network to share important documents between ships around the world. Surprisingly rather than all being top-secret intelligence, most of this was housekeeping. A naval ship essentially share many characteristics of an office, with a large number of people aboard who need to share information on activities, events, usage of disposables (you'd not be too pleased if your office ran out of coffee or toilet rolls), it is in many ways a floating community, and the Naval intranet allowed the organisation of many aspects of daily life.

I was a developer at the time, and following a course had been working (initially at home) on a Perl script which I got my managers permission to pitch to our customer. I'd written my application from scratch, but today we'd recognise many of it's attributes as a shipboard-Wiki (I called it the Generic Web Page, having never heard of Wikis at the time). It allowed sharing of information pages (which could be instantaneously modified by users with suitable permissions) between both personnel onboard ship as well as between other ship servers. Whereas before ships saved and exchanged information in document form (which took time to replicate on the Naval intranet), my ship-Wiki could be updated instantaneously.

The Navy were impressed. Although it was in their opinion not suitable for anything secure, it had a lot of potential uses due to it's speed over the document sharing method that was in use. They wanted to try it out in an upcoming NATO exercise onboard the UK flagship for the exercise, HMS Invincible.

Of course management was delighted to get such an enthusiastic customer response. But they had a fear – could I install my Generic Web Page application safely onboard the Invincible server? This was their dilemma … if I got this application working, it would be a huge statement about our companies can-do attitude. And if it worked well, we had all kinds of ideas to expand the product and provide more of these kinds of applications as a whole new line of work.

However if something should go wrong – then we risked knocking out the information sharing ability of the Royal Navy flagship in what was likely to be a billion dollar joint-forces exercise. The Royal Navy would look bad in front of it's NATO partners, and we would look unbelievably incompetent to our customer.

Risk and gain – all change has elements of both. What was decided, we took a replication of HMS Invincible onto a spare server, and we rolled our change on, we ran the server for a day. Then we rolled the change off, and checked we'd removed it.

We did this to develop a list of steps to apply the change, and to confirm we were drilled in it's use and application, but mostly to check and explore for any potential issues we might have. We wanted no room for error – so we did this in total of 4 times. Then on the 5th day, we used our steps on a completely different server to make sure there were no other potential surprises (unexpected configuration perhaps).

We produced a report, and our Naval adviser was satisfied we'd taken adequate precautions, so we were allowed to install on the HMS Invincible herself. That was quite an experience to get in the field and perform this change – amazing to be in the belly of such a grand titan. Did you know the ship has a small supermarket called the NAAFI inside that sells soft drinks, chocolate bars etc? Yes the ship has more people and shops than some villages I've lived in!

The installation was a success, and the software proved itself within the exercise. The Navy decided it wanted to develop that kind of capability, so the piece of work was extended to a much bigger program, although I didn't continue to develop on that project.

Testing the process of change

In testing we get quite used to our test environment – during the course of testing it undertakes so many changes and tweaks to get things right. Sometimes it's new builds (which are easy to track under configuration management), sometimes though there's a setting tweak that a developer tries which perhaps they made but forgot to write down.

At the end of testing, you produce an exit report which signs off that what is in your test environment has been checked, and seems acceptable for production.

But how do you confirm the change outlined for production will produce an environment which echoes the one you've signed off? For most releases to new systems, it's usually not a new build that's applied, but a series of changes just to modify elements of your applications and settings.

How can you test that your release team has all the changes needed for production? Well our enterprise on the HMS Invincible was a good start. Start with an environment under test which mirrors production, apply your changes, and test the end result as you would a production verification test. Then roll back, and confirm your changes have gone. Encourage the change team to repeat this process under as production-like conditions as possible (especially regarding timescales). Are any services interrupted during the change? Do any defects encountered during testing seem to come back (that is, you've missed a change somewhere)? Can you do the core, high value actions? Can you effortlessly roll back? Have you developed a definitive list of steps and actions that work every time?

It's amazing how many times we take for granted that what we sign off in our acceptance test environment will be delivered to production. Is your project making steps to ensure nothing's been missed?

No escaping me in October ...

There's no escaping me this October. After a few busy months, I have articles in no less than three testing magazines …

That Barton Fink Feeling

Is an article in Tea-Time With Testers about a feeling we'll all come across at some time. How do you deal with accusations you're not being a “proper tester” because “testers on my last project ...”.

The Ethics Of My Father

Back in KWST2 we were talking about ethics, and I gave an experience report of growing up as a teenager and seeing one of the difficult ethical choices my father had to make, and how that affected me.

Championing The End-User Using Personas

A look at how at a previous company we learned to scale naval products for the fishing industry by developing the persona of our end-user.

This is a brand new magazine NZ Tester, which Geoff Horne is putting together, and even if you're not from New Zealand I urge you to give this magazine your support.

Enjoy!

The Tester's Video Library ...

My son is a bit of a YouTube addict, and to him and his generation, this is something they turn to when they need to learn about an area and learn fast. This to me can seem quite lazy and almost a cheats way of learning, but as his depth of knowledge of Second World War history has shown, it can be very effective.

Yes - time to get with the 21st Century (so my son tells me).  Learning is no longer just about going to the library and "reading a book".  Today we have blogs, and through YouTube we have on-tap tuition when we need it.

Over the last couple of weeks I have been following his lead and trawling the internet to watch talks about testing – and I too have found it very educating. Some videos have touched upon pain points I'm all-too-aware of, others have really challenged me to raise the bar in my testing and work. And so I'd like to share them here

I'd like to thank my fellow Twitter testing wingman Kenneth Aarseth‏ for providing some of these links.  If you know any must-watch videos, please add them to the comments!

Janet Gregory's Learning for AgileTesters

I first watched this in 2010 when it was recommended on the Yammer feed of the test consultancy I worked for. It had me hooked, and it's probably fair to say it inspired this blog. The idea (which I've previously discussed here) that any learning helps you to diversify and grow as a tester. Marlena Compton provided me her blog entry on the subject which she calls interdisciplinary studies, which is a good read.

Janet Gregory's Testing Traps

That woman again. This video was filled with good stuff, of course a lot of it I knew from reading Janet and Lisa's book on Agile Testing. But the one thing which really stuck and challenged me in this talk was around “am I doing enough to make my testing visible?” to the whole project. Challenging. Am I making my testing visible? Yes I know I am. Are there ways to make it better visible to the people who matter? Erm … I think I need to reflect on that. There are always ways to do things better aren't there?

Uncle Bob Martin's Requiem for C

I loved this video, although many testers may not learn huge amounts from it compared to the others, it's still worth experiencing. Uncle Bob is an amazing storyteller, and takes us through the development of software and computers during his career. As I am an ex-developer in C I was rivited.

The main thing you will take home from this is how computers are changing constantly – they have evolved during Bob's lifetime, we'll see the same. Our skills can rapidly become outdated, so continuous learning throughout our career is something we need to embrace.

Michael Boltons Thing's could get worse: Ideas about regression testing

Michael challenging us on,

• What do we mean by regression testing.
• Can we benefit from automated checking tools?
• Why human testing can investigate and discover issues checking tools will miss.

What I really came away with from this was the discussion about automated checking vs testing.  That using automation we can only check for the problems we could imagine when we scripted. There are all forms of issues which can quite easily slip through the net because you didn't plan your script to adapt and cover them (because you couldn't imagine them happening). This is where a sapient human tester can adapt and investigate where an automated script either grinds to a halt or even worse, carries on an the issue goes unnoticed.

Johanna Rothman's Many years of managing testing

I saved the best until last! Simply because it touched on so many points which are relevant to me this year. I recognised in this talk things I'm doing at the moment, but need to chase up from this talk and get better at. 

Key points in Johanna's talk were,

• Testing is about gathering information on the product not ensuring quality
• Manage your communications about testing via a testing dashboard
• Learn to not spread yourself thin and to say “no” when needed
• As a test manager, organise your testing portfolio. Make it transparent where you have no resources.
• Don't move around testers between projects, you'll lose their expertise in areas, which is what makes good testers.

Johanna Rothman: Yes or No? What to do when faced with the impossible

Some more Johanna, talking this time about managing your time and requests to avoid falling into the pitfall of multi-tasking.  Managing expectations - knowing when to say yes and when to say no, how to get people understanding your pressures and priorities.

Johanna Rothman: Lessons learned in project management

This is a must-watch if only for the piece at about 10 minutes about "all out people were good ... except testers, they let everyone down".

The science of software ...

Back in 2002, I had my own form of Buccaneering.  As a developer on a UNIX aircraft project, I'd been in the buisness for over 5 years, and noticed a few parallels between some fundamental laws of physics and software engineering.

I'd written these up, and had them pinned on my desk as "The Talks Physical Laws Of Software Engineering".  Unfortunately I've lost all copies of them, so I'm recreating them from memory ... as you can see they're not comprehensive and were meant more for fun, and from a development over testing perspective.

But what they're an example of the concept of Buccaneering, taking a parallel idea from (in this case) physics, and bringing it into the software engineering context.  Of course none are particularly ground-breaking (although we were caught out by rule 6 at the time) ...

1)  Problems in code (defects) will remain, unless work is done on them [Newton's First Law Of Motion]  Really, defects just don't go away ...

2)  As you are doing work to remove defects, you are also doing work to potentially introduce new defects [Newton's Third Law of Motion]

3) Software that is developed without any monitoring will tend towards chaos [Second Law Of Thermodynamics]  You are just going to hope that everything is good?  Let me know how that works out for you ...

4) Small deviations applied over long enough distances cause massive errors [Trigonometry]  So try and find any defects early.

5) Any system will have levels of uncertainty [Heisenberg's Law of Uncertainty]  Of course you want to minimise uncertainty (which is part of what testing's about), but you cannot remove it altogether!  It will always be there in unknown finite levels.

6) The more closely you observe an event, the more likely you are to be impeding it [Heisenberg's Law of Uncertainty]  Coined as we learned that trying to rerun defects using a debugger could prevent the problems re-occuring - mainly as many debuggers of the time would initialise data stacks which would otherwise contain random data.


Maybe you've read these, and come up with a couple more examples in your head?  Congratulations!  You're thinking outside the box ...

Buccaneering knowledge ... or International Think Like A Pirate Day ...

So when I was talking about my man cave in the last entry, I was talking about a room with things which fascinate me, and which I'm passionate about.  Key amongst this is a bookcase which is filled with a variety of books, for which software testing is perhaps the least well represented.

Software testing as it relates to the technology of computers has only been around as long as computers themselves, a mere 60 years.  However software testing is about more than understanding computers - it's a big part of it yes, but it's not the only part.  It's about how human fallibility, it's about how people are psychologically wired, it's about how people behave in groups, it's about what people say vs what they mean, it's about human leadership, it's about how you take theories and prove/disprove them, it's about how ideas evolve ... it goes on.  And all of this stuff has been going on so much longer than computers.

As long as humanity has been around it's been building ... whenever it's been building there have been projects ... whenever there have been projects there have been mistakes.  Who is to know if the Tower of Babel failed because of God or because simply the architects weren't speaking enough to the on-site project managers?

So people have been analysing and writing about many of the themes common to software testing for hundreds of years.  They've analysed successful military campaigns and leaders, they've analysed the anatomy of a disaster.

James Bach when I met him this year talked about the idea of "buccaneering knowledge", going out looking for ideas and practises in other fields, and if they have value, bringing them and applying them to expand and pioneer the field of software testing.  I love the word "buccaneering" as it's so apt, you're going into another field and taking away their most valuable ideas ... at gunpoint.  But the point is you don't take everything, just the stuff you can see has value.  And being able to have an eye and a mind to challenge what you see before you and separate the good from the bad is how you become a truly innovative tester.  Because let's face it, testing comes with enough dirty laundry and baggage of it's own, without forcibly hoisting away another disciplines!

I've heard many other testers talk along similar lines.  Early in my blogging I was lucky enough to come across a video about Learning For Agile Testers by Janet Gregory which really inspired me (click to watch).

This is why it's important to be surrounded in a room by things which interest and inspire me beyond software testing, because I'm forever pooling and channelling this in with my writing, looking for the parallels and the parables and how to apply to testing.

Much like Janet says in the video, the path for your own learning by doing this is not by following someone elses passions, it's about following your own passions, and finding a way to bring it back and communicate your learnings from this area to the wider testing community.

Here are some other examples of testers whose passions feed the wider community,

• Lisa Crispin, co-author of Agile Testing has a great compassion towards animals, most famously donkeys.  Speaking with her, the ideas of teamwork and compassion are important ones - and she's generally a motivator more by carrot than stick.
• Bernice Ruhland, a regular contributor to Testing Circus really does feed the passions of the testing community, as she's passionate about good cooking and baking in her blog.  She's probably more of a motivator by carrot cake than stick.  In fact going deeper, we've had some conversations about following-a-recipe vs improvising-from-available-ingredients, which go deeper than cooking.
• David Greenlees, who is deeply passionate about martial arts, and seems to have learned to break code with his BARE HANDS!

If you're reading this now, let me challenge you with this.  What are your passions, and how can they benefit the testing community?  Think on it ...

And now for something a completely different ... ethics!

Back in June this year I was very honoured to be invited to the Kiwi Workshop on Software Testing peer conference on ethics that was hosted by James Bach.

Many of my own peers asked me "can a conference on ethics be relevant to software testing".  And I will admit I had my own doubts.  But I was proved wrong.

Following up on this subject, I've decided to raid some of my old material.  Before blogging about software testing I wanted to be a fiction writer, and would write quite a few short stories which my friend Violet would review and give support for.

There follows a couple of examples of historical fiction, which I'll be picking up on the ethical themes of later ...

Godspeed Neil Armstrong

An incredibly sad day indeed – Neil Armstrong, the first man to set foot on the Moon has died at the age of 82.

Although Armstrong did a stint in the US Navy air force, seeing action in Korea, he actually left the military to gain a degree in aeronautical engineering at college.  He had a love of flying though, and ended up embarking on a career as a civilian test pilot. He was a different mould to the Hollywood stereotype of the hot-headed test pilot, being quiet and methodical due to his engineering background. Some test pilots like Chuck Yeager felt such pilot-engineer testers got themselves into more trouble because they would try to think their way out of situations over follow their instincts, and in Chuck's mindset trying to over think could lead to the kind of hesitency that a test pilot could not afford.

He joined the NASA space program as one of the few civilian test pilots. His first space flight was Gemini 8 where his capsule performed a docking with an unmanned Agena module. This was a world first.

However soon after docking, something went wrong with the Agena, and the two linked vehicles started to go into a spin, as the program controlling the Agena thrusters seemed to react against the Gemini 8 command capsule. Noticing they were burning fuel trying to correct this, the Gemini pilots decided to disengage, however the Gemini 8 capulse then started to tumble even fasted when removed. It was only due to some steely decisions made by Armstrong that the pair did not black out, and the Gemini 8 capsule was stabilised.

Some members of NASA said that the Gemini 8 crew had compromised the mission by not following the malfunction procedures for such an instance. Despite there being no such procedure for this eventuality. Armstrong being a true tester just worked through the problem and found the best solution he could for a situation no-one has anticipated.

It was his calm manner and lack of ego which made him an ideal candidate to be the first man on the Moon. He was a perfect choice – when he returned to the Earth he didn't use his position to influence politics (despite being asked to) and was careful in the choice of any company he chose to endorse.

Of course he didn't get to the Moon on his own, he got there as a figurehead of a team, from the controllers at NASA to the guy on the assembly line. He always spoke with great humility acknowledging those whose input he always wanted us to remember.

He may have been the first man to set foot on the Moon, but he was also in many ways one of us, a tester – although one who would put his life on the line in his pursuit of testing. He used his pilot and his engineering skills in equal measure to probe for problems then investigate and overcome them. He later served on the inquiry into the Space Shuttle Challenger disaster.

The world cheered in 1969 when he stepped off of his lunar lander. It now must stand for a moment in silence to mark his passing.

Godspeed Neil Armstrong …

Mystery creates wonder and wonder is the basis of man's desire to understand.

It suddenly struck me that that tiny pea, pretty and blue, was the Earth. I put up my thumb and shut one eye, and my thumb blotted out the planet Earth. I didn't feel like a giant. I felt very, very small.

The important achievement of Apollo was demonstrating that humanity is not forever chained to this planet and our visions go rather further than that and our opportunities are unlimited.

Well, I think we tried very hard not to be overconfident, because when you get overconfident, that's when something snaps up and bites you.

Here men from the planet Earth first set foot upon the Moon. July 1969 AD. We came in peace for all mankind.

Testing in pictures

How we testers see ourselves ...

How we view business owners ..

How we feel requirements are written ...

How we view developers ...

How we really need to behave with our teams ...

How what we deliver is viewed ...

How our jobs can feel like some days ...

What it feels like to encounter a high level defect ...

What it feels like when we miss a defect ...

How the project manager behaves when a deadline looms ...

How it feels when it goes into production ...

[Take with a pinch of salt - just a bit of fun]

To ISTBQ or not to ISTBQ, that is the question …

It is without doubt one of the most contentious points in software testing at the moment. “Do we need an official certification to be software testers”.

There are arguments all over the place, and I've found myself inspired by an article by my fellow Twitter collaborator Jari Laakso to write up my own opinion.  You can read his article here.

In favour of certification of course is the fact that “anyone can call themselves a software tester”. Any industry where you can just call yourself a role without needing some form of registration, means that the “cowboys” bring down the reputation of the industry. I might be able to call myself a plumber, but the almighty mess I'd leave your house in, coupled with a crippling bill for call-out would lead you to curse plumber-dom if I was allowed to behave in such a way.

However against certification, there is the fact that it very much becomes a whole “industry” in itself to sell you the course, the exam and the shining certificate. And how do you measure someone's ability? Frankly the way it's examined in a multiple choice exam is laughable – give any good tester options A, B, or C and they will always come up with a brand new answer D.

I myself qualifying for the ISEB exam back in 2005 – it was a compulsary course for my company, and I took it with someone who was new to testing. I found myself wrestling with the syllabus, because in many ways the material hooks into such an idealised form of software delivery that I've never seen it in the real world. I kept finding myself asking “but what if ...” and “but surely ...”.

That said, I really enjoyed the course, and I managed to learn from some formal ways and background theory of doing things which I'd previously just done intuitively (without really being able to say way). And despite being the slight heckler in the class, I formed a good relationship with Rob my tutor, and we kept in touch for a few years afterwards.

As a test manager myself I do find myself looking for someone with the foundation qualification, as I know we'll have a relatively similar framework of testing vocabulary to work from.

However I also agree with a lot of the criticism. The exam based on this certification is based on multiple choice and a rigid syllabus. It's the idea that there's a “right way” and a “wrong way” to test, like everything is black-and-white.

Sure there are many “wrong ways” to test. But there is also no single “right way” to do it either. Testing is taking the pure theory of software delivery model, and then using imagination to find a best model for the project in front of you, the way it demands updates, the way software is developed, the methods used to update your test environment. You don't put together a test plan using multi-choice …

I had a great boss (Stephen Pedrick) in 2005 when I took my course. Not only did he book me on the ISEB course, but he booked me on a following course “Testing – Putting Theory Into Practice”. This dealt with really looking at the theory and making practical test plans, evaluating sample projects. This was almost an antidote to the ISEB course, taking the theory we'd learned (which I did enjoy) and really thrashing it in real work scenarios.

The ISEB course was good, the Putting Theory Into Practice was invaluable. Sadly though, where 30 people took the ISEB certification, only 4 took the one follow on (for which you didn't get a shiny certificate). And this indeed is one of the many potential traps of certification, believing that certification is “all you need” to train as a tester.

It's not, as Obi Wan said to Luke Skywalker, “You've just taken your first step into a larger world” ...