Monday, December 8, 2014

Voices From The Past 3: Searching for the Holy Grail

My son was born in 1998.  I found parenthood an initially terrifying experience.  Like many you're bringing a new life into a world, that you have to admit "you don't really understand at times".  Having the next generation in your hands brings that into focus - at some point ideally you'd like to help this new life deal with the world as well, if not better than you do.

Over years I would call this quest for understanding by several names.  One I've come to like is "pathfinding", but back in this initial entry, I found that "the search for the Holy Grail" seemed best ...

The Search For The Holy Grail [1998]

This is a beginning, and as I write these words they pass on into history.  This is the way of things.

This is a book about the quest for a Holy Grail, in this case not the cup of Christ, but the quest for a truth which grips and defines our lives.  It is a record of my thoughts and feeling laid out on paper, not just for myself but for my son and any other children to come.

Our Family Motto  [1998]

Our family motto is this,

Nothing of steel is made without fire
And the trials of life are the flames 
from which we draw strength

Life can be full of hardships and difficulties - all too often they seem overwhelming.

Unfortunately life does not seem to be about finding a calm idyllic life, but instead dealing with one obstacle after another, and it's easy to lose faith and hope.

In facing each obstacle, you can find within yourself new abilities to cope with.  Abilities that you're not even yet aware of.  These challenges make us stronger people.  Hence challenges can make us stronger people.

From the day we're born, life is full of difficult things to learn - if we gave up after the odd failure we'd never walk, talk, or learn to read.  But as adults we forget how hard the learning process can be and get to fear failure.  [Interestingly I wrote something similar 12 year laters]

Never lose hope.  Burn our motto into our hearts when things look hardest.

Looking back  [2001]

I have found this book after having lost it since we moved from Portland.  Rereading it, I found myself thinking "wow - it really resonates with the choices I have made in the last few years.

Recently I have found myself "pathfinding", searching for my unique spiritual and mental path in life against the needs of day-to-day life.

Of course for me, rereading this is a weird feeling, and gosh (did I ever used to say "gosh"?) do I wish I could transfer my knowledge now to the guy who wrote that first page.

But you cannot pass knowledge back in time - only forward, and that is the aim of my writing ...

The Two Forces [2001]

Our life is gripped by two great forces which while balanced give us life and health.  They are known to us as renewal and decay.

The force of decay is constant and unfaltering and come from outside.

The force of renewal is generated from within - from our heart, our mind, our body.  One day it will fail and we will surrender to decay.  But for today, it is enough.

A few years ago I saw a dead mouse on the way to University.  Each day I passed it, it was a little more decayed than the one before.

I realised the that the mouse not not decaying because it was dead.  Every day of it's life it was decaying.  But at the same rate it's body was also renewing what was lost to decay.

When it died there was no more renewal, and only decay remained.

So?  Nothing is static, everything changes.  That's part of the trick of life - things that do seem static are in a state of static equilibrium (where two opposite forces are producing no net effect) or decaying at a very slow rate (mountains erode but over thousands of years).

In relationships particularly we can think that once we are married it becomes a static thing that will last.  No!  To maintain a relationship even after marriage we still need to put in energy, time and love to keep it alive.

Of course we should not be afraid of letting it evolve - as long as we are not letting it just decay ...

My grandmother who died this year kept a journal for years, which my mother is inheriting.  I know for myself, I'm not ready to read them yet, her loss is too recent.  But I hope one day I will.  With Violet, I have found her occasional writing, and it's a bittersweet affair to remember her words and thoughts.

Voices From The Past 2: The wannabe sci-fi writer ... what it taught me ...

On my last article I was talking about how I've discovered a box of my old writing, and how looking through them has been an interesting exploration ...

What I was really excited about though were a set of old exercise books that I found which have various ages on them.  The oldest date back to when I was 13, and are my attempt to write a "story bible" for the universe of space opera stories I tried to write.  Those ones are pretty dreadful - they describe the Space Scout Fedaration, and a starship called Zeko (there are a lot of words beginning with "z", classic bad sci-fi).  The whole thing reads like such a bad fusion of Star Wars and Star Trek, that I had to check the cover to make sure it was my name there, and not JJ Abrams (nope, not a big fan of what he did in Star Trek, so not looking forward to Star Wars VII).

What interests me most is how different the style is now compared to then.  A lot of that change came from my first writing coach who was my mother who gave me feedback to get better.

I would see something in my mind, and I wanted to describe exactly what I saw, so another person would see the same.  But the problem is it was kind of dreary.

For instance, if I was writing Star Trek fiction, it'd be "the starship Enterprise moved into orbit around the planet.  It was a large spaceship some 300m long, with a large saucer front section which housed its many crew which connected to a large engineering cylinder which housed two large cigar-shaped engines on struts just behind the main saucer".  Ouch.

Thanks to my mum, I learned to let go of trying to cram that kind of detail into people's heads.  It's confusing - most people know what the Enterprise looks like, and in actual fact it's typically not that important to reading a story.  However I felt I needed that detail "just in case".  Really you don't - if your ship has been hijacked in engineering and your bridge crew need to battle down there, you can always have someone go "but Captain, the engineering section is 13 floors down, and we don't have control of the turbolifts" (obviously someone has a fear of stairs).  If you really need to, you can have someone explain something (as with that "13 floors down" statement) as a method to explain to the audience.  In Star Trek The Next Generation, everybody seemed to need to explain everything in this manner to Riker - he always seemed a bit slow on the uptake (and he kept wondering why they stopped offering him his own command).

Some of this of course came back during Let;s Test Oz, where we had a workshop on communicating with Joanne Perold and Carsten Feilberg.  The aim of the exercise was as a business owner I had to describe a Star Wars ship to my team, and they had to build it from Lego.  Yup - we were back in "describe the starship Enterprise" territory as above.

For the first round - describing Luke's landspeeder, I just played along - "it's a vehicle like a bath tub, with half way down sitting for two people behind a glass windscreen ... there is a rocket either side at the rear, and another one above on a strut".

In the second round, the subject was now a pod racer from The Phantom Menace.  This time I decided to using my mother's teachings - I'd not describe in detail, but I'd give an overall impression of it and "leave it to the team to be inventive".  We'd actually been on a tour of Weta workshops recently, and it seems when a director such as Peter Jackson or James Cameron ask for a prop, they tend to explain what it does, and overall look and feel, but the fine detail is left to the designer ... because he's a designer and designing things well is what he has experience of.

And hence I went to my team with this, "I'd like you to think of Ben Hur ... in space.  Imagine a sci-fi chariot, only instead of horses, it has these two massive, and I mean giant rocket engines.  And tethered behind is a small cockpit being dragged behind by the poor guy brave enough to pilot this thing".  The team loved the challenge, and what they built really looked amazing (damn that I didn't take pictures).

There is of course a point of this, and it links back to testing.  Throughout this year we've been discussing scripting - when it's useful, and when it's not, together with the question of detail.  A lot of people will say "include a lot of detail, because then it leaves nothing to chance".  But although you can read my starship Enterprise description and agree it's technically correct, at the same time it's just not very helpful at all.  On the other hand, Douglas Adams in The Hitch-Hiker's Guide To The Galaxy described the starship Heart Of Gold as "a sleek white running shoe".   And I tend to think what was good enough for Douglas Adams ...

For me a real awakening came in 2003, when I happened to be in a test lab during UAT, and heard someone reading aloud a script I'd written in 2000.  I'd come off an automation project, and felt that "loads of detail = GOOD".  But hearing it, knowing I'd written it, my heart sank, because it was dreadful and overdetailed.  Knowing the system better now than then, about a third of the description was really necessary.

This is why we live, we learn, we do better next time ...

Yup - I really did think is just this much detail ... but did a reader really need to know it?

Yup - this was how I described the starship Zeko at first.  A couple of years later I'd got better, and described it as  "a large, city sized disk, bristling with so many terrifying weapons".  Which I bet you can read ...

Because if you're building a ship that big, you're not going to walk ...

Voices From The Past 1: Introduction

Over the last couple of weeks we've been doing a garage and storage clean up.  I know we're not alone in storing personal items way past their "sell by date".  Hence it's time for a good clearup of items, some destined for sale or for dumping.

"What kind of things?", you ask.  Well I've found some VHS cassettes (we don't have a player any more), boxes of video games we kept "just in case" (we don't have the original game system), some old computer bits and pieces (128MB memory anyone?) and a text book on Fortran 90 (yeah Fortan ...).

What I was really excited about though were a set of old exercise books that I found which have various ages on them.  The oldest date back to when I was 13, some are from University, and others "relatively recently" after the birth of my son.  Sadly there was a couple I'd hoped to find, but they may well be MIA.

Some were short story attempts, some were lab notes (with poems in the back), some were role playing campaign designs.  I'm happy to see them all and reconnect in different ways with my past.

Most interesting to me though are the journals I've tried to create, and often been unsuccessful.  This wasn't helped by me keeping a diary during high school which got stolen, and basically the details passed around.  Ironically these days through this blog I probably live my life much more openly than anything I'd ever put into my diary back then - probably because age has taught me to not be afraid, and I realise sometimes you need someone to come out and be open about some of the key subjects I've chosen to talk about (thinking here about my mental health series).

Looking back at anything you've written over a decade ago has an element of fear around.  I know in testing over the 4 year life of this blog I've found my own attitude and understanding of testing has changed.  I'm sometimes tempted to go back and change or even delete some entries - I found this recently when I reposted The View From The Test Manager's Gallery which was originally published in Testing Circus in 2011.  There are subtle changes I'd make to that piece, especially the slight emphasis of metrics, whereas today I'd call it "reporting".  We live ... we learn.

But to my surprise there are things there that I've really enjoyed writing, and even worthy of looking at on this blog.  And hence this series where I look back at some of my writing from the past and either repeat it, or else analyse how it impacts me in the present.

What has surprised me reading - although I've only been writing about testing for 4 years, I've been trying to explore the big picture of "what is important, what matters, what is truth?".  These are good questions for testers - testing is about being the philosophers, scientists and explorers after all of the digital world.

Enjoy ...

Saturday, December 6, 2014

The View from the Test Manager's Gallery

This article was written back in 2011, and first published in Testing Circus.  It's an article I've always had a lot of affection with, because it was my attempt to rationalise my first steps into a test leadership role.  I found a lot of it was spot-on with where I'd continue to develop, however there are a few things I'd really like to change - for instance today instead of saying "metrics matter", I'd go "reporting matters".  But at the time I was for the first time seeing how project managers were using our reporting and circulating them up - the pain those reports could bring was becoming more visible to me than in my roles previously.

An interesting trivia point - this article was originally written for the Ministry Of Testing's Testing Planet magazine, but was rejected.  Which felt a little awkward as I was assisting as an articles editor at the time (hey, you can say for sure it wasn't a clique).  I say this not out of any ill-will for Ministry Of Testing, but just to make clear that we all suffer the odd knock-back at times.  In fact the next piece I wrote for Ministry Of Testing I got fabulous support editing from Simon Knight on, and the article produced, "The Software Minefield" would become the name of my first collection of articles.

The View from the Test Manager's Gallery

This year has been one of change for me – the much sought-after promotion to test manager finally happened, and with it a new set of challenges.

Several months in, I realise that although a good test manager does need to be a good tester, the skill-set and emphasis is different in many ways to that of being a senior tester or even test team leader.

Indeed, I’ve learned that the view from the test manager’s gallery is very different from that on the factory floor of testing …

You need good people

Several of the areas that have needed testing this year have been new projects, with no allocated staff for testing.  Thankfully there are several very capable test consultancies who'll send me contracted test resources.

Ticking off the details, two of my new projects were customer facing - the kind your average man-on-the-street should be able to walk in and use, with no documentation required. Systems integration testing was being done elsewhere, and we would concentrate more on usability testing, that the interface functioned “as designed”.

So I reasoned with no real needed knowledge, I could use anyone really to perform this testing, the more junior and cheaper probably the better for my project manager (who'd be picking up the tab).  The tester I ended up with was not junior, but one with more testing experience than me.

Not what I asked for, but in hindsight something I’m very thankful for - you see, I'd planned for myself to be a lot more hand-on during the test phase, overseeing what was happening day-to-day. The reality (and the best laid plans of any manager often go this way) was that when testing started (there were some delays getting a working build to us) I had another project to work on which thanks to timescales moving (test managers should be used to this) meant we were doing our best to try and get two projects tested concurrently.

So what did Brian, my experienced tester bring to my project? Being experienced, he had worked on enough projects to need minimal supervision – I set out for him on the day he arrived what I needed to do, what our testing objectives and requirements were, who the key people were, and a couple of test script samples for him to copy the style and get him started.

He went away and created the rest of our required test scripts, and got testing. Overall he used his own initiative, would inform me of his progress and any key issues/decisions, and generally “got on with it”.

With a more junior tester (as I'd requested) I'd have expected to have needed to be much more hands-on, overseeing their work, to the determent of my other project. Instead we had clear objectives, and a framework for success passed to my tester, and they used their initiative to achieve that result.

Taking a step back

As a senior tester on past projects I usually felt obliged to take on “the hard stuff” of a project, the difficult technical testing, to leave the easier stuff for the junior testers.

Likewise as a team leader last year, I felt an obligation to do the hard items myself due to my increased exposure to the technical meetings, and let the rest of the team pick up on the rest.

I realised though there's a big issue there when you step out of the role of senior tester and towards team leader or test manager. The issue is one of control-freakery. You're taking on the tough stuff to go easy on the team, but in doing so you're also showing a distinct lack of trust and respect in the abilities of your team. You're saying some parts are beyond their skills, and setting them to fail before even giving them a chance. How can the people in your team really grow if you don't give them some challenge?

I realised this when I reviewed Brian's completed test scripts. I liked them, they achieved the desired results, but … a small piece of me felt a little unhappy with them. The reason? Because his scripts were not quite how I'd have done them myself.

After a quick ego check, I realised this was something I'd have to let go. The scripts were good, and just because they weren't quite how I'd write them, it was no reason to have parts of them redone. That's petty and doesn’t matter, as long as the end result is a good testable script.

It's like when you make a cup of tea for someone, they sip it and mmm they say it's a nice cup of tea. They then ask “did you put the milk in before the water?” and you say no. They then get quite upset saying, “but you should always put the milk in first”. Does the method really matter if the end result is still a nice cup of tea? With some people and some managers it really does – they often are the worst kind of micromanager, the kind we all hated working for ourselves, and ironically the last person we wanted to end up as!

Metrics Matter

I used to get annoyed with my test manager a few years ago when he bugged me about metrics. Running around booking metrics figures felt like it just created delays, and stopped me from actually testing.

But the fact is that metrics matter.

Let's face it, testing is always the last phase before a piece of work “goes live”. Invariably a piece of work enters testing late because the development took longer than expected.

So as a test manager we often have to deal with an expectant project manager or business owner who wants to know if their target go-live is still achievable.

Metrics are an important part of this. But they are just a part. Together with any metric or graph, there also has to be an analysis of what it means. Otherwise management have a nasty habit of seeing trends in your numbers and graphs which just aren’t there.  This mastery of metrics is an important part of the test managers role, and they can be useful or a hindrance.

A discussion on metrics and communicating to management could be an article in itself! But there are a couple of general topics to be covered,

  • the progress made through test scripts
  • defects unresolved, especially “the big ones” which are high severity and we should not go-live without
  • an action plan on the “big defects” if you have one from your developers
  • an explanation of any impediments that have affected the day, such as the system being down all morning due to connectivity issues.

Management are a busy lot, and they get a lot of emails. I find that it's useful to send them these daily reports, but also to catch them a couple of times a week, and verbally given them the 2-minute summary of how it's going and what the “big issues” are.

It's useful too as during an extended test period you're sending these emails everyday, and no-one really ever seems to respond to them. So it's nice to have feedback that people are aware of the issues and risks you're flagging on a daily basis.

So the big ticket items I’ve learned to date?

In summary, it's important to have staff who can as much as possible “get on with it” and use their initiative. But you won't keep that kind of resource if you never allow that tester the authority and space to make those kind of decisions.

But as a test manager you also need to realise your role has changed, you're taking a step back from the testing factory floor, and trying to support your test team – attempting to hussle to get a test environment, design specs, software releases – to allow them to do their job. But also to tell your test team’s story, you need the dreaded metrics to communicate to management and business owners how soon they'll have their shiny-tested product in the market.

Friday, December 5, 2014

Building a testing culture

This article was originally produced as a series of articles for Testing Circus back in 2012 which can be viewed here.  I was extremely proud of the end result, and really thankful for all the effort those who participated gave.

Apart from being published as an article, I also used it with my company at the time, to show the variety of opinions within the testing community and to support the direction I was trying to move testing in.  It's an example of how the community can create a positive voice to encourage testers everywhere ...


Without doubt one of the largest challenges any tester has within their organisation revolves around building up their testing culture and identity, engaging with the rest of the business / project about what testing is about, and being able to have the freedom to apply their skills so they give their company the return they need.

Without these things, we're seen as liabilities on projects, and potentially our hard work not being either understood or appreciated. Worse still it potentially leads us to perform testing tasks which are deemed important to non-testing staff, that we feel professionally are not really adding value.

Throughout August I went around a number of peer testers, asking them the same questions, trying to collect ideas and opinions from testers across the industry, testing disciplines and the world.  The responses I got back really opened my eyes to a lot of common issues and vision across the testing community, and function as the backbone of this series of articles.


I would like to thank the following people for giving their time and passion to this project, and helping this series to represent the testing community. 

Adrian Howard  - 'Generalising Specialist' with a strong focus on more Agile ways of doing things.  Twitter id: @adrianh

Ajay Balamurugadas - Senior QA Engineer.  A man I've come to respect for his devotion to learning and teaching testing.  Twitter id: @ajay184f

Bernice Niel Ruhland - Software Testing Manager.  I regularly exchange ideas with Bernice, who is a font of wisdom.  She's also a regular contributor to Testing Circus magazine.  Twitter id: @bruhland2000

Betty Schaar - Sr. consultant and Training Services Lead at BenchmarkQA.  Another teacher of testers! I've come to know her through Twitter where I recogniser her wicked sense of humour.  I was really pleased to get Betty onboard as I'd ended up talking to a lot of Agilists, and her opinions gave a view into the non-Agile testing world, as well as worries about Agile.  Twitter id:  @swQualityQueen 

Elisabeth Hendrickson - "technically founder and president of Agilistry Studio, but you can call me whatever you want. Test Obsessed is a fine title".  I think we all know and love her as the later.  Twitter id: @testobsessed

Geoff Horne -  Prinicpal Test Manager, ISQA.  A man I was truly honoured to meet at the Kiwi Workshop on Software Testing on ethics (KWST2), and I was bowled over by his insight and experience.

Harry Bowen, Jr. - QA Contractor for Userplane.  Another tester who I've met and befriended via Twitter.  Twitter id: @Passionfortest

Janet Gregory - Agile Coach and Trainer for DragonFire Inc.  She's the Canadian based co-author with Lisa Crispin of Agile Testing: A Practical Guide for Testers and Agile Teams.  I need to stop teasing her about the snow in Canada.   Twitter id: @janetgregoryca

Jari Mikael Laakso  - Head of QA.  One of the most passionate testers I have interacted with, who is always on Twitter trying to engage with other testers and question, question, question.  That's a good tester trait.  Twitter id:  @JariLaakso

Simon Knight - Test Lead.  He is also the senior editor on the Testing Planet magazine, and technically my boss there (I'm a co-editor). Twitter id:  @sjpknight

Stephen Janaway - Senior R&D Manager in Nokia.  I have found his insights into mobile testing essential reading.  I am gutted we used to live in the same town as Stephen in the UK (before I moved to New Zealand), and I never got to meet him!  Twitter id:  @stephenjanaway

QUESTION ONE:  If you were asked to provide a testing mission statement for your organisation what would it be? Do you currently have one? How comfortable are you about measuring up to the expectation from one?

To start the ball rolling, Elisabeth provided her elegant and to-the-point catch phrase, “Empirical evidence trumps speculation. Every. Single. Time”.  I like this statement a lot – as often on projects there is a lot of hearsay and promises that a product is ready, but it doesn't mean anything until you've actually started to try and use it, and this is a great starting point for engaging with the rest of the project and managing expectations.

Bernice was rightly cautious about a mission statement that attempts a one-size-fits-all approach to testing “I have thought about writing one but not sure what purpose it would serve. For me I would rather put that time into assessing risk and testing approaches for each release we are working on. Each has its own challenges that need to be addressed. “

However the question got many thinking, Stephen echoed many people's initial reaction, “We don't currently have one.”, but then went on to say, “We should have one - previous groups I have run have had one. I'm comfortable with them - without something like a mission statement then it's not easy to get everyone behind a common goal. I think in testing especially it's important to have a common, shared goal, if only for motivational reasons.”

If I had to write one now it would be something like 'Taking a pro-active approach to uncovering product risk' - that is, it would not mention the word testing nor the the words quality or assurance. “

Indeed, Betty also ventured her own definition, saying that if she was to write one it would most likely read, “To contribute to the delivery of 'good enough' software (as defined by the software stakeholders) that serves our customer's needs and that is supported by/with 'just enough' useful documentation for operational needs. “

Having considered the idea, Betty went on, “I like the idea of having an agreed upon testing mission statement that can be used as a touchstone for making sure our methods and actions are aligned and support these objectives.  One caveat though is that since testing is cross-functional nowadays, this really ought to be bought-into by all groups that do testing.”  Which is a really good point.

Like Bernice, Jari saw the potential pitfalls of having a generic testing mission statement, “My team is testing 4 different products. Each for the same customer, but the testing is very different. There is no common mission for the team. I guess this is slightly different for outsourcing companies than product companies.”  Indeed, when I was pooling experience within my own organisation, I found very different software delivery lifecycles, lines of supports and consequently impact of risk, which led to slightly different perceptions of the impact of defects (and hence testing).

Simon was another tester without one, but would hazard an opinion that it should be something like "provide rapid feedback to developers and clients by utilising the most effective automated and manual testing techniques intelligently and efficiently."

Ajay also felt that there was no single statement that “everyone is aware of” in his organisation, but again ventured it should be about “delivering value to the customer” by “reducing (the number of) unknown customer bugs.”.  Empowered by the idea he had this to say, “I think this testing mission should be part of every tester's goals. Therefore, I am always working on that direction to accomplish that goal” by communicating this to his team.

Janet put excellently that her personal mission statement with clients was about the need to "be curious, ask questions and collaborate with all team members to work towards having a shared common understanding of what we are building in the effort to prevent defects; Evaluate and test the product to find defects, work with programmers to fix them as quickly as possible so they are not shipped to the customer."  She went on to say there was a bunch of other things she'd probably want to add, but by having too long a mission statement it diluted its goals.

Whilst not a mission statement in itself, Adrian had this acknowledgement of the importance of testing “what we do have is a passion for great products and happy customers. Testing helps us make those happen”.

I have encountered other excellent statements for the mission for testing have come outside of this questionnaire.  I like my colleague Pam Forrest’s response that testing was a foremost about “ensuring a product was fit for purpose … that riskier areas which could impact business and customers have been evaluated within the limits of test system”.  Likewise, I found myself nodding along when James Bach says “the purpose of testing is to cast light on the status of the product and its context, in the service of my clients”.

QUESTION TWO:  Do you feel the attitude towards what testing is and what it should deliver has changed during your career? What do you feel is driving that change? Do you feel this is good or bad for testing?

Stephen reflected back on his journey with his current company, “within Nokia testing has become more accepted and is now seen as a critical aspect. Time is given for testing and resources too; this was not the case when I joined when one manager remarked to me 'can't we just find some housewives to do the testing?'.”

He continued to warn about the potential Cassandra syndrome, where testers become too linked with news the rest of the business is not prepared to believe, “I believe across the industry testing is becoming more accepted although this is mainly due to the push from the community rather than a pull from outside. We need to be careful not to push too much or risk being ignored as the ones who always shout perceived bad news normally are.”

Adrian was originally a developer, and “my early encounters with testing folk were not good. Very bureaucratic. A focus on obeying the letter of the process, rather than the spirit. For example getting new features that were making customers happy marked as 'defects' because they weren't in the original spec. The testing folk 'owned' quality, and that ownership seemed to cause problems.”

He mused that such Testing Police who were “often very focused on checklists. If it passed everything on the checklist it was okay - even when there were glaring problems elsewhere”.

However that was in the past and “I now realise that what I had encountered were bad examples of the testing profession, but without knowledge and experience of 'good' testers I tarred the whole profession with the brush of badness. I know from conversation with peers that I wasn't alone in this view. Developers views were being shaped by their encounters with bad testers and bad testing departments is still a problem I think.”

Adrian during his career had become involved initially in XP programming and later Test Driven Design in Agile.  It led to a real change of understanding, “I started reading more about testing. I was fortunate enough to work a couple of excellent testers, including those with a strong exploratory testing focus. I saw people doing much more inclusive testing with the team as a whole. I saw the value that having people embedded with the team having a strong testing specialisation brought.  I saw testers as people who helped us build quality in, rather than people who (poorly) found problems after we'd built stuff.”

Elisabeth had a similar warning tale from history “when I started in this field there was a trend toward separating testing and programming. That was a dark, dark time in our industry. It led to massive amounts of waste and ironically lower quality at a higher price. Fortunately the rise of Agile has resulted in organizations re-integrating testing into the development cycle, and often integrating the testers and programmers within teams. This leads to faster feedback and a whole lot less duplicated effort, wasted time, and higher quality.”

Janet had this to say, “I definitely think there is a difference, but only for those teams moving to Agile ... What I see more these days is testers trying to be more proactive trying to collaborate with the programmers and customer's more, worried more about delivering the right value than adhering to a given set of requirements.  However, it is a slow and hard change for many because of the radical mindset shift.”

Not everyone was sure that Agile was the way for all software development to go Betty had some concerns, “I've been frustrated a bit by the whole agile movement because of the general lack of focus on documentation (and formal requirements). All is fine so long as the group that started the work stays with it forever but that is not a good expectation in this day and age. I do not advocate for documentation that isn't of value but it's no small task to determine what is just enough/good enough documentation.” 

Given Betty's background in medical scanners, this is understandable – surely in some environments detailed requirements and strict gateways between project phases made sense.  I had some difficulty seeing some of my avionics programs I'd worked on being Agile over V-model.

Harry too had felt a little burned by an Agile project, where going Agile was, in his view, seen as an excuse for 'no documentation'*.  “Testing was made difficult without said documentation as the requirements were never set in stone, but rather evolved with the slide of Executive opinion, which could change at a moment's notice.  We went full circle on one of our products in 13 months.”

[* Of course Agile is about so much more than just “no documentation”, it's closer to Betty's definition of only having documentation which adds value]

Ajay felt that testing has changed within his organisation, and he felt two powerful agents for this change was through, the “reputation and credibility of the testers” and “general awareness (they raised) through testing war-stories”.  Reminding us of our social obligation to build and communicate our values to the rest of the organisation.

QUESTION THREE:  Are there obstacles and expectations around testing which you find challenging? How do you confront them?

A colleague from the UK who wished to remain anonymous came forward with what would be a common theme, “Time. Time. Time. Development slips and gets more time.  The result?  Testing gets squeezed. I don't think there'll ever be an answer to this, you just have to shift your risk analysis parameters”

Geoff had this to say, “testing is often seen as the poor cousin on the SDLC & project teams often consider a project delivered once it is released to test.  Testing in New Zealand is slowly changing however when budget and timeline are the primary considerations, often very difficult to translate testing into benefits over the medium to long term.”

Betty echoed this, “one aspect I find frustrating,  is that the amount of time and the amount of resources needed for testing is so often way underestimated. We're always having to justify our needs and often still not given the adequate time/resources.   I have been known to use the argument that the old rules of thumb for testing resources should be thrown out (n dev/1 tester) unless we're willing to go with double or triple the testers to dev ratio.”

Elisabeth felt herself time and again challenging the prejudice that testers were somehow the bottom of the project ladder, and people who failed to make it as developers, “people expect testing to be easy. 'It's just testing,' they say. But good testing is difficult. It requires an understanding of what we're building and why we're building it, what can go wrong, and how to ferret out the bad things before they bite our customers.”

She put the case for a better valuation of what the abilities testers need “that means that people who test need to be skilled. They have to understand technology, the business, analysis, and test design techniques. Testing yields information. The quality of the information depends very much on the caliber of the person testing.”

However, because the product of testing is an intangible and ephemeral thing, it can be extraordinarily difficult to tell the difference between a skilled tester and someone who is just going through the motions. I suppose that's why it is so tempting for so many organizations to view testing as something that anyone can do with absolutely no training or skill. I have been fighting this 'anyone can test' attitude my entire career, so I doubt it will get better any time soon.”

Repeating our common theme, “furthermore, organizations tend to baulk if they view the cost of testing as too high. They try to measure the cost of testing separately from development, and minimize it. They fail to understand that testing isn't like insurance where it makes sense to make a risk tradeoff. It's more like scaffolding where the building just isn't going to go any higher up unless you have it. Any attempt to minimize the cost of testing is just as likely to increase the overall cost of quality. It's a systems thing and sometimes difficult to help executives understand.”

Simon echoed the feeling about testing being the poor cousin even on some Agile project and often 'the last to know', “I guess the other thing that tends to annoy me is not being involved in more planning/scoping/requirements gathering type meetings. I always see these as areas where test can add value, but nobody else ever seems to agree”

Bernice echoed the issue of attempting to complete testing against deadline, “I think one challenge is time. To manage time, I need to understand priorities plus perform risk assessment before a testing cycle begins and throughout testing. This helps identifies the scope of testing which may change as we learn more and determine how to pull in different tester's strength into the testing.”

Elsewhere there were several comments about the common misconception that testing is responsible for quality (and breaking the product).  Stephen spoke about challenging the idea “that testing assures quality. It of course does not, it highlights risk, and this is the main challenge - carrying out a role which is seen, when executed correctly, to be raising issues while taking none of the responsibility. Confrontation is best done through education and experience.”

Adrian had much the same experience, remembering that “I still encounter the old testing-department-owns-quality attitude occasionally. Changing that can be hard since it's often embedded into a larger organisation problem of how departments are rewarded/graded.  This is made especially hard since it then colours the rest of the organisation's views on what the value of testing is.”

Harry too talked about a changing role and relationship between tester and developer.  The responsibility for quality was not a 'tester thing', it was something they were trying to challenge, though “the idea that Developers should be responsible for the quality of their code is a bit foreign and partly not fair to them.  They write, we test.  As I pointed out in the presentation, 'Welcome to the 21st century!'.   Writing code and testing it are two different mindsets.”

But just as developers needed to become more test driven, so too testers needed to become more technical, “for me at Userplane, I'm testing mostly web technologies with a sprinkling of mobile.  Becoming adept at Javascript  and Selenium as our automation solution is imperative for success.  My thought processes in testing and managing the process must now include code ability and automation.  The impact is beginning to be felt and they are (so far) very happy with the QA mindset.”

Simon was another tester inspired to get more involved in the technical side, “sometimes I think I'd like to try my hand at writing some production code. I work on a small agile team and really there's no particularly good reason why I couldn't pitch in other than division of responsibility. I don't push it too much though because I've got plenty of other stuff to crack on with usually. Perhaps I just need to get stuck in rather than waiting for an invitation! “

A few anonymous colleagues from the UK and Europe echoed a familiar story “the management and sales teams don't understand testing”.  With the perception that“testers are causing delays on projects.”.  Some of this harked back to a comment that Elisabeth made that the team from developer to manager need to learn to accept that “it's not done until it's tested”.

Ajay touched a little on challenging this, also going back to touch upon on Elisabeth's mission statement of “empirical evidence trumps speculation” by stating that testers need to build “reputation and credibility … You slowly build reputation by showing them the proof to start with, then you show the bug again.”  And that he has started to record and share his testing sessions to emphasise this point.

He is also driven to save time and effort in testing, going on to explain, “the whole company was using test cases and then shifted to user stories but in an excel which took an hr to go through multiple reviews - peer review, development review, product review.  Now, after my demonstration of using mindmaps, the entire company has thrown away user stories in excel, started using mindmaps for review, tracking and saved a lot of time.”

Janet asked about why we expect everything within testing to be written up, even when other phases of the project take a lighter approach, “one of the biggest challenges / obstacles is the need for massive amounts of documentation required - mostly as a 'Cover Your Ass' response. I think this stems from a blame culture of many organizations.  It is hard for testers to let go, and realize that maybe we don't need all those metrics, and documentation.”

QUESTION FOUR: Is there anything your team does to promote and raise the profile of testing within your projects?

Stephan talks about how Nokia encouraged the whole team to be involved with testing, “we are organised as cross disciplined Agile teams and share the responsibility for testing when the deadlines come. Each team has (specialised) testers who will do as much testing as they can in the sprints”, however when push comes to shove near a deadline, everyone from developers to UX design gets involved in testing, with the specialised testers “as test advisers to others  … We use a lot of exploratory testing (ET) and session based testing which help share knowledge and experiences between the whole team.”

The result worked, as “distributing (test team) members to each project, and getting them sitting in the project teams, was the best thing I've seen in the last 12 years for promoting and raising the profile of testing.”

Betty echoed the importance of good relationships with the rest of the project, “fortunately I have been able to generate good rapport with the team and management pretty much wherever I've been and it's by repeated project successes that I've been able to garner the respect for my role. I speak up, point out issues in a way that is non-threatening to individuals, I highlight where actions taken have prevented problems, I work on developing rapport and generating good-will.  I use jocularity liberally as I've found humour to be an invaluable tool for making the work environment fun.”

As a test consultant Elisabeth observed that, “I don't usually need to raise the profile of testing, but sometimes I need to help my stakeholders understand that it cannot be an afterthought. I help them understand that features are not "done" until they are tested, meaning both checked and explored. And I help them understand the connection between fast feedback cycles and lowered risk.”

Geoff echoed much of this sentiment saying it was important to create “continuous impression upon leadership to test early, test often, test properly, test appropriately”.

Harry picks up on a theme from Ajay about building credibility within your organisation “on the previous team, we pushed for other teams to allow for us to do their testing to save them time and resources.  We developed a great reputation for fast and reliable testing.  It allowed for the company to move Mobile to our shop, and  consolidate desktop testing“ within his team.

Ajay said he helped to advance the culture of testing though introducing innovation such as mind maps, as well as developing a reputation for “raising good bugs and good bug advocacy” and engaging with stakeholders and peers about “conducting testing talks” to convey testing values.

Bernice talks about engaging with the rest of the organisation by “providing information on what we are seeing in testing to the developers and business analyst is a value-add. For example we may ask a Business Analyst if what we are seeing is what they expected from the functionality. We try to understand the bigger changes that are in the pipeline to allow us to provide any input and initial risk assessment. We show that we can add value to the overall process and not just test at the end of the process. We offer to test code in the developers sandbox if they would like our testing input earlier. “

Janet really sums up a lot of answers here.  The core thing being “I encourage all the teams I coach is to make testing visible. Create testing tasks for all testing activities. Use big visible charts such a a release test matrix to show progress, as well as extent of testing. Test results - keep them visible on the continuous integration”.

QUESTION FIVE:  What changes have you seen introduced to your test projects which have made significant impacts?  Were these changes met with initial hostility or scepticism?  How was that overcome?

Stephen feels a significant change came within the “team structure as a result of the change to the Agile wow. I've also introduced automation (as well as), using testers to pair test with developers,”.  At Nokia there were “no problems with hostility”.

Stephen then harks back to a common theme of getting testing visible, and getting everyone involved, “as mentioned above, changing to exploratory testing and SBT was a big change. Initial sessions used only testers; we invited everyone but since it was optional and testing related then only one developer every showed up. Some bribery was needed (cakes essentially) to draw people in, and after that we got a steady number of developers coming to the sessions. They would really add value with their different view-points and it was very motivating to the testers who organised the sessions to see them there.”

Betty talks again about the technical challenges of modern systems, “at one point I was having to test service oriented archictecture/middleware.  That was new for me and new for the organization that I worked at. We had to feel our way through to creating a test harness to adequately test the code without a calling application (since there weren't any yet anyway).”  But salvation came in the form of relationships, “fortunately the development team was responsive to helping us create tools to do a good job of testing”.

She also wasn't afraid to question the expectations of what testing should be delivering, “we also found that we had to deviate from standard deliverables mandated by the organization because we were a square peg being forced into a round hole. At one point, when the standard documentation wasn't giving me what I needed to do the testing, they asked me to define what should be contained in the design documents that would be useful.  I have never shied away from digging in on the technical stuff and that has taken some developers by surprise when I start asking technical questions.  I've seen initial resistance to that line of questioning but eventually a new level of respect and rapport with developers when they see how what I am asking for can help overall.”

Elisabeth makes a good point about challenging where testing should sit and 'who does testing', “I don't have "test projects." Testing is part of any project.  That by itself is a change for many organizations, and one that is sometimes met with hostility and scepticism. But when I work with organizations, part of my mission is to help them integrate testing throughout so that they don't end up with a huge inventory of unusable source code that they then have to spend months stabilizing before it's ready for prime time.”

It takes a whole lot of patience to get people within an organization to shift their thinking from test-last thinking to testing throughout. It typically gets easier when they have experience with the horror show that test-last practices yield, or if they see big improvements from integrating testing throughout their projects.”

Harry like many wanted to introduce new ways to his organisation, “I pushed for exploratory testing to be done as a means of exposing bugs differently.  Skepticism was high so I had to continue writing out test cases and pursuing the exploratory on my own.”  Again the common theme of reputation and credibility kicks in, as “ after a few months of exposing things never before encountered for us, they got the point that we needed to stop doing things 'the way we have always done it'.  We tried for years to get automation brought in to allow for us to get past regression and allow it to be done via automation.  It took until this March to see the beginnings of that implementation (selenium).  As I pointed out in a presentation I gave last fall, test cases are only as good as the last bug written against them.  Exploratory testing will allow for us to do code-path/decision point discovery.”

Jari makes an excellent case for doing the ground work within an organisation and building the relationships by talking and involving others, “I had to make a reorganization of a test team once it grew  from 2  to over 60 people (over a relatively short time). I decided I wanted some team leads/senior assistants in the team as most of the testers were very junior. A few of those people didn't have even a year of testing experience, but they had the kind of attitude and personality I thought would fit the role.”

As many will sympathise not everything was embraced, and “in the beginning the change was challenged quite a lot. I spent a large amount of time talking with individuals, the small teams and the whole team to explain them why I made that change and why I chose those people. I also made it very clear I welcomed comments about the changes.  After some time things started calming down and the team was again productive. Some hiccups here and there, but those are always coming when dealing with people. All in all, I am very happy how we managed to overcome the change as a team.”

Ajay talks about the changes he introduced “some of them were challenged but as soon as the value was demonstrated, others could also highlight the value, people embraced it.”

Bernice makes a good point, reiterating Jari's point about engaging with people, “I find people are open to change if you do not mandate it and if you allow them to participate in pilot programs to understand testing approaches to adopt.”

Janet underlined the importance of starting small and winning people over to create a momentum of change, “my life is about introducing changes, many of them with significant impacts. Most of them are met with initial hostility / scepticism.  I find that the easiest way to overcome them, is to show success. I encourage start with something small and easy, see success. Then move to the next thing. Also, a really important thing to develop is relationships. Sometimes I introduce a practice to one eager person(s), get it to work successfully so that there is a frame of reference, and we can showcase that success to tackle the rest of the team members.”

QUESTION SIX:  Working smarter and not harder, are there any activities you’ve found your team focusing more on?  Consequently are there any thing you find yourself doing less of?

In a common theme involved testers questioning how much value detailed test cases and documentation really delivers, Stephen said that he saw testers “doing more testing and writing less test cases. In the beginning, using waterfall, we used to write hundred of tests, taking weeks then run them. Regression efforts took huge amounts of time as did maintenance of the test cases. Switching to automation, using dedicated developers in test, then swapping most of the manual effort to exploratory testing empowered the testers by recognising their expertise, whilst also removing a large amount of time spent documenting and maintaining test cases.”

Stephen had one final concern, “one area that does need to be checked at the moment is the feeling that you can just automate everything” over understanding the skills that human testers bring to a project.

Simon sees Agile being a big part of the way many products are developed in future, “Agile Test Driven Development tests in a continuous integration environment are VERY helpful and mean I spend much less time regression testing. Mobile (development and testing) is a big area of focus for the future.”

Elisabeth repeat the common themes of more “exploratory testing plus the disciplined engineering practices associated with Agile including TDD, ATDD, automated regression” going on to talk about how we need to do less “bug triage, applying test reduction strategies in an attempt to cut down the regression cycle, manual scripted testing, test documentation (including traceability matrices).”

Harry looks back at how his team has evolved and will continue to evolve “we do less manual (scripted) testing and more exploratory as well as learning/implementing automation.  What passed in the past should continue to pass and it should not fail.  Need to keep moving forward.  Once the bug is fixed, test case passes.  If automation picks up too many bugs, someone goofed”.

Betty examines the ways of working she's finding adding value in order to make testing for her clients simpler and more focused on adding real value, “I use things like test checklists, common tests, decision tables over detailed test cases where I can (and it makes sense to) in order to work smarter.  I use scenario based tests for testing third-party software unless/until issues are discovered that warrant more detailed functional tests. As a contractor who is intended to go away, I have felt strongly that I should leave behind test assets that can be used in the future.  I also test lots of third party software and shouldn't need to be doing intensive functional sorts of testing so the concept of 'just enough' testing documentation is my motto.”  On a technical level, “I have used tools to identify and pull production data, scrub production data or to generate/manufacture test data.  Virtual Machines are a huge time-saver/help when testing multiple client computer configurations.”

 Bernice  sees the value of whole-team approaches “working in teams to approach testing problems allow more testing ideas plus cross-training for the testers. Spending more time on risk assessment throughout the project to understand how risk change and to ensure that initial risks were addressed. Identifying different tools the testers can use based upon what they are testing. This can include Session-based testing, mind-mapping, test matrices, testing ideas - the level of documentation is based upon what is useful and risk level. Plus it should be a living document as it should change as we learn more through testing.

Betty has taught testing classes, and sees in modern software that databases have become more and more the heart of every application, thus “I've been surprised how many testers do not take the step of doing SQL queries to verify the stored data and simply trust what the UI shows them.  Our testing work has definitely gotten more technical over the years and that is one aspect that I've noticed that I have had to grow my skills in order to be better at doing my job - reading database schemas and related documents, design documents, writing SQL queries, etc. have all been skills that I've developed along the way. Long gone are the days of black box testing where we only needed to focus on inputs and outputs.”

Bernice picks up on the importance of engaging with ideas from the world-wide testing community, “we hold a lot of journal clubs and focused discussions with testers who are interested in exploring certain areas. For example, Michael Bolton's News Story approach from STAREast is a popular discussion with testers on how to implement his approach. It is important not to mandate what the testers must do for all testing assignments. Each should be assessed for the proper testing approaches or tools. Whereas a session-base test may work well with one testing assignment, a traditional boundary matrix may work better for another testing problem”.

I will leave the final word with Janet, who sees the future of testing quite simply, “faster feedback - continuous feedback”.


From all of this there are a number of key take-home points which come across, many of which come as no surprise,
  • It's important to engage with stakeholders, and make sure your testing goals and theirs align.  If not, how can this be achieved.  As Ajay points out time and again, try to get them to think like testers through education.
  • For testers to add value, testers need to be valued.  Challenge the idea that “anyone can do testing” by making your testing visible, and building your credibility.
  • Build relationships.  Testing is more effective if it pools on the ideas and experience of the whole project (developers, market managers etc), not just testers.
  • Innovate.  Try to challenge the idea that “we've always done testing that way”, especially if testing has always been that way and delivered little value.
  • Pioneer.  If documentation is not adding value, how can it be made to?  Or should it be dropped altogether?
  • Time is always going to be a constraint, and thus it's important to always to do risk analysis. Assess the risks and limitations in your testing, and give feedback on them to the project leaders.  Help them to make informed decisions.
  • Software is getting more complex, and as testers we need to not shy away from that complexity.
  • Challenge the idea that testers are responsible for quality.  The whole team is responsible for quality.