A tale of Bill and Jill

Over the weekend, we lost my great-aunt Jill – she was in her 80s, and thankfully suffered only a brief illness. 

Death is always a time for reflection – and this time is no different. But I'd like to really take a moment to reflect back on her life, as she and her husband Bill influenced me in ways that only death gives us clarity on.

My uncle Bill was related to me on my mother's side. He was my grandfathers younger brother, born into a mining community in Stoke-on-Trent in the 20s. My grandfather is one of the smartest (and most stubborn) men I know, but Bill was academically brilliant, and was offered the opportunity to study at Oxford – almost unheard of for someone from the lower classes at that time. This study he achieved with some financial support from my grandfather.

There Bill met Jill, and they fell in love. But Jill was a spirited woman, and when he asked her to marry him, she turned him down, as she desperately wanted to do voluntary work overseas, and wasn't ready to settle down. She travelled to Africa, which she fell in love with, and did all sorts of educational work with the local tribe, becoming an “honorary chieftan” I'm told.

Bill meanwhile turned to teaching – and taught my father across town. When Jill returned from Africa, they ended up picking up from where they left off, becoming married. Their house was a fascinating place filled with relics from Africa such as tribal masks and a wooden case she'd used in her travels which she gifted to my parents.

They were regulars for our Christmases, which we'd spend at my grandparents. With them both being teachers, they'd arrive punctually at 11am on Christmas Day, and they were so entertaining. We never got toys, only books – typical teachers! I remember Bill being able to be quite funny and yet coerce us into doing things – asking one Christmas if we could find the “stealth function” our Starbird electronic (and noisy) spaceship toy. Such a comment could have irritated, but I remember finding it was funny, and played along. It's no wonder he became a headmaster.

Sadly though he died in his 50s when I was a young teen. It was a tragic death, that shocked us all. When I look back, I feel slightly robbed, as I cannot help think how much I would have loved to talk to him when I was older, when I was going through University, and when I went through teacher-training. I'd have liked that a lot.

Inevitably we saw less of Jill without Bill, but we did still see her. And my grandfather being the person he is kept tabs on his brother's wife, as often as possible. She was always family to him.

They left me with a legacy – I was inspired to give teaching a try because of them (although I didn't end up sticking with it), Jills tales of overseas made me want to try living abroad – which inspired me to take the opportunity to study in Jena, Germany and gave me enough of a taste for adventure to consider travelling from England to New Zealand.

But most of all they left me with a book. When I heard about her death, I immediately thought about a book I'd been given by them on Astronomy. As a child I was fascinated like so many with space and the stars. So they bought me a book on it – it was full of stories of the planets (although Voyager 2 hadn't gone to Jupiter yet), about Skylab and the Apollo program, about how stars worked. It was one of the last books Bill and Jill would give me.

I would read through it time and again, and it would inspire me, and fire my imagination. It fuelled a passion for science, and made me want to study Astronomy, which I eventually did at the University of Sheffield. But even though it became dated - there were later missions which gave more information and better pictures - I kept hold of it. I threw other books away but never this one.

At University I would read some pieces to refresh my mind on the basics, before opening my University books, and putting the mathematical framework around what I'd just read. As a teacher I would look to it to work out how to simplify the detailed concepts I knew so younger minds could digest.

But this weekend I realised that the reason I could never throw it away was the book had inspired me, and reminded me of two people who I loved in my own way.

It seems an odd and maybe failed epitaph for a couple to be summed up as “people who bought me a book”. But not if it's a book that changed your life, not if it's a book that inspired you.

Christmas is coming – do something amazing for someone, and buy them a book!

The last temptation of Pekka Marjamaki …

First off, this is all a bit of fun – so don't take too seriously (or consider me a Satan worshipper) ...

Last night Pekka Marjamaki was on Twitter talking about a Tarot reading he'd had done, and if anyone could interpret for him, so we caught up on Skype to discuss it.

I'm a follower of Jungian psychology, and I believe that inside of us is a subconscious that is trying to guide us as best as possible, and sometimes warn us. Unlike our conscious, our subconscious mind cannot talk to us in words, it lurks below our babbling thoughts and can only make itself aware to our conscious mind through dreams and feelings. Sometimes the subconscious should be followed, sometimes rejected – but as often as possible it needs to be understood.

For me, Tarot is one method I'll use to try and check my subconscious, it's part of my toolbox. I don't believe Tarot predicts the future – but I do feel it allows me to read minds, more specifically my own. This isn't magic, this is psychology. But it's still useful.

To me it works like this – a Tarot card is loaded pictorially with very rich and often complex symbolism. When you look at a card within the context of a reading, there will be an initial reaction to it, that reaction is the whole point. It reveals something to you, if you listen to your own mind.

And so when Pekka told me about the cards, I looked up a picture of each card on Google images, and stuck with the reaction I got from that image. Hence when I say this is Pekka's reading, in truth it's more accurately a reading of myself than Pekka.

Pekka explained that he'd asked for a reading about his software testing career, and three cards had been chosen – past, present and future (which provide the context). Put the cards together with my reactions and intuition and it tells a story … this is what we pieced together …

Past – The Priestess

To me, this card really triggers the ideas of learning and being mentored. The Priestess is an enlightened soul, but she's also a keeper of ritual. If you look, she's following instructions from a “Holy Book”. The book guides and enlightens if read with wisdom. But it also can enslave and control withing a prison of conformity we're not allowed to challenge.

This to be really talks volumes about many of our pasts in testing. We learn the secrets of testing, but testing can become an instruction manual – you know, just follow the script in the Holy Book.  For a good few testers, this is all it becomes, slavish routine.

Present – The Magician

Whereas the Priestess follows the book of instruction, the Magician in this picture feels different. This feels like an individual who is in tune with the nature of things. The scroll in his hand is not something he's slavishly followed. It's knowledge he's gained with his own insight, it belongs to him, but he rules that piece of paper and its contents, not the other way around.

This does feel, especially in the context of it's position in the present like someone who has evolved. They have listened to the Holy Book as a Priestess, and they've taken the next step. They don't need the book anymore, they've taken that knowledge, and become attuned to the nature of testing. It is this enlightenment from within and from their observations which now guides them.

In many ways they have become a thought leader.

Future – The Devil

This is a complex one, but as you'd expect, The Devil denotes temptation.

If you look back to the present where The Magician denotes a thought leader, this denotes all the ways we can fall from the path. Perhaps its a desire to in some way “sell out” about testing - not because we want to, but because either there's something to gain, or perhaps because championing testing can feel too hard at times.  We might know how testing works, but go along with following a test strategy as outlined by someone else, even though we know it won't work.

Elisabeth Hendrickson's article Why I Won't Go Back was very much in my mind with this card (not surprising, I'd read it earlier that day). And in it she talked about how sometimes under pressure as testers we'll go along with Cover-Your-Ass activities on projects, which we know won't help the project, and deliver little real value. But we do them out of fear or because we feel it's expected, and somehow we feel our performance and contribution will be seen better if we follow what non-testers expect.  The one that came to mind for both myself and Pekka was “following an ISTQB policy, when we see little real value in it”.

Both me and Pekka, found it an interesting conversation, and these themes (unsurprisingly) really resonated with both our experiences in testing. For me the card about the future really reinforces the responsibility we have when we become enlightened testers not to sell testing short, because it damages the whole community when we do.

Notice in the picture above - once we give into the Devil, it makes slaves of us all!

* For the record, no animals were sacrificed during this consultation.  However when I look at Pekka's current profile picture on Twitter, I do wonder what he's doing to that chicken ...

Oh brother ... it's change management ...

I was really pleased to get such positive feedback on my article yesterday on testing proposed changes before they hit production. But special mention has to go to the following insight from Simon Talks, who is not only my brother, but a very successful Change Manager who is based in Sheffield, England ...

Hey Bro!

There is considerable value in what we term the pre-production environment.

The pre-production environment for us follows all the restrictions of the production environment and devs have no access it (as they should with production!!!).

For changes planned for production there is always a dry run on pre-production and some testing is ran against the pre-production environment.

Far more changes fail in pre-production even if they run in the development and test environments, due to complications in parallel development that usually enable releases to work that wouldn't work in the production environment.

Testing the process of change ... lessons from HMS Invincible

Back in 2003 I worked for a company who provided the software used on the shipboard servers of the Royal Navy. The servers were used as an information network to share important documents between ships around the world. Surprisingly rather than all being top-secret intelligence, most of this was housekeeping. A naval ship essentially share many characteristics of an office, with a large number of people aboard who need to share information on activities, events, usage of disposables (you'd not be too pleased if your office ran out of coffee or toilet rolls), it is in many ways a floating community, and the Naval intranet allowed the organisation of many aspects of daily life.

I was a developer at the time, and following a course had been working (initially at home) on a Perl script which I got my managers permission to pitch to our customer. I'd written my application from scratch, but today we'd recognise many of it's attributes as a shipboard-Wiki (I called it the Generic Web Page, having never heard of Wikis at the time). It allowed sharing of information pages (which could be instantaneously modified by users with suitable permissions) between both personnel onboard ship as well as between other ship servers. Whereas before ships saved and exchanged information in document form (which took time to replicate on the Naval intranet), my ship-Wiki could be updated instantaneously.

The Navy were impressed. Although it was in their opinion not suitable for anything secure, it had a lot of potential uses due to it's speed over the document sharing method that was in use. They wanted to try it out in an upcoming NATO exercise onboard the UK flagship for the exercise, HMS Invincible.

Of course management was delighted to get such an enthusiastic customer response. But they had a fear – could I install my Generic Web Page application safely onboard the Invincible server? This was their dilemma … if I got this application working, it would be a huge statement about our companies can-do attitude. And if it worked well, we had all kinds of ideas to expand the product and provide more of these kinds of applications as a whole new line of work.

However if something should go wrong – then we risked knocking out the information sharing ability of the Royal Navy flagship in what was likely to be a billion dollar joint-forces exercise. The Royal Navy would look bad in front of it's NATO partners, and we would look unbelievably incompetent to our customer.

Risk and gain – all change has elements of both. What was decided, we took a replication of HMS Invincible onto a spare server, and we rolled our change on, we ran the server for a day. Then we rolled the change off, and checked we'd removed it.

We did this to develop a list of steps to apply the change, and to confirm we were drilled in it's use and application, but mostly to check and explore for any potential issues we might have. We wanted no room for error – so we did this in total of 4 times. Then on the 5th day, we used our steps on a completely different server to make sure there were no other potential surprises (unexpected configuration perhaps).

We produced a report, and our Naval adviser was satisfied we'd taken adequate precautions, so we were allowed to install on the HMS Invincible herself. That was quite an experience to get in the field and perform this change – amazing to be in the belly of such a grand titan. Did you know the ship has a small supermarket called the NAAFI inside that sells soft drinks, chocolate bars etc? Yes the ship has more people and shops than some villages I've lived in!

The installation was a success, and the software proved itself within the exercise. The Navy decided it wanted to develop that kind of capability, so the piece of work was extended to a much bigger program, although I didn't continue to develop on that project.

Testing the process of change

In testing we get quite used to our test environment – during the course of testing it undertakes so many changes and tweaks to get things right. Sometimes it's new builds (which are easy to track under configuration management), sometimes though there's a setting tweak that a developer tries which perhaps they made but forgot to write down.

At the end of testing, you produce an exit report which signs off that what is in your test environment has been checked, and seems acceptable for production.

But how do you confirm the change outlined for production will produce an environment which echoes the one you've signed off? For most releases to new systems, it's usually not a new build that's applied, but a series of changes just to modify elements of your applications and settings.

How can you test that your release team has all the changes needed for production? Well our enterprise on the HMS Invincible was a good start. Start with an environment under test which mirrors production, apply your changes, and test the end result as you would a production verification test. Then roll back, and confirm your changes have gone. Encourage the change team to repeat this process under as production-like conditions as possible (especially regarding timescales). Are any services interrupted during the change? Do any defects encountered during testing seem to come back (that is, you've missed a change somewhere)? Can you do the core, high value actions? Can you effortlessly roll back? Have you developed a definitive list of steps and actions that work every time?

It's amazing how many times we take for granted that what we sign off in our acceptance test environment will be delivered to production. Is your project making steps to ensure nothing's been missed?

Experiences in automation ... WeTestWorkshop

I have never lived anywhere quite like Wellington. The thing that constantly amazes me about this place is the sense of community amongst technical folk, aided by the various meetups which are organised by people who are passionate about their relative crafts.

I'm already a regular face at the AgileWelly events that are organised. I was thus noticably pleased when following KWST2, several testers decided we needed a more regular workshop, and thus the WeTest workshop was born.

This Thursday was the first event, sponsored by the consultancy I used to work for, Assurity. Katrina Edgar (the organiser behind the whole WeTest meetup) led with an experience report on the subject of test automation. It was a potentially tough crowd – about half the room were ex-developers turned testers, and 75% had experienced test automation in the last two years.

Katrina, like myself, was an ex-developer turned test automation specialist. She'd encountered 3 major projects so far in her career ...

On Project A, she was allowed to branch out into automation on a typically very manual task. She'd been given enough free reign to choose whether to automate or not. She used it to automate the very onerous task of setting up multiple configurations, which removed a lot of monotony from her tasks. The work was relatively low cost, and reaped high benefit. But that said, she felt just left to get on with in, and her automation was expressly that, HERS. It was never reviewed, and no-one else ever used.

Project B was more advanced. She came onboard and there was an automation framework already in place – a Jenkins machine which provided continuous integration tests using Selenium scripts. She felt where as Project A was more about “what are you testing” with the free reign on “how are you testing”, life here was quite the reverse. Everything was about “how can we test on Jenkins”.

The system had a Concordian front end to show whether tests had passed or not, and it was considered that if the test passed, it was “all good”. There were no follow-on manual tests, because it was believed “well our Selenium scripts test it all, don't they?”.

The problem was that the testing was highly technical, and to understand the script content, you had to be able to read Java. This meant not enough people did read through and understand the scripts and what they did. Testers would modify and make new scripts, but no-one ever reviewed them. So on the whole no-one could be sure whether these tests did test the core values of the project.

Project C echoed a lot of Project B. It was a similar system where everything had been done by automation. But it was a much older, legacy system, and all the original staff, and much of the expertise had moved on.

Thus the scripts were flaky, and needed a lot of maintenance by people who didn't really understand them all. A lot of time was spent fixing them, but no-one knew everything they did. But they'd always seemed to work before, so no-one wanted to tamper with them too much either.

Her experience report finished, the discussions around the room began. And this is where a peer-conference drastically differs from presentation-based events. It's much more interactive with many joining in, asking questions, sharing tales. Whereas in a presentation you walk out with on persons experience and ideas, at the end of a peer conference, you've had those ideas pooled together by everyone in the room.

Thus by the end of the 2 hours, we'd investigated and reached consensus in a way which was surprising to both myself and Katrina. In fact no-one could have predicted it – which is what can make these things so exciting. These were some of our take-homes by the end of the night …

Know why you are automating

Automation should be almost always about addressing a pain if you tried to do something manually. In particular it should use some strength of the computer against an area where a human being is much weaker and slower.

Here are some areas in which computer excel over humans, especially in a “testing” capacity (I will explain the use of quote marks later),

• they are much faster
• they can do the same task over and over again with no variance
• always do exactly what they're told

On the flip side, this is what an experienced human being can do which computers can't,

• does not need a script to start using the software
• uses software in the same way as an end user (if a human is going to use your end-product, you need a human opinion on it during testing)
• can investigate as they test
• can notice things in software even if they're not on the checklist

To make efficient use of automation (and thus get a return of investment on the time you spend automating), you need to be addressing a pain point in your testing, and you need to be doing something in your automation that computers can do well (from the first list) rather than something that humans do well. It also needs to be doing something that you're likely to do again and again - so once scripted, it'll save you time every time it's run.

If you're Agile, and 3 days of every sprint is taken with your testers running repetitious regression tests on a mathematical function, this is the kind of pain point you can and should automate to free up some of those 3 days of testing effort.

Know what you should be automating

When test automation was first introduced in the 1990s there was a belief that many test departments should have suites of 100% automation. Experiences of the last decade have really challenged that idea.

Test automation has a place, but it's not the alpha and omega of testing. In fact many like Michael Bolton believe automation tools should be called automated checkers over automated testers (hence the quotation marks before).

The reason for this is an automated script can only check for what it's scripted to check for. It will never tell you if a screen flashes pink and yellow unless you tell it to check for that. It will never notice the kinds of things that a human tester will go “well is it supposed to be like that?” where something is not necessarily against a requirement, but not quite right either.

The cyborg tester

I've heard the concept of the cyborg tester before, and this is what started to come out from people's experience with automation. I believe I've heard it from people like James Bach and Ben Simo on Twitter – the idea is that testing isn't about doing testing all by humans, and not by doing testing all by machines.

The cyborg tester is a fusion of both man and machine, using both blended together to produce superior testing.

Automated checks are fast, repeatable, and if done right, anyone can push “play”. But they miss a lot of defects a tester would find. They are best used essentially for unit testing between building a piece of code and giving it to a human tester.

We've all had scenarios where developers have delivered new builds daily – when asked if it passed testing, you are greeted with “well it built” (which does not mean it passed any kind of test). The test team start to use it, and there are major issues, with elementary functionality failing. That means anywhere from a half day to a full day of testing is lost because we have a bad build, and no capacity in some systems to rollback to a previously working build.

How much better then to include those kind of smoke checks as part of the build process, and if the software doesn't pass those checks, it's not deployed? Such a policy follows the “test early” philosophy, and means manual testers are protected from bad builds which are so fundamentally flawed it would force them to down tools until addressed. [A working old build allows more investigation than a new, broken one]

Such a system is one of synergy, allowing testers to continue investigating on a previously stable build until a useful new build with basic core functionality can be delivered.

Automation TLC

As alluded to by Katrina, and in line with comments I've had from Janet Gregory, the automation you are doing needs to be clear and visible to the whole project. Everyone should be encouraged to look at what you are doing, review, and give feedback, especially as to whether or not it addresses business and technical needs enough.

How can you be sure your automation really addresses the core business values of your end-product? You need that feedback to target the important stuff, and cut away anything which doesn't add value (otherwise you waste time running it, you waste money maintaining it).

But more than that, many places will automate phase one of a project and like Katrinas Project B and C, will say, “we're done”. Automation isn't a “we're done” thing. Automation in an ongoing commitment.

Every time you make changes to your code base, you need to have someone looking at your automation scripts and going “does anything here need to change”. That's how you keep your automation working and relevant. If you develop for a year, and only then start to use the scripts again, you might have a nasty shock (much like Project C) where nothing seems to work any more. You might even be tempted to bin it all and start again. At this point, the automation which was there to remove your pain in testing, actually becomes your point of pain!

But more than just making sure you have resources to maintain scripts, you have to ensure your scripts are maintainable. In software code, good practices are to have commenting within code to say what each pieces intent is, peer reviews of code and to even have coding standards of things you should try to avoid (forever loops anyone?). Being an ex-developer myself, these are things I encourage in any test automation project. Going around the WeTest workshop, it became clear I was not alone.

When can we get time to automate?

This was the final question of the night I was involved in (alas I had to leave for a train at this point).

But the comment would be one many would be familiar with, “we're going to flat out with our manual testing, we're having trouble creating our base of automation scripts”.

It's tempting to think it's possible to go flat out on a project and also be able to do process improvement as you go along. Sure you can make small improvements. But to achieve significant benefits you need to allocate effort, because you need to concentrate on that task, not run it in spare time (for more on this subject read this article).

If you are finding you have too much testing to do, and find it harder and harder to achieve deadlines you need to go back to our first point, look for the areas of pain that are taking time, and see if automation will help. You might have to say to your Project Manager “look we need to pause for a bit to get some breathing room here” and perhaps extend timelines or investigate other options – it's possible development need to pause to do some cleanup themselves. But you don't need forever, you just need enough automation to ease your painpoints, and then enough resource to keep it maintained when needed.

Overview

A fantastic first meeting, and looking forward to future – thanks to all those who turned up and made this such a memorable workshop! I certainly came away with a lot to think about, and have enjoyed revisiting it here ...

The following photos are taken from the WeTest site ...

No escaping me in October ...

There's no escaping me this October. After a few busy months, I have articles in no less than three testing magazines …

That Barton Fink Feeling

Is an article in Tea-Time With Testers about a feeling we'll all come across at some time. How do you deal with accusations you're not being a “proper tester” because “testers on my last project ...”.

The Ethics Of My Father

Back in KWST2 we were talking about ethics, and I gave an experience report of growing up as a teenager and seeing one of the difficult ethical choices my father had to make, and how that affected me.

Championing The End-User Using Personas

A look at how at a previous company we learned to scale naval products for the fishing industry by developing the persona of our end-user.

This is a brand new magazine NZ Tester, which Geoff Horne is putting together, and even if you're not from New Zealand I urge you to give this magazine your support.

Enjoy!

The Tester's Video Library ...

My son is a bit of a YouTube addict, and to him and his generation, this is something they turn to when they need to learn about an area and learn fast. This to me can seem quite lazy and almost a cheats way of learning, but as his depth of knowledge of Second World War history has shown, it can be very effective.

Yes - time to get with the 21st Century (so my son tells me).  Learning is no longer just about going to the library and "reading a book".  Today we have blogs, and through YouTube we have on-tap tuition when we need it.

Over the last couple of weeks I have been following his lead and trawling the internet to watch talks about testing – and I too have found it very educating. Some videos have touched upon pain points I'm all-too-aware of, others have really challenged me to raise the bar in my testing and work. And so I'd like to share them here

I'd like to thank my fellow Twitter testing wingman Kenneth Aarseth‏ for providing some of these links.  If you know any must-watch videos, please add them to the comments!

Janet Gregory's Learning for AgileTesters

I first watched this in 2010 when it was recommended on the Yammer feed of the test consultancy I worked for. It had me hooked, and it's probably fair to say it inspired this blog. The idea (which I've previously discussed here) that any learning helps you to diversify and grow as a tester. Marlena Compton provided me her blog entry on the subject which she calls interdisciplinary studies, which is a good read.

Janet Gregory's Testing Traps

That woman again. This video was filled with good stuff, of course a lot of it I knew from reading Janet and Lisa's book on Agile Testing. But the one thing which really stuck and challenged me in this talk was around “am I doing enough to make my testing visible?” to the whole project. Challenging. Am I making my testing visible? Yes I know I am. Are there ways to make it better visible to the people who matter? Erm … I think I need to reflect on that. There are always ways to do things better aren't there?

Uncle Bob Martin's Requiem for C

I loved this video, although many testers may not learn huge amounts from it compared to the others, it's still worth experiencing. Uncle Bob is an amazing storyteller, and takes us through the development of software and computers during his career. As I am an ex-developer in C I was rivited.

The main thing you will take home from this is how computers are changing constantly – they have evolved during Bob's lifetime, we'll see the same. Our skills can rapidly become outdated, so continuous learning throughout our career is something we need to embrace.

Michael Boltons Thing's could get worse: Ideas about regression testing

Michael challenging us on,

• What do we mean by regression testing.
• Can we benefit from automated checking tools?
• Why human testing can investigate and discover issues checking tools will miss.

What I really came away with from this was the discussion about automated checking vs testing.  That using automation we can only check for the problems we could imagine when we scripted. There are all forms of issues which can quite easily slip through the net because you didn't plan your script to adapt and cover them (because you couldn't imagine them happening). This is where a sapient human tester can adapt and investigate where an automated script either grinds to a halt or even worse, carries on an the issue goes unnoticed.

Johanna Rothman's Many years of managing testing

I saved the best until last! Simply because it touched on so many points which are relevant to me this year. I recognised in this talk things I'm doing at the moment, but need to chase up from this talk and get better at. 

Key points in Johanna's talk were,

• Testing is about gathering information on the product not ensuring quality
• Manage your communications about testing via a testing dashboard
• Learn to not spread yourself thin and to say “no” when needed
• As a test manager, organise your testing portfolio. Make it transparent where you have no resources.
• Don't move around testers between projects, you'll lose their expertise in areas, which is what makes good testers.

Johanna Rothman: Yes or No? What to do when faced with the impossible

Some more Johanna, talking this time about managing your time and requests to avoid falling into the pitfall of multi-tasking.  Managing expectations - knowing when to say yes and when to say no, how to get people understanding your pressures and priorities.

Johanna Rothman: Lessons learned in project management

This is a must-watch if only for the piece at about 10 minutes about "all out people were good ... except testers, they let everyone down".