Farming Experience

Yesterday I talked a little bit about pure obstinacy, how we have to do our own thing in the wake of a lot of good advice.  It's a natural phenomenon of the world.  It's also a huge part of the difficulty of life.

These two blog posts were inspired from Twitter, where I get a huge source of inspiration from,

Ah yes, that feeling when you don't feel listened to as part of a team.  Especially where you've been brought in for your expertise, but sub-sequentially sidelined.  Been there, and look ...

I'm going to put this unpleasant experience under the microscope, and I did a bit of the ground work yesterday.

Let's revisit the "alcohol abuse" scenario again.  Sorry, but we have to!

My parents gave me advice, and it's good advice.  And it's based on their experience.  But I ignore it because I want to get drunk.  The fundamental problem here is their experience is their experience ... it's not mine (until the morning after).  Until that experience is my experience, I'll only pay lip service to it.  It's human nature.

I find the older I get, the more I empathise with my parents.  It's a very weird experience, but probably an important part of growing up (yeah, I'm in my 40s and talking about growing up).

I'm often brought into projects to "bring in your extensive experience" in IT.  I'll roll up my sleeves and ... no, you're not going to do THAT are you?

Often as the subject matter expert you have an uphill battle.  People can want you there, but not really do what you advise, if it's contrary to what they'd like to do.  Like a lot of people, I can sometimes get very annoyed and frustrated about this.

Here are some steps I take to try and turn things around

Look first to yourself

I noticed something I do from looking at my post from yesterday.  I'll get annoyed about something, some way that people have behaved.  I might be ready to pull out my soap box ... but wait.

Something I try and do first when I find fault in others is to first find fault in myself.  I'm annoyed at people who voted for leave in Brexit, I'm annoyed for people who voted for Trump.  I'm angry that people can be so self-focused, lack empathy and just roll their eyes when any kind of "expert" gives evidence against the politician who is offering them free lollies, so they just yawn and go ...

Many bloggers would just launch into an attack, it's easy to do.

Though as I covered yesterday, although my parents weren't the most supportive people when I had a hangover (my mum would sing opera very loud if I had a bad head), they brought me up that before I find fault in others, I should try and find fault in myself first.

The older I've got, the more important this is.  So at the beginning of my talk yesterday before I talked about people who ignored all the evidence and voted Trump, I talked about the times I just plain refused to listen to my parents because "I have to do my own thing".

If I have an experience where I've been a jerk in a similar way to the way to somebody who's currently infuriating me, it raises a very important point.  Maybe their behaviour isn't so much spiteful as some kind of human behaviour.

At this point, I have to take a quite critical look at myself, and it's really an uncomfortable thing to do.  The question is, "do I still behave like this, and it not, what changed me?".  I really hope the answer to that question is always a "no, you grew up".  But not always.

If the answer is a "no", it gives a very important starting point.  Something changed me, what was it?  That experience is my ally, because what worked for me might be an inroads for working for change with a group or other person.

Exposing to experience

Let's look to how my parents looked after me.  They would leave me to feel dreadful for having a hangover, and would gloat over how bad I felt.  But there was an incident where I had a combination of alcohol and cold remedy which was nearly fatal, and they got medical help for me.  So you let people feel rough, but you avoid them getting into danger.

As said above, you want to expose someone to some of your experience, for things to get a bit rough so they feel the heat a little.  That is how your experience becomes their experience, but it's your role to save them from the worst (but note, not from everything).

What you need is the equivalent of what you get in sitcomland where the father of the house catches their 10 year old trying smoking, so they get them to try some cigars with whiskey, to create an inevitable vomiting episode which will "teach them a lesson".  Of course this can sometimes go horribly wrong ...

One thing I'm a huge fan of is workshops and activities.  Within such activities things can go wrong, but it's a safe environment.  That's why I use activity for my introduction to testing workshop at Summer Of Tech, and why I'm working more on workshops over presentations this year, especially at TestBash Brighton (come see my strategy workshop).

It's also why I feel the Software Testing World Cup is a must for people to have a go at, we really enjoyed competing, and why I'm trying to get some of my company at Richard Bradshaw's LEGO Automation Workshop.

It's also why I try and work so hard at the water cooler.  I try and use people's experience to give me an inroad, to socialise my message, put it into terms they understand.  You can tell I went to one of those Churches which encouraged us to evangelise about Christianity whenever we could ... well I do that now about testing.

Someone excited about a space probe?  Let's talk about how NASA tests (not so much ESA right now).  Actually, I lie, something like how the ESA Mars Schiaparelli lander is great to talk testing.

But currently in agile, my job couldn't be simpler.  As I talked about previously, it's not my job to make every sprint a success, it's my job to do my best.  If the rest of the team want to do something that I think is risky, but won't listen to me, then sometimes the way to find this out is to go down this road, and revisit at the retrospective.  With luck there'll be something, and so they'll get to feel the heat just a little, but enough to go "hey, let's not do that again".  Now they should have a bit more experience, they played with matches, burnt their fingers but you're on hand to make sure the whole house doesn't come down.  Also, "hey Mike, why were you standing next to the fire extinguisher?"

What I've found to work is when it does happen, just give them a gentle nudge.  Don't do the whole "oh I told you, and you didn't listen".  I wasn't too keen to hear that from my parents when I had a hangover, although it did reinforce things.

Say "this is why we need to X", and leave it at that.  This is how you build your reputation in a group, and as people come to trust you more, they'll listen to you more.  I work with two amazing testers in my company whose behaviour seems impossible.  They are the opposite of me, being quite quiet and reserved - but by simply making their case, nudging in retrospectives, and not making a big deal of lessons learned, they've built up a great reputation and a lot of trust, to the point where they've quietly pushed a mandate for testing, but using a great deal of patience.

I've learned a lot from these two.  But speaking up, giving space, reinforcing points without using blame can change things.  Often frustratingly slowly, but it's real change.



Edit - December 2017

During the year, this has been a major theme of a lot of what I've done.  Running internal workshops and indeed at TestBash Brighton and Agile Test Days.  It's not the only way to learn, but it's something I align very strongly, and it has a name called "kinesthetic learning" or "learning by doing".



Now Playing: "The Others", TV Rock vs Dukes of Windsor

Post-Election Hangovers

Let me start with a tale which will probably be uncomfortably familiar.  Occasionally at home, I'd be up in the morning feeling dreadful and the following interaction would happen with my parents,

ME:  I feel so ill, my head.  I think I'm going to be sick!!! 

MOTHER:  You drank too much again didn't you?  Haven't I told not to have so much, you'll get a hangover, but you didn't listen did you?

DAD [Getting in on the act]: No, he didn't 

Never-the-less, they'd still give me a glass of water and paracetamol if I needed it.

This was in those awkward early twenties.  I loved alcohol.  I was such an introvert who struggled with social situations, like many of my peers.  I'd go out together, we'd start drinking, and we'd feel more liberated.  Surely the more we'd drink, the happier we'd feel right?

Morning the next day brought that delusion crashing down.

And she was right, she did warn me.  In fact I was often reminded of the scene from The Hitchhiker's Guide To The Galaxy,

“You know," said Arthur, "it's at times like this, when I'm trapped in a Vogon airlock with a man from Betelgeuse, and about to die of asphyxiation in deep space that I really wish I'd listened to what my mother told me when I was young."

"Why, what did she tell you?" 

"I don't know, I didn't listen.”

It's true though, and it sparked a conversation with my friend Violet back in 2007.  My parents tried to bring me up well, and they gave me a lot of advice.  And it was really good advice, and well meant advice.  Advice that was there to stop be hurting myself.  I just didn't listen.  Why oh why didn't I listen?

And I dare to find any child who hasn't done this to some degree.  Part of growing up is initially accepting the framework your parents give you, and then at some point when you're adolescent, you feel the need to challenge that framework to work out things for yourself.

So no amount of advice about hangovers was going to equal the actual dreadful, messy experience of actually having too much to drink one night and experiencing it for myself.  With your head stuck down the toilet if I'd really overdone it.

And even then, you'd think a lightbulb would go off, and I'd realise my parents are right, and that I would rationalise my behaviour over drink.  But still it doesn't work like that - some of us are thick headed and stubborn!  You still want to drink, and to drink a lot, but you remember how bad the hangover was so maybe,

• I won't drink quite as much
• Gav told me that having a kebab at the end of the night soaks up all the alcohol and makes you better
• Wayne told me you need to have a bit of hair of the dog in the morning

All of these are half-hearted measures which cling to a desire of "I really want to get blotto".  None of these really work, and it was only when I was in my late 20s after many, many hangovers that I accepted the truth of my parents advice, and enjoyed my alcohol, just made sure I didn't drink as much.

Indeed, if you meet me at conference, you'll see I count my drinks, something Lisa Crispin's husband Bob observed at Agile Testing Days at Potsdam.


[By the way if "drinking too much" doesn't trigger for you - maybe it was "don't eat so many sweets you'll be sick", "don't go out without a coat you'll catch a cold", "don't eat that it's a week out of date", "I know that car looks like a bargain, but it's a brand renown for maintenance problems".

If this still doesn't feel like something familiar to you, let me remind you adolescents "test boundaries", so do testers.  I'd never hire someone I'd not felt had committed mischief at some point of their life!]

Now you might be laughing at my young adult stubbornness and plain stupidity in action there.  Then I remind you where we're heading thanks to 2016.

Politically, 2016 had a lot of disturbing trends.  Most of all was the way in elections like Brexit and the American election of Trump, we had a lot of experts warning us of what would happen if we voted for either Britain to exit the EU, or for Trump to become the next President.

There was a lot of logic and advice laid out.  These things had historical parallels, "look and see where similar actions led us to in the past.  Look, look at the evidence!"

The problem is much like with my parents sound advice on alcohol we don't listen.  We dismiss this advice with a "bah - experts".  The logic usually goes "aren't these so called experts disagreeing all the time, yesterday they told me sugar was bad for me, today they tell me artificial sweetener is bad for me".  And that's a bit true, we sometimes have scientific studies jumping the gun, and we're overloaded with changing facts as science learns more, and we change our position.

So there's been an increasing tendency to not listen to experts, to dismiss their advice, and make a purely emotional decision.  We really like emotionally what either the people behind Brexit or Trump are offering.

They promised that if you vote for them, you will see the following happen,

• There'll be more money for the NHS (so much so you put this as a promise on the tour bus)
• No more immigrants - will mean more money for you!  You'll be able to afford your own house and not have to wait for hospital appointments because we're overloaded!
• We'll make America great again
• More jobs for all
• Less tax
• The status quo is corrupt, you can trust me

You can't help but see the appeal of such statements to ordinary people.  The problem is the people who doing all the promising don't have a very good track record.  I'd say they have a habit of lying, to which they might retort "but what is truth anyway".  So let's not get tangled up in that!

But much easier to prove, they're constantly contrary.  They say one thing, then soon afterwards will claim "I said no such thing".  Which makes it very hard to make them accountable.

But as you know from previous postings on denial ...  we start from what we want, what we desire.  If someone keeps saying that they'll lower tax, and lower taxes is what you really want, you'll be attracted to them.  If that desire is strong enough, you'll ignore any evidence to the contrary with a "yes but...".

As you know, both these received enough votes in 2016 where it counted to become reality.  All the advice, facts, logic, warnings couldn't sway the voting public.

Pretty much as warned, once Brexit voted for leave, the UK economy has taken a battering, with at least another 2 years ahead and an uncertain future.  Many voters have expressed regret in voting for leave from this realisation (but it's all rather too late now).

Likewise, President Trump's tenure begins this week.  Under the claim that "America has suffered enough" over the Affordable Care Act, the first measure will be to scrap the scheme.  As a reminder, this is the scheme which provides treatment to people who'd otherwise be told "you have a life threatening condition which is completely treatable ... you just don't have cover".  If you're doing that saying you're going to "relieve suffering", you have a fucked up sense of what suffering is (and yes, that's the first time I've used that swear word on this blog, and I stand by it's appropriateness ... my parents read this blog and will be in contact about my language).

Like the post Brexit "I voted leave, but now..." regret, there has been a little bit of that from Trump voters already.  You'll no doubt have come across several stories such as this of someone who is absolutely pleased that "Obamacare" is being repealed, then horrified to learn that "Obamacare" is a term Republicans have used to describe the Affordable Care Act, which this particular voter depends on.  I'll admit it's hard to know for sure how wide-spread such thinking really is, but it seems to be there floating about on social media and the news.

There's suspicion that anything that replaces it will look a little bit like this,

Ever understanding of the common man, Trump has even stated an idea where people pay for healthcare from savings.  Only a billionaire could come up with that one.


In the end that's a flaw of democracy, sometimes we'll make crazy decisions as a group, but we have to ride through them as best we can.  Sometimes we have to fight them all the way, hoping somewhere along the way people will wake up and have their hangover realisation that maybe that advice they were given was really good, and they need to not follow the same course of action in another 4 years.

I hope that's possible, that people won't just stick with Trump for the full 8 years because they'll have buyers regret, and be sure everything has to get better soon.  I hope people wake up and realise all the change they were promising hasn't come - but it involves being able to challenge a President that if told "why has unemployment risen by 2% in the last two months" feels he can shrug it off with "that's a lie, why there's plenty of jobs out there, just today I heard we created another 100, next question, you with the white hood!".

As a reporter from Russia explained about Trump's role model Putin, holding the leader to account might be harder than expected.

But I hope it's possible.  I'm hope people will wake up and realise how the politics of the far right only services the greed of a small few, and they're not on the invite list.  That people will yearn for real change, which to my mind only more liberal politics can deliver.

But in the next four years it will take a lot to wake people up to the point where they can change anything.  That means a lot of people who are going to die of treatable illness to be told "now had you been diagnosed when we still had ACA".  It means four years of damage to the planet's ecology as people embrace "global warming's just a myth ... but isn't it odd how smog's returned?" (which will also look to include a ban on any research).  It means four years where you're afraid for your safety if you're not white, if you're not male, if you don't identify as straight CIS.  It means potentially four years of superpowers Russian and America working like a vice to squeeze, terrify and bully any country that's between them (and you wonder why Trump doesn't like the European Union).

It's a terrifying four years ahead.  But like the far right who have been campaigning and undermining Obama since he took office, if this is important to you, you need to make a stand lest the whole world follows this path of crazy.  Already around Europe there's been an increase in support for "alt right" groups.  It very much reeks of the rise of Fascism in the 30s, extremist groups looked to Mussolini's Italy, and attempted to mimic it.  We even had a Brexit Leave supporter gunning down a campaigner for Remain, this is how extreme the politics involved are on the alt right side, where the rhetoric of hate inevitably ends.

If the way the world is going bothers you, be prepared to have uncomfortable conversations about politics with your friends.  If you don't, someone else will.  Also, give serious thought to joining a political party, even in a small way.  At the start of the year, I joined one in New Zealand that although I don't agree with everything, I agree with most.  Find that party, and make a change!


Also check out this great article on fake news by the BBC.  I know, it's up to you whether you believe it or not.


Now Playing - "Killing In The Name Of", Rage Against The Machine

Memory Lane

I've previously written a lot about memory - but today I'd like to explore a particularly important one from long ago.

Being middle aged, and having moved around a lot, I have a many memories from different places, some going back to my early years.  Mostly they're well behaved and "remain in a state of slumber" ... and sometimes one will need a little jog before it goes back to sleep.

This was the Twitter picture which triggered one of my bittersweet ones,

My wife Collette and I have been together for over 20 years now.  That's so long that it's easy to take for granted.  But a memory of Liverpool is enough to stir those of our early days together.

I always found dating kind of awkward until I met Collette - with her there was never any need for pretense.  In came this woman with an incredible energy and personality to both match and at times complement my own.  We were both incredibly passionate people, although that passion didn't always align, and could cause some rough waters at times.

It was an important relationship - she always challenged me to be more than I could, an odd mix of loving me and pushing me.  But there was always in those early days a bittersweet ritual we had to live out.

Collette's parents were quite traditional, and we never wanted to upset or shock them.  So although she often stayed with me, she could never stay the night, having always to be back home come the morning.

Thus in the early hours, typically around 4am, one of us would wake in the others arms, wrapped together as we were in order to share a single bed.  Driving her home through the dead of night in Liverpool was always such an eerie experience - a city so vibrant during the day, so empty and desolate in those early hours.  The city slept whilst two lovers prepared to farewell each other.

We rode mainly in silence, occasionally Collette ghoulishly pointing out a street corner where a gangland slaying had occurred.  It was all to take our mind off the goodbyes to come.  Alone in the early morning, it felt we wrote our love on the streets, as each red light in our path was an opportunity to hold her soft, warm hand in the bracing morning, or to steal another kiss while we could.

All too soon she would be home, and I'd face the lonely journey home.  All the time thinking what a wonderful thing it would be to be able to wake up in the morning together.  It was like living in a fairy tale - I had someone I loved and who loved me very much, but I always had to give them up before the dawn ...




These days, I like Sundays the most.  Sundays there are typically no alarm clocks, no pressing reason to get out of bed.  Most Sundays you'll find me awake, but lying in bed, next to Collette, and remembering how to my 25 year old self, waking in the morning next to her was all he felt he needed to make him happy.

I think one of the saddest thing about being human is we yearn so much for things, but when we get them, we so rarely get to enjoy them or appreciate them.  All too often one sense of want is replaced with another - we're forever hungry and insatiable for some need.

Sometimes the right memory will help you remember how much what you have now is all you ever wanted.  Find yours, and never take it for granted.

Thinking about test strategy ...

As background to this - this year I'm trying to collect a lot of ideas on test strategy.  It's my hope I might have enough material for a workshop on it next year.

Right now some of these ideas are getting structured, but at an early stage - so do contact me below or on Twitter with your ideas and comments.  I'd really like to push further on this!

Last week I was running an evening bootcamp on testing for Summer Of Tech.  It's an awesome opportunity not just to touch base with New Zealand's upcoming tech people, but also to champion a little bit about the fun and mindset of testing.

During the presentation though - I gave my current definition for testing ...

I've been using this slide for about a year - but realised how within that statement is something I'm thinking about more and more recentl.  Something that's been under our noses for a lot of the time.  Maybe we're so busy, blinded and programmed, because we've never really taken the time to explicitly think about it.

Control-Expect-Observe

Let's look at the standard test script format we've all had to use at one form or another ...

In a test script when we use action vs expectation we're really talking about "something I control" and "something I observe".

It's fair then to sum up a test scenario as using something we control to produce something we observe which can then be compared with something we expect.

All testing is a form of play between these factors - even exploratory testing which we talked about previously here.  We use a heuristic to vary something we control, we use an oracle to set our expectations, and we observe - raising defects if there's a variance.

Exploding the idea of strategy

At it's heart then, a test scenario seems a very simple beast - we control something, something happens ... is it what we expect?

This is probably where testing gets it's reputation for being easy - a single test scenario presented like that looks simple.  Even if we overlook how observation can be a difficult to master.*

The problem is that testing is not the execution of a single scenario.  The skill of testing is coming up with a series of scenarios which cover a system.  Again, people can come up with a scenario or two, but that doesn't make them skilled testers.

To me, being able to come up with a selection of scenarios to cover a product is not easy - and there are no guarantees.  There is such a vast variety of ways a product can go wrong, that it can be overwhelming.  Consequently we often will see members of the testing community talking about how huge testing can become.

Unfortunately that can become problematic.  I'm currently reading Daniel Kahneman's Thinking Fast And Slow, and we've come to the idea of substitution.

Simply put, when we're asked a very complex question, if it leaves us perplexed we will sometimes answer another related question for which we do have an answer.  In his book, when they asked students in one survey "are you happy?" followed by "how many dates have you been on this month?".  "Are you happy" is a complex question.  Put this way around, they found no correlation.

They then asked another group of students the same questions, but the other way around - "how many dates have you been on this month?" followed by "are you happy?".  This way around they found very strong correlation.  The reason being that when the complex question about happiness came up, they substituted a question they had just answered "how good is my dating life?".

Now remember how I mentioned how testers talk about "the possibilities of what you can test" being huge?  That means when a new system is put in front of us and we're asked "what will you test?", that it's a big complex question.

So no surprises, we tend to perform a substitution of our own, and hence we tend to answer the question initially as if it was one of the following questions ...

• "What do you control?"
• "How have you tested other systems?"
• "Where have you found problems before?"

Now none of those answers is a bad first answer.  The problem is when we're falling into the trap of using one of those without realising.  They are each a very good starting point, but if you don't push further, you'll end up doing shallow testing, and potentially missing something important.

"What do you control?"

I'm going to revisit material from a previous article on Back To Basics, where we looked at registration.

Let's take a look at Facebook registration for a change ...

It's easy to take a look at this and come up with a list of the following which are things we can vary and control ...

• First name field
• Surname field
• Mobile number field
• Re-enter mobile number
• New password
• Birthday drop down lists
• Female / Male radio buttons
• Create an account
• Various hyperlinks

When we look at these items our first approach is to create a list of test scenarios which use different data and permutations.  And that's as I've said is a superb start.  Where it's not great is if once you've exhausted ideas for this to consider you've tested everything.

Because you haven't.

What you've just done is a very competent job of functional testing.  You've chosen items that you know are simplest to control, and you've created scenarios.

What you've omitted are items for which you don't know how to control.  I think an important part of test strategy is being able to recognise "I ought to be able to vary X, but I don't know how" - then asking the question "how can I control X?".  Unless there's explicit instructions, we'll often have a blind spot for the "everything else" which defines a product which is often thrown into the "non-functional testing" basket.

What are some examples of tricky things you might want to control?

• Platform / Browser you test your system on.  Different browsers react differently - installing other browsers is easy.  And you can set up different virtual machines to simulate other platforms.  Maybe you need an Apple Mac (good luck prying it from your architect's cold, dead fingers) - and maybe some devices such as the latest iPhone or tablet?
• Screen resolution.  When was the last time you adjusted your screen resolution?
• Server load.  Does the system work well with multiple users on?  You can use a tool like JMeter to simulate a lot of users on your system - simply it's a tool that allows you to control a lot of concurrent calls to your system and measure response time.
• Hardware down.  If you've got a load balanced system, do you have a scenario with part of your system down?  It's not an obvious thing to control, but you should ask how you can get someone to do that.

Of course in talking about the above I've provided potential solutions - which was naughty of me.  You need in strategy to be brave enough to say "I need to be able to control X", even when you're not sure how.  Ask your team, and around anyway.  Maybe there's a tool out there, or maybe the team needs to build it into the system as part of testability.  But don't be too quick to say "I can't control X, so I won't bother".

At worst, if you can't find a way to control, then put it as an assumption "we weren't able to test around different X".

"How have you tested on other systems?"

Again a super method to get started, we use our experience on a previous system, and copy and paste it over.  This is great apart from,

1. Your testing will only ever be as good as your last project (what if instead of good testing, you were instead just very lucky?).
2. If the current system is almost identical to the last one, that's great.  But more than likely there will be differences - and do you need to account for them?

A good example, if you didn't need mobile device testing on your previous project, you're in danger of thinking that can apply to this project - are you sure?

"Where have you found problems before?"

Once again - this is a helpful method, with a hideous blind spot.  If you use only test areas where you imagine there could be problems, what if there are problems in the system you're just not imagining?  How can you think bigger in your testing approach?

If you only go looking for old bugs, the problem is you might not be taking a different path to find new ones!


A useful guide for strategy

Is there a particular approach you tend to favour?  In the early days, I tended to get stuck on "where have you found problems before".  I now tend to obsess a little on "how have you tested on other systems".

When you're looking at a new piece of work, capture all of your initial ideas and try not to think too much about it.  Once this has dried up, think how you've approach this - has it been as if you were answering,

• "What do you control?"
• "How have you tested other systems?"
• "Where have you found problems before?"

Think about the blind spots associated, and dig deeper.  Try answering the other two substitute questions, and see if it forces you to go deeper.  And of course, try out the Oblique Testing cards to shake things up a bit!






Regarding Observation

* If you look at my book How To Test, there is a chapter on observation.  However dispite over 400 readers, no-one has yet Tweeted me to say "where is Chapter 7?", this is one of the deliberate mistakes in the book -

Dan Billing has accused me of overplaying the "smart arse" card on this one ...

Based on what?

My son is currently adjusting to the change of education of moving from school to university.  One of the more interesting changes of how his chosen subject (history) is talked about can really be summed up in the question "based on what?".

Up until now, most of his textbooks have included an unchallenged narrative of events and decisions in history.  It would go something like "King Ichabod marched his armies to the Crane Valley, where he fought an unsuccessful battle with rebels.  He decided to move back to a resupply point in Sleepy Hollow to regroup for a second assault".

At University, much more is expected of him, and of his materials to prove such a set of claims.  For instance,

• How do we know he fought his battle at the Crane Valley?
• How do we know he lost?
• How do we know why he chose to move back to Sleepy Hollow?

These claims are based on what?  Basing them on what's been written in other books isn't a terribly strong to build your case on, you can only be as strong as the case they make.

I didn't get to study history quite as far as my son, but when I studied at advanced level, we were given this puzzle "does history really exist?".  The unnerving fact is that really it doesn't.  We can only make educated guesses at what happened X many years ago based on the evidence we have.

• We can have confidence a battle took place at Crane Valley based on contemporary accounts (of the time).
• If we go digging around and find grave sites and abandoned weapons - all of which match the period, that gives us more confidence (even so you tend to occasionally find some battlefields further away than they were believed to be).
• Who won is an interesting one - we generally only read about wars from the accounts of the final winners.  And it's been known for some people to rewrite lost battles in their favour.
• We can know something of what someone was thinking and feeling from letters they sent, their personal diaries, and speeches they gave (if recorded).  Even so, historic figures are well aware of the power of spin and self-delusion.  A defeated general on the run might well put the blame more on his troops than on himself.

Of course talking to my son, what interests me is this is key to being a good tester - to challenge the narration we're being given, by asking for proof and evidence where you can find it.  Whether for history, critical thinking or testing it's a mindset well worth exploring.

TED talks worth watching

Generally this blog is about original material.  However, once in a while I will find something so extraordinary that I just want to link to it, and "keep it" on the blog - so I and other readers can come back to it.

I don't do it often, but when I do it's because it either so moved me, or I have an absolute need to say "+1" to it.  Here I look at some recent TED talks which have completely blown me away for one reason or another ...

How the worst moments in our lives make us who we are

Here Andrew Solomon talks about people who have had to deal with adversity.  In particular how those who transcend show an ability to forge meaning in their trials ...

https://www.ted.com/talks/andrew_solomon_how_the_worst_moments_in_our_lives_make_us_who_we_are#t-1195138

[If you only have time for one talk, this is the one I'd recommend]

How frustration can make us more creative

A recommendation from my parents no less!  How well they know their son.

Contrary to what me might seem common wisdom - it's often when we're just a little uncomfortable.  When thinks are a little disorganised and maybe even chaotic.  These are the time we're actually much more switched on and creative ...

https://www.ted.com/talks/tim_harford_how_messy_problems_can_inspire_creativity

Really "mixing it up a bit" and Brian Eno's Oblique Strategies sound useful for many an agile team.

Trial, error and the God complex

Economist Tim Harford looks at complexity in the world, and the "God complex" trap - where we think we understand things, and try and simplify our approach.  He argues that we evolve our approach best by adding a bit of randomness and taking a trial and error approach.

This really resonated with me.  Recently I took part in James Bach's  challenge on integration testing.  He set a problem, and I gave how I'd approach testing it.  When he asked me why I'd do some of it - what problems I'd expect to find, I was tempted to change my scope, and do less.  But he reminded me you don't always have to justify everything - if a test is simple, and is there out of curiosity, don't feel the need to justify every test you do.

Likewise in agile, we'll test functionality around a story, because we know that's where the most change has been.  But sometimes we want to test manually in areas we know haven't changed.  The God complex tells us that area hasn't been changed (that we know of), so it's a waste of time testing.  However, once in a while, we find something has got through - and our God isn't as infallible as they led us to believe.  Trial and error will find things that God complex cannot anticipate, and it's an important tool for us a testers!

https://www.ted.com/talks/tim_harford

[This talk echoes some themes I'm encountering reading Thinking Fast And Slow]

Denial 104: "But we spent a lot of money on the license ... and customising it!"

Previously we looked at the psychological effect of denial, and what drives it and makes it tick.  Today we look at an example of denial at work within testing - especially other's perspectives of it.  Ladies and gentlemen, I give you exhibit C ...

"But we spent a lot of money on the tool license ... as well as customising it!"

Again here there is a base assumption that spending a lot of money equals "this has to be really suitable for us".  When we looked at peer pressure previously, I tackled some of this - especially the myth that a test management tool means "reporting for free" ... I of course automatically get nervous when people offer me anything for free, mainly because of this guy ...

Chitty Chitty Bang Bang's child catcher - he lured kids with promised of lollipops "for free".  I'm sure he now makes a living trapping unwary IT projects in enterprise license agreements.

As we discussed, test tools generally trap you into working in a particular way.  If it matches how you want to work, that's great.  However if it doesn't align to how you need to operate on your project, then you're stuck with such a difficult and clunky methodology that will constrain the way you work, and so has an effect on your team which is difficult to measure.  Except for the fact that your team hate using the system.

But wait ... there's something worse than an "out of the box" product which partly addresses your needs.  And that's a product that's been "tailored".

Naturally, I have a ridiculous (yet true) example from my past.  Many years ago we had to use Microsoft Team Foundation Server (TFS), and some industrious manager had decided that to get the most use out of the reporting from the system, they would replace the out-of-the-box defect lifecycle below with something that would be more granular, and would allow for better in-detail status reporting.

So out went the simplicity of ...

Difficulty setting: Easy

In came the following hierarchy of states before a defect could be closed ...

• Proposed
• Defect confirmed
• Assigned to developer team leader
• Assigned to developer
• Developer working on
• Code fixed
• Ready for unit testing
• Assigned to unit tester
• Completed unit testing
• Ready for system testing
• Assigned to system tester
• Completed system testing
• Ready for pre-prod testing
• Assigned to pre-prod tester
• Completed pre-prod testing
• Ready for deployment
• Deployed
• Checked in deployment
• Verified
• Closed

Each state above was mandatory by the way (no cheating and skipping states you naughty tester).  It wasn't too bad for tracking a production incident, but for a defect in a project which wasn't in production, it had way too many states - most of which made no sense for what you were doing.  Just to make it worse, every time you moved between states, you had to save and a comment was mandatory (so people could capture even more detail).

It meant that it took ten minutes just to close a defect that you'd already tested.  No wonder there was a bit of a rebellion - testers wouldn't use TFS to track defects because it wasn't suitable.  Yes, no-one could use it easily, but heck, it gave us great audit trails!  [We eventually won pressure to considerably simplify that lifecycle by the way]

I have seen similar comedy-of-errors played out when,

• an industrious manager introduces more states into test/defect tracking system
• the same manager then finds they have to chase/bribe everyone to update the defect tracking system.  Why?  Because they're finding it just too awkward to use.  It's not at all helpful.

The denial in this case comes in two places ...

• The test team needs to slave their process to the tool.  Not the tool "make things easier for the tester".  The more money was spent on the tool, the more justification that it's the tool's approach that's right over the testers.  Throw in a few choice phrases like "enterprise standard" and "best practice" to add to the smokescreen of why the team has to bend to the tool, and not visa versa.
• More customisation = better.  Our tailored defect flow above that fits one purpose fairly well, but was lousy for anything else.  I see a lot of projects which go out to try and customise their product to the nth degree - usually powered by "but what if ..." scenarios.  The problem is the process which typically comes out looks more like a map of the London Underground.  And whilst that might cover a few choice scenarios really well, what typically occurs is even the simple stuff is nightmarish.  The bottom-line is no-one wants to read a manual to understand how defect flow.

Difficulty setting: "Is Government involved?"

Ironically I see the same temptation with sprint task boards to "overcomplicate them", because they're too simple.  Believe me, simple can be beautiful!  Because needless complexity is almost always unusable.

My rule of thumb is if someone is pressuring you to have a state "just in case", resist at all costs.  And if you find a state which you frequently skip on the way to another state, think about removing it - that state is not telling you anything!

Denial 103: "But we spent a lot of money developing our automation"

Previously we looked at the psychological effect of denial, and what drives it and makes it tick.  Today we look at an example of denial at work within testing.  Ladies and gentlemen, I give you exhibit B ...

"But we spent a lot of money developing our automation"

I know the lure for a lot of people when it comes to automation is,

• the tool is cheap (free is best yes?)
• it's quick to produce automation scripts
• it can run scenarios faster than a manual tester can

In a way this trap shares a lot of ground with the "testers spent a lot of time scripting" fallacy we discussed last time.

If it's quick for you to create scripts in, it's easy to try a short one-week demo of the product, and choose it as your desired automation strategy.  However, six months down the line, your manager is wondering when the automation magic will kick in asking "where's my ROI?".

Now, as much as I hate that term, they've got a point - by now you have a large library of automated scripts.  But they constantly need running and correcting (that wasn't in the brochure). So much so that they constantly fail, mainly in small but irritating ways, and typically the fails require a modification to the automation script rather than find problems in software under test that they're supposed to check.

And any time there's a major change to the system, large numbers of scripts need modification.  Heck, they all needed some modification when the login screen was modified!

This is because when you evaluated the tool and the method, you went with cost and how easy scripts were to make (hey, record and playback, how much simpler could it be?).  What you missed was maintainability.  This was covered in a WeTest Workshop from way back, and dealt with under "Automation TLC".  But an article by Messers Bach and Bolton which has just been released covers some similar ground.  [As a hint, I've found that if you don't know what the term "code reuse" is and you're writing large numbers of automation scripts, maybe you shouldn't ... ask a developer instead].

The denial mindset here (much with manual scripts) is that you've invested a lot of time and effort into automation scripting.  At some point that really should start paying off.  Right now it's not going any faster than the tests the manual testers used to run.  And although the automation occasionally find problems in new software builds, about 80-90% of the time they find a problem in the script itself which needs changing.  And typically if it's a problem in one script, it's a problem in many scripts.

The problem is, if you've not chosen a tool and built up your automation scripts with maintainability in mind, it will NEVER pay off.

Your scripts are too brittle, and they will continue to break in multiple places from small changes.  The more scripts you have, the bigger your maintenance bill of stuff that needs changing.  The hard thing is, it's probably easier to start from scratch with a new tool, than to try and build in your maintainability with what you already have.

So we chose a bad tool ... and implemented badly as well.  Is this denial?  If you're continuing to fix it every time it breaks, then yes, it is.

What we have here is what I like to call "technical testing debt", and it shares attributes with other testing debt.  This debt keeps rearing it's head - you don't have the time or backing to go back and deal with the fundamental issue.  So you band aid it, and then band aid it again, and then again.

And because you're addressing the problem so often there will be a perception problem, that surely that technical debt is decreasing with each occurrence you fix.  Right?  The technical debt is causing you to sink in time and resources - and therefore the sunk costs fallacy makes people say "as you're spending time on it, it must be decreasing".

No - not at all.  Every time it's being fixed, the fix for it is only for those occurrences where it breaks - the places where it surfaces.  To really do justice to the problem, you have to pretty much come to a full stop, and do a serious reworking of the approach to the problem, going much, much deeper.  That's something that's very hard to get backing for - especially if someone thinks you're addressing that technical debt piecemeal "because it keeps cropping up as a problem".

The good news - not every automation process goes like that.  But talk around, everyone I know has the tale of a suite of automation which went this way.  The trick is to know that if you're fixing breaks in your scripts than finding problems in your software under test, you need to ask if you're in a state of denial, and you need to address some fundamental problems in your automation strategy.

Post-script

Sometimes after I launch a blog entry, someone gives me a link so good, I need to add it to my article.  So thank you Simon for recommending this article.

Denial 102: "But we spent a lot of money writing test cases!"

Previously we looked at the psychological effect of denial, and what drives it and makes it tick.  Today we look at an example of denial at work within testing.  Ladies and gentlemen, I give you exhibit A ...

"But we spent a lot of money writing test cases"

I talked a little about this phenomenon in a post way back in 2014.  In my career, I've had to revisit old testing schemes several times... heck, let's just call it what it is "grave robbing dead/dormant projects".

Prior to every excavation of archives, I will have heard the legend "well we were late going into testing and the testers spent a lot of time writing scripts".  This usually comes from non-testers, and gives the indication that as far as they're concerned I am sitting on a tester's gold mine of material.  It has to be valuable, because we spent a lot of money on it.  [It's value to me has to be equivalent to the cost sunk into it]

Here's what I hear in that sentence ... "well we were late going into testing" in my ears means that there were a lot of problems.  So many that a basic version of the software could not be made to work enough for testing to start.  That says there were fundamental issues on this project.  Could one of them be that no-one really knew the specifics of what they were building?  And if that is really the case, what's the chance that testers magically had a level of clairvoyance which eluded the developers?

And then there's the part that goes "and the testers spent a lot of time writing scripts", which roughly translates to ...

I kid you not - during my career I've seen some managers try to send any test contractors on leave for a month until it's ready for testing "to save costs".  So sadly if you're a contractor and you want to be paid, there's a benefit to looking busy.

As I've mentioned in my previous article, a quick attempt to correlate the execution logs with the test scripts often shows me whole reams of planned testing which was dumped and never run.

New Zealand is such a small place, which means you might have worked on that project.  You might have left the company.  You might have left the city.  BUT I WILL END UP FINDING YOU!

It frequently happens - informally I might add - and I'm glad it does, because it allows me to dig deeper to find out what really went on and find out another of a project's story.  And it typically points towards a problematic project - from vague outline to changing scope - rather than a tester who just wrote scripts they had no intention of running.

Sooner or later though, it's my job to burst people's bubble that they're not sitting on a goldmine of test scripts that are ready to run.  I'll take a scan through, and try and use what's been written to test the system to see how much use it'll be.  Invariably though it's the results or execution logs which tell me a lot more about what was done than a folder filled with test scripts.

This aligns with James Christie's discussion around how ISO 29119 impacts testing.  That standard focuses more on plans and scripts as auditable artifacts.  James Christie argues that an audit always wants to focus first and foremost on what you actually did (over what you planned to do).  And he has an excellent series of articles on that starting with part one of "Audit And Agile" here.

What I've learned with bursting bubbles is to do it gently.  Always work out ahead of time if anything is salvageable, and have your counter offer ready (so we're going to exploratory test instead maybe?).  That person who thought the testing was pre-written was probably hoping that your test effort would be minimal because you'd be able to build your effort from past work.  Let them know the degree you can capitalise on - heck if it just gives you a good set of starting ideas that's something significant.

Even if you have a good group of test cases written which match the execution log you have, you'll still have to spend time learning the system and it's intricacies.  For instance there might be a line going "expire the account", and it takes some time to find they have a test tool written which will age an account to it's expiry date, but you need to find where they kept that tool.  Almost always a lot of verbal knowledge will have gone, or the details are there, just buried in a heap of other details.

Also remember having too much documentation, as above, can be as much of a bane as none at all.  Because you have to spend a lot of time going through it all before deciding if it's any help or a hinderance.  And I don't know about you, but I'm a bit of a slow reader.

Denial 101: Something I find hard to believe ...

Previously we looked at the effect of peer pressure when we try and adopt the new idea that's "all the rage".  Today as promised we're going to look at the impact of denial, an effect I find very much paired with the "group delusion".

As originally planned, I would be diving right in and exploring the places where we find denial as a very real effect within IT departments.  However as I wrote and expanded my ideas, I found a little too much material - so I ran it by a friend who said it really deserved to be split up over several posts ... so change of plan!

Although I talked a little about denial in my piece of the psychology of The Force Awakes (and believe me have I seen that piece validated in recent weeks), I want to spend this first post looking further down the rabbit hole understanding more about it.

Like any of the effects we've talking about, it's easy to feel a little smug and superior as we talk about it.  As if these issues are something that "happen to other people".  But denial is such a powerful effect on us as human beings because it works on our emotions (like many of the psychological effects we're looking at).  We'd love to think that we use rational thinking to control our emotions, but often it works the other way around - our thinking tends to be slaved to rationalise the emotional outcome we want.

We are all led astray in our thinking when emotions are entangled.  The point of this series of articles is to shed a bit of light on common traps we fall into, so we can be a little wiser - maybe asking ourselves a little "is this a MacGuffin effect?".

In a nutshell, denial is the rejection of facts because of an emotional reaction.  I like how this is covered within the Your Deceptive Mind chapter on denial where it says that people commonly fall into a denial trap and will start with their desired (emotional) outcome, and use it to systematically reject any evidence which does not support that outcome.  This form of reasoning increasingly requires the presence of conspiracies to support their model of thinking.

And indeed a really good example of this is the piece I wrote in 2014 where we tried to convince someone who believed in a flat earth that the Earth was round.  In that piece they initially respond to evidence with vague science, saying "the Earth is round ... but flat ... like a plate".

Then when it's said they could try Skyping someone in another part of the world to see if it's night there whilst day where the sceptic is - however they're convinced the other party will be part of the conspiracy, so debunks the experiment.

And of course there's the line which several people have said is their favourite, "When someone flies from London to Auckland via America, it's okay, because that route exists.  But if they go via Asia, then the pilot flies around Africa for a bit until the passengers get disorientated and it gets dark.  He then heads via America, to New Zealand, stopping off at a secret replica of Singapore they've build deep in the Andes".  More conspiracy.

So what leads so many people to go down that path?  Whenever we make any kind of decision, we essentially make an investment of time, money, ego, and pride into that decision - we're obviously committed to that decision "coming out alright".

But sometimes pride and ego will mean we just stick to that decision, even in the face of new and glaring evidence.  We want to be seen to be someone consistent, not someone who is ever wrong.

So rather the rectify our decision, we'll choose to undermine any contrary evidence just like our flat earther.  This is known as the "escalation of commitment" - where people continue to justify commitment of time and money based on an initial decision and "we've already invested in this course of action".  The phrase "throwing good money after bad" is of course one which perfectly sums up this behaviour.

Here's an everyday example for those of you who can remember driving before the days of SatNav.  A couple of friends, Thelma and Louise, are driving to Mexico City.  They're supposed to be stopping at the Grand Canyon as they do so, and they passed a sign that said they'd see it in 10 miles ... but that was 15 miles ago.

Thelma wants to go back as she thinks they've missed a turning.

Louise says she has an excellent sense of direction, and is sure they'll find the Grand Canyon soon enough.

Thelma says she's seen a sign saying they're heading towards a place called Bitter Springs, which means they're heading in the opposite way to both the Grand Canyon and Mexico.

Louise is sure that Thelma is just reading the map wrong and is not about to turn around now.  This way has to end up in Mexico eventually.

Right?

The couple having an argument about directions, and someone just won't turn back, because "we've come this far".  Sound at all familiar?

Our flat earther has invested time and pride into his worldview, and because conceding that worldview means conceding his pride, he refuses to.  Even to the point of turning down a "round the world cruise" he won on the lottery, because it means admitting being wrong.



So - the question is, where does escalation of commitment happen in testing?  And I'm afraid to say, it happens everywhere!  Pretty much anywhere you've sunk time and money into doing something, there is a state of mind that wants to continue doing that course of action ... because it has to pay off eventually, right?  Oh dear God please, it has to pay off!!!

Other everyday examples of denial and escalation of effort you might want to think about ...

• A friend pointed out the Vietnam war followed this behaviour all too chillingly.  It started out as a small involvement of US forces.  But as it went badly, more and more forces were brought in from the American side, because "we've come this far".  You see something similar in the "one big push" mentality in The Great War, particularly in The Battle Of The Somme.  I talked about the Battle Of The Somme, and "sticking with the plan" in the face of changing evidence here back in 2013.
• "I read about this amazing diet last week.  I mean, I'd tried a few other diets over the years, but this one actually works.  I read it in a magazine."  You can substitute the world "diet" with any piece of revolutionary fitness equipment which "is like having a gym in your own home", and has so changed the world, it's not available in shops ... only to order over the phone on a midnight infomerical slot.  [It's like the gyms are conspiring to keep your membership]
• The right wing American politician whose response to this month's school shooting death toll is "the victims and their families will be in my prayers".  Just like they were last month - except this time they'll pray really hard.
• The Aztecs used human sacrifice to appease the god of rains.  If there was no rains, then obviously they'd not sacrificed enough people.  I talked about this here in 2014.
• Variants on this meme when politicians have committed to a policy, despite it not producing the results they promised ...

If any of those points made you squirm uncomfortably, then well done, you're waking up.

[By the way - I've taken a few shots at the right wing there, and I'm an out and out socialist at heart.  But critical thinking still applies to me - especially if someone is sharing on social media a "new item" which aligns with my beliefs.  I often am somewhat suspicious if it falls into the camp of "I knew it", and start checking up on it.  To be honest such fake stories annoy the heck out of me, because they undermine my political stance, and make it just way too easy for friends who have different opinions to "score cheap points" over me.]

Peer 102: The MacGuffin effect in testing ...

Previously we looked at the then-to-be-released new Star Wars movie as a way to explore some psychological phenomena which we'd unconsciously be exposed to in December 2015.

Today I want to focus on one of those key factors (group delusion), and see how they affect us at work as testers.

Group Delusion

The term "group delusion" feels an awfully derogatory one - but as I discussed last time, it's subtle and can affect us all.  I personally see it as akin of peer pressure.  To me there's somewhat of a grey-blur between them, and they add up to the same kind of thing - an unconscious drift in our thinking.

I want you to imagine this scenario ...

You've just meet up with a couple of testers, Bev and Steve, who you worked with two years ago. Although it was meant to be more a social catch-up, it doesn't take too long before you start talking shop!

Bev mentioned how they've recently started implementing a MacGuffin strategy to their testing, and she's finding it really interesting, but a bit challenging.  Steve snorts a bit - his project has been using MacGuffin for over a year, and would never go back!

That evening, you get some email spam from a recruiter (you remind yourself you really should unsubscribe one of these days), when you notice the first job calls for "experience applying MacGuffin is a must!".  You scroll down, and it's not the only role which mentions it.

It only gets worse the next day when one of the senior managers calls you and several other testers into an office asking why you've not started implementing a MacGuffin policy yet ...

So we need to get down and start implementing MacGuffin right?

Hopefully your first reaction really is "what is MacGuffin, and why will it help?".  Sadly not everyone will.

A MacGuffin is a great term for this kind of effect in testing - it's a word often associated with Alfred Hitchcock, it's tool used as a plot device, and often kept vague and mysterious.

The mysterious MacGuffin used in Pulp Fiction

Over the last few years I've seen a few MacGuffins being traded around social media in association with testing.  I notice the MacGuffin effect when I start to see people wanting to build up their strategy not because a MacGuffin will help them with a specific problem, but because everyone else seems to be working with MacGuffins, and they're worried about being left behind.

It doesn't matter what they are, there is a peer pressure which creates a kind of what's called a "keeping up with the Joneses" effect.  If your neighbours, the Joneses, go out and get a new BMW and a 50 inch plasma TV, you end up asking yourself why you don't have these things, and thinking that really you ought to go out and get them yourself.  Even if it's something that previously you felt you never needed.

Let me give you a few examples of MacGuffins, and I'll break down the pitfalls I've seen ...

• Test automation
• Agile
• Test management tools

Test automation

I'm actually a huge fan of automation (though I agree, it' more checking than testing).  Used well it can help my job as a tester because it will mean a developer can check that software works to at least a level where I can interact a little with it.  If it's so fundamentally broken that a user can't even log into the system, far better that's found with a little automation, than for me to find on Monday, then be told I can't get a build for another week.

The problem is automation is somewhat over-sold.  This really goes back to the 90s, where the sales blurb for the expensive software talked in terms of there being no need for testers.  All in the order of "costs half a testers wage, tests 100 times faster!".

The problem is you need a really good tester to come up with the ideas for scenarios to check.  Then you need someone to program it all up.  Then to run it.  And check the results if they show issues.  Then to maintain it when you change your code.  You still need people, and they're kind of doing "testing-esque" roles (especially determining scenarios, and investigation).  [Though you might need less of them]

If one of your tests "shows a red" (ie a fail), I guess you'll need to try and manually rerun the scenario to see why you get that result ... isn't that most likely to be a tester?

So even in a perfect scenario, you're still going to need people working testing (sorry if you bought the sales pitch).

However when you're sitting with Bev and Steve and they're going "automation is amazing ... it helps us to test really fast", everyone would want a piece of that action!  Especially if they scoff and go, "what .... you're STILL testing manually?".

I myself have experienced this kind of pressure - I had a manager at a previous company want to know why we weren't using automation for our current project.  This project was an integrated piece of hardware similar in many respects to a supermarket self-service kiosk.  They'd heard you could get some free software called Selenium, and a friend was working it on another project nearby.

Well, that was an unexpected item in my bagging area!

I was lucky to be able to go through the details with this manager, and get them to understand why it wasn't suitable, and I was glad we were able to go through it together.

My points were,

• For automation to work well, we really needed the product in our hands in as finished a state as possible quite early on.  The reality was we'd scripted up some scenarios, but we would not get so much as a demo of the product until it was delivered for testing (and believe me I'd tried).  We were on a tight schedule, and so automation would jeopardise that.
• It takes a long time to automate a test.  But it pays off if you expect to have to run that test dozens of time.  We had a 6-8 week window, expecting to rerun tests at the most about 3-4 times.  Once released we'd not touch this product again for years (if ever).
• Primarily though, Selenium works on web-based application.  Our application was not even remotely web-based.  The technology just wasn't suitable.  I had to go through some Selenium websites to show this to him, but again worth doing so he'd understand.

All the same, it's easy to see how he got MacGuffined.

Agile

Again - I'm a huge fan of agile.  Just not when it's mis-sold.

I see and hear this a little - companies who've felt compelled to "jump on the agile bandwagon" because they've heard about everyone else's huge successes.  But without really "getting what agile is".

Agile though is a hard sell - when you tell customers the reality that "agile isn't about succeeding, it's about making failure more up-front so it causes less damage, cost and is easily rectified", some look a little shocked and terrified.

I thought you promised us success!

Agile has become such a big MacGuffin, that no IT company can afford to say they don't do it.  But unfortunately there's a little bit of "just call it agile" out there, where the principles aren't really being followed or understood, it's just a rebranding of what was done in waterfall.  As I've talked about addressing our agile transformation - there's an awful cargo cult trap of trying to keep doing what you've always done, but "wack an agile label" on it.

Warning signs for me tend to be,

• "Well ... we do stand ups" [That is - that's the only difference which has been noticed]
• "We have a 2 week development sprint ... which is followed by a 2 week testing sprint"  [Mini-waterfall]
• "We' then run our retrospective ideas past management" [Sounds awfully like the agile team are not empowered]
• "We do SAFe"

Test Management Tools

I'm often told by their advocates that "the best thing about this McGuffin Test Management Tool is that you get all your reporting for free".

Actually, let me repeat this and add emphasis - "the best thing about this McGuffin Test Management Tool is that you get all your reporting FOR FREE!!!".

Just like agile and just like automation, test management tools can be a very useful.  For very complex projects, with huge numbers of testers, it allows you to break down the many areas of testing into different areas, track which tests touch which requirements.  And that's useful.

When advocates of these tools scoff with "well, you can't just use Excel, can you", I usually squirm a little, and reply "well, often Excel is my tool of choice".  Am I mad?

Here's my problem with test management tools.  Many of them whilst they do allow you an overview, it comes with a price, which is not free.  And I don't mean the "per seat license dollar cost".

Test management tools often constrain testers to work and test in a particular fashion - one which often is not the natural/logical/comfortable/most-efficient method to how some testers operate - especially exploratory testers.  If you have a tool which isn't really suitable for how you work, it feels clunky, it slows you down, and creates a drag on your work velocity.

They can also give a false perspective - because the kind of high-up managers who love them rarely dig down to the detail.  A couple of years ago, I had one such manager mention how nice it was to be able to see 500 scripts scoped out for the testing effort, and like a commander they could watch their status when run.  When I looked into some of these tests, they had a title, but the majority of them had no steps ...  It's something I'm aware of.  I know of some projects around Wellington where they had the test management software mandated, and the testers found using the system for testing so difficult and clunky, they ended up only using it to track bugs.

This is how some people think the charts in test management tools work ...

I like to use Excel when practical, because I can use it in any manner which makes sense for me to keep notes for testing and for breaking down the effort.  If a team is less than 5 testers, you probably don't need one.

I also know looking at graph or output from these tools does not show me where we have problems (and I'm a trained scientist good are reading trends in graphs, but also noticing noise).  Often those problems can lie under the surface of the pretty graphs.  Far better to have a daily stand-up with other testers, and get them to talk about any problems they're encountering.  That avoids me having to stare at numbers or statuses on a graph or dashboard trying to "read the signs" like some fortune teller.  I trust my testers to tell me about problems than I do a tool.  If I don't trust them, I shouldn't have them working for me.

Likewise when it comes to scaling up a group, I'd rather have 15 testers split into 3-4 groups with a team lead, and all those team leads doing a stand up with me daily to pass down problems and pain points.  Even if we're using a test tool, it's important to keep talking to the people using it.

[I'm going to avoid a discussion on test metrics and reporting here, as I've something upcoming on just that topic]



So that's the MacGuffin effect leading us into sometimes costly mistakes!  Any of that feel at all familiar?  Any other MacGuffins you've come across?

Next time we'll look at the phenomena of denialism, and the sting in the tale there...