Thursday, May 26, 2016

Jutland Fallacies

This week commemorates The Battle Of Jutland - the largest naval battle in history that took place during The Great War between the navies of Germany and The United Kingdom.  In this piece we're going to look at the run up to the battle, some of the thinking of those in charge, and looking to see if fallacies were in place.  And even try to spot them!

Setting the context

At the start of the Great War, the British Empire covered the globe, serviced as it was by a giant navy which since Nelson and The Battle Of Trafalgar stood unopposed.  Britannia ruled the waves.

Such a large sea empire demanded a large navy - and Britain had the largest on earth.  Until recently it was large enough to challenge the next two sea powers combined.  However a new kind of battleship, the dreadnoughts had so revolutionised naval warfare with their huge guns and steam turbines that they practically rendered all other ships outdated and obsolete.

The British Empire was in a building frenzy to maintain dominance, but Germany was catching up fast by the outbreak of war.

The German Navy had sought to trap pockets of the British Navy and whittle them down, however at Jutland, the entire German Navy fell into a trap, being lured into a headlong battle into the entire British Navy.

The British Navy were in superior numbers to the German Navy - 151 ships to Germany's 99.  The British Navy were not only in prepared firing ranks to maximise their firepower, but were also one of the best trained and professional seamen in the world.

As the clickbait says "what happens next will amaze you ..."

Germany 3 - Britain 1

Jutland is a complicated battle to sum up.  From the actions on the day, the battle is a decisive German win.  When looking at "the big ships sunk" Germany will lose 1 battlecruiser, whilst Britain will lose 3.

But there is more to it than that - German battlecruiser Lutzow would be hit 24 times by British shells.  The damage was extensive and it was originally hoped she would make it back to port.  However eventually her handling was so poor she had to be scuttled.

On the British side though, there's a very different story - the battlecruisers Indefatigable, Queen Mary and Invincible all suffered a very similar and violent end.  In each, the ships exploded after just a handful of hits.  In each case, of the crew of over 1000, only a handful of men survived.

The day would lead a British Admiral to commend "there seems to be something wrong with our bloody ships today".  What had gone so wrong?

Tales from the magazine...

Diagram from Wikipedia 

To answer that question we need to cover off normal operation of a key component of a battlecruiser - the gun turret, and how it's fed ammunition from it's magazine.

The magazine - where shells and cordite explosives are kept have always been in the deepest parts of the ships. The theory being they're below the water line, and harder to hit with chance shots.

In peacetime onboard British ships, priming the big gun with ammunition was all about safety.  The shell room and magazine for cordite are highly armoured, and isolated from the lift shaft that feeds the turret by safety doors, to prevent the chances of a blast blowing through into either - a disasterous situation which would spell instant doom for the ship.

That makes sense on paper...

That was peacetime, but in 1916, this was war!

Within the modern warships of the time, there was a kind of iron triangle rule in place,

  • Armour
  • Firepower
  • Speed

Pick two.  The battleship focused on armour and firepower, to the determent of it's speed.  The battle cruiser though picked more firepower and speed - it was seen as almost a naval form of cavalry to do lighting raids and get out, particularly picking off enemy shipping.

British battlecruisers focused more on having big guns, sacrificing armour compared to comparable German counterparts.  There was a logic to this - British battlecruisers could strike German battlecruisers from further away, where the German ships would not be able to strike back.  Their range would be their armour.

Being a better drilled and practiced navy, Britain could maintain a faster rate of fire from it's big guns than German, and during the war, this rate only increased.  And this too made sense to the Admirals, because if the Germans were being heavily shelled at a faster rate, they would struggle to reply.  So the British fleet's rate of fire would be their armour.

To make this rate of fire as fast as possible, it was decided in drills to stop the practice of shutting magazine and shell room doors whilst feeding munitions up to the turrets above.  They would be kept open to "keep feeding the guns".

This would of course be risky - these people weren't idiots.  But it had a logic to it - the more shots you got off at the enemy, the less time they'd have to respond, the quicker they'd sink the enemy.  Obvious.

But this also presented a problem - the sustained firing rate meant in a heated battle, a British ship run out of ammunition before it's German counterpart.  There was only one logical thing to do - to make sure you were carrying more ammunition - so it was decided to carry about 50% more ammunition in wartime.

Of course there was a small problem - there wasn't room in the current magazine for any more ammo.  So instead it was stored throughout the ship, wherever there was space.  Of course this wouldn't be in an armoured space, and was incredibly risky.  But there was a logic to this - this ammunition would be used first, and besides with the better range of guns, faster rate of fire and speed, it's not like the ship was in much of a risk of being hit at any rate.

F-A-L-L-A-C-Y spells BOOM!

On paper, each of those arguments seems to make a certain kind of sense,

  • Big guns means being able to hit from a range you can't be hit back. So you'll be safe.
  • If you hit them more, they have less chance to fire back. So you'll be safe.
  • The more you shoot them, the faster they sink. So you'll be safe.
  • Take more ammunition. So you'll be safe.
  • Because you're so safe, you don't need to follow full safety precautions which only get in the way.

I love the definition Wikipedia has under fallacy, "a fallacious argument may be deceptive by appearing to be better than it really is".

To me, a fallacy is an idea which seems to initially make perfect intuitive sense ... just so long as you don't start thinking too much about it, and finding all the holes.  I've found Daniel Kahnman's Thinking Fast And Slow has been filled with superb examples where our fast, intuitive thinking can lead us astray, and where to be wary.

All those arguments make a certain logical sense.  What's missing though is that as you take each step of action, you're taking a course of action that's putting yourself at greater and greater risk.  The question not being asked is "does the extra aggressive punch this gives me take off comfortably with the risk?".

At Jutland the answer turned out to be a resounding "no".  With all the British battlecruisers lost, the explosion in the magazine happened soon after a chance shell hit the turret, setting off a catastrophic chain reaction - one which could have been prevented by following the normal peacetime safety guidelines.

HMS Lion - narrowly avoided a similar fate

This was almost the fate of the HMS Lion as well, but thanks to the quick thinking of Francis Harvey, he managed to shut the blast doors on the turret to prevent flash fires reaching the magazine below.  Although Francis himself perished in the explosion.

Lessons to be learned

It was only a few years ago we were looking at the series of bad judgments that caused the Titanic disaster.  There are a lot of similarities here - when we try and be too clever, we try and twist the logic of reality for our own purposes.

However as Richard Feynman pointed out several times, "nature cannot be fooled".  Your clever arguments hold no weight against pure scientific fact.  I like to think this is really about the core need for testers as well within IT projects - they're the people onboard whose job it is to try and think deep and ask some important "now hey, wait a minute" type of questions.

If every a course of action seems too neat, attractive and instinctual, that's a time to sound off your fallacy alarm, and wonder "okay - what could go wrong".  There's always a drawback, so find it, before it can find you!

Interested in the history of Jutland?

Want to know more, including how despite the British fleet being so badly hammered, it's also considered a strategic British victory?

I highly recommend this documentary on YouTube.

If you live in Wellington, and would like to talk more about the battle and it's history, my wargaming club Wellington Warlords are running the last game recreating the battle from 10am on Saturday 4th June 2016.

Wednesday, May 25, 2016

Testing for all?

I've talked a little in the past about qualifying as a teacher just after University.  Although I never spent much time in the profession, never the less I wouldn't give up that qualification for anything.  I learned a lot about myself and also about how people worked.

One assignment never feels too far away - we were given an essay to write called "science for all?".  And given the remit to explore diversity and inclusion in how we taught.  Each of us seemed to go our own ways with this, and some of the conversations we had on student teacher pub nights were the most illuminating learning I did on the course. It opened my eyes and my horizons.

I find it sad we have a "professional qualification" in testing which doesn't make us think as much as that single assignment on my teacher training di.  Do use the comments below or tweet to me ( with the hashtag #testingForAll ) to tell me what challenge you think of most when you think of "testing for all?", why is there a problem, and what can be done to cause real change?

Think inside and outside the box.

Wednesday, May 18, 2016

Memory Lane

I've previously written a lot about memory - but today I'd like to explore a particularly important one from long ago.

Being middle aged, and having moved around a lot, I have a many memories from different places, some going back to my early years.  Mostly they're well behaved and "remain in a state of slumber" ... and sometimes one will need a little jog before it goes back to sleep.

This was the Twitter picture which triggered one of my bittersweet ones,

My wife Collette and I have been together for over 20 years now.  That's so long that it's easy to take for granted.  But a memory of Liverpool is enough to stir those of our early days together.

I always found dating kind of awkward until I met Collette - with her there was never any need for pretense.  In came this woman with an incredible energy and personality to both match and at times complement my own.  We were both incredibly passionate people, although that passion didn't always align, and could cause some rough waters at times.

It was an important relationship - she always challenged me to be more than I could, an odd mix of loving me and pushing me.  But there was always in those early days a bittersweet ritual we had to live out.

Collette's parents were quite traditional, and we never wanted to upset or shock them.  So although she often stayed with me, she could never stay the night, having always to be back home come the morning.

Thus in the early hours, typically around 4am, one of use would wake in the others arms, wrapped together as we were in order to share a single bed.  Driving her home through the dead of night in Liverpool was always such an eerie experience - a city so vibrant during the day, so empty and desolate in those early hours.  The city slept whilst two lovers prepared to farewell each other.

We rode mainly in silence, occasionally Collette ghoulishly pointing out a street corner where a gangland slaying had occurred.  It was all to take our mind off the goodbyes to come.  Alone in the early morning, it felt we wrote our love on the streets, as each red light in our path was an opportunity to hold her soft, warm hand in the bracing morning, or to steal another kiss while we could.

All too soon she would be home, and I'd face the lonely journey home.  All the time thinking what a wonderful thing it would be to be able to wake up in the morning together.  It was like living in a fairy tale - I had someone I loved and who loved me very much, but I always had to give them up before the dawn ...

These days, I like Sundays the most.  Sundays there are typically no alarm clocks, no pressing reason to get out of bed.  Most Sundays you'll find me awake, but lying in bed, next to Collette, and remembering how to my 25 year old self, waking in the morning next to her was all he felt he needed to make him happy.

I think one of the saddest thing about being human is we yearn so much for things, but when we get them, we so rarely get to enjoy them or appreciate them.  All too often one sense of want is replaced with another - we're forever hungry and insatiable for some need.

Sometimes the right memory will help you remember how much what you have now is all you ever wanted.  Find yours, and never take it for granted.

Saturday, April 23, 2016

RIP Choppers

They've been saying that 2016 has been a bad year for celebrities - with a huge number of our UK childhood icons passing away.

This week has been more of the same, with the deaths of wrestling's Chyna, Britains much loved comedienne Victoria Wood, and iconic pop powerhouse Prince.

Then this appeared on my Facebook feed ...

Choppers, one of the last of the PG Tips chimps has died.

Thankfully - you might not know who the PG Tips chimps are - but in the 60s and 70s there was a series of adverts.  The ads involved chimps being dressed up to behave like human beings - it was a source of great amusement for many Britains.

Sadly though, we eventually found out there was a very dark side to our amusement.  The chimps lived in Twycross, and were a major draw there.  However one time my mother had a bit of a clash with the trainer.

It turned out the chimps were brought up and rigorously trained with human mannerisms.  The trainer who came across as an absolutely hateful woman was making a big deal about how the chimpanzees used for the adverts were carefully kept away from other chimps so that their behaviour didn't get contaminated by real chimps.

My mother - who is no Dian Fossey - was horrified, saying "not allowing an animal to behave naturally, never heard anything so ridiculous in all my life!".  Indeed it was the last time we ever visited.  Thankfully the ads fell out of favour in the 80s as we understood more about ape psychology through work of people like Dian Fossey.

Unfortunately the damage had been done - the PG Tips Chimps never really managed to integrate back with other chimps.  Indeed the current manager of Twycross Zoo is open about the damage that was done.

It's interesting to take time to think about this.  Zoos originally began as a Victorian curiosity where animals from around the world would be collected for the amusement of paying customers - with confined conditions and taunts from the public commonplace.

The PG Tips chimps are a relic of that kind of thinking, where the primary concern is the amusement of humans.  However zoos around the world have been trying to shake that image - now the focus is not so much amusement, but helping with the conservation of endangered animals, creating better habitats and considering animal psychology, and using the animals as ambassadors for why conservation matters.

It's a powerful reminder how to stay relevant, institutions need to be prepared to change, and to question the behavior and practices of the past.

RIP Choppers - and let's hope we've learned our lesson.

Thursday, April 21, 2016

The danger of letting extremes and corner cases dictate our systems

I found myself quite sickened today reading through the news.  Anders Breivik recently won a human rights appeal case against how he is treated in his incarceration.

Breivik is a notorious Norwegian mass murderer who killed 77 people in several terrorist attacks.  Much like Beate Zsch├Ąpe, he identifies as a neo-Nazi - in only the way someone from Norway who was born after the Nazi occupation remotely could.

In my mind, the unrepentant killer is an absolute monster and there is no greater crime against human rights that the commitment of mass murder.  And so the verdict was somewhat sickening to me.

It's been the cause of a great amount of thought all day.  Stories like this often make the news, because we obsess how something in our society functions when we look at the "corner cases", the extremes of society we can't predict.

Breivik himself presents a conundrum - we was 32 when he committed his crime, but in Norway the law system has a maximum sentence of 21 years for any crime (although under the containment clause it can be extended).  They had never really imagined having to deal with a criminal like Breivik before.  Although murder can be found in any corner of the world, it was just still an unprecedented act in the country (ironically) since the Nazi occupation of World War Two.

We seem to be drawn to these kinds of corner cases in any of our systems,

  • The mass murderer and his human rights
  • The silly health and safety rules
  • The social security fraudster

Often the obsession is not to fix the system - but to use these extremes to dictate the norm - which is usually used to support an overeaction that involves penalising the whole system.

We talk much about health and safety rules being silly (and some are interpreted trivially and obstructively), but lose track of the fact that they are in place to save lives in the workplace. And have helped to contribute to a fall in the number of work related fatalities by ensuring a right to safety equipment, and evaluating areas for potential problems.

The fraudster on social security is used as an excuse to make social security more restrictive and punitive.  When the system exists to help people in genuine dire straights (but they're often forgotten in the outraged hunt for the fraudsters).

In many ways, human rights as they stand in Europe are the legacy of Breivik's idol, Adolf Hitler.  After the monstrous acts committed by the Nazis, they are an attempt to legislate against ever seeing such state-fueled atrocities from ever happening again.

They are rights that are supposed to be there to protect each and every one of us.  The hideously bitter pill for me today was the realisation that for human rights to mean anything, they have to apply even to a monster like Breivik.

Today as I brooded on it, I would have loved to have a clause which could mean you could take away the human rights of someone like Breivik.  And then it had a ring of the familiar to it - and I realised that human rights can't be something we're entitled to.  They have to apply to all of us, the best and the worst, or they don't apply at all.

Once you create a clause that allows you to define someone as "not human", you have set a precedent.  You have created a reason to take away a persons human rights, someone else will find their own reason ... and another and another.  Once you have made it law that such rights can be ignored "in some cases", those "some cases" will always be exploited by others.  That's human nature.

Adolf Hitler managed to convince a country that the Jews were "sub-human" - much like I'm desperate do with Breivik.  Then it became the homosexuals, the mentally ill, and before too long anyone who disagreed with him.  We all know how that ended ...

Even in my old homecountry - you just need to look at the Prevention Of Terrorism Act 2005 which used the 7/7 Train Attacks as it's justification.

This act allowed special powers to opt our of human rights act - and would only be used to fight terror, right?  In fact it became one of the most abused laws on the books ...

It's a great example of the slippery slope that once you create the loophole, people will expand it to use it for anything.

In the end though, though Breivik won his case, it was a pyrrhic victory for him.  In supporting his human rights, we totally rejected his ideology - that we avoided taking a step down the path where we can ignore the rights of a human being, even one like him.

Somehow for our society, our systems, our thinking and our lives, we need to be able to cater for extremes and corner cases.  But not let the norm be dictated and overshadowed by them.

It's easy to forget, but when we protect the rights of someone like Breivik, we are making it that much harder to take them away from everybody else.

Thursday, April 14, 2016

Thinking about test strategy ...

As background to this - this year I'm trying to collect a lot of ideas on test strategy.  It's my hope I might have enough material for a workshop on it next year.

Right now some of these ideas are getting structured, but at an early stage - so do contact me below or on Twitter with your ideas and comments.  I'd really like to push further on this!

Last week I was running an evening bootcamp on testing for Summer Of Tech.  It's an awesome opportunity not just to touch base with New Zealand's upcoming tech people, but also to champion a little bit about the fun and mindset of testing.

During the presentation though - I gave my current definition for testing ...

I've been using this slide for about a year - but realised how within that statement is something I'm thinking about more and more recentl.  Something that's been under our noses for a lot of the time.  Maybe we're so busy, blinded and programmed, because we've never really taken the time to explicitly think about it.


Let's look at the standard test script format we've all had to use at one form or another ...

In a test script when we use action vs expectation we're really talking about "something I control" and "something I observe".

It's fair then to sum up a test scenario as using something we control to produce something we observe which can then be compared with something we expect.

All testing is a form of play between these factors - even exploratory testing which we talked about previously here.  We use a heuristic to vary something we control, we use an oracle to set our expectations, and we observe - raising defects if there's a variance.

Exploding the idea of strategy

At it's heart then, a test scenario seems a very simple beast - we control something, something happens ... is it what we expect?

This is probably where testing gets it's reputation for being easy - a single test scenario presented like that looks simple.  Even if we overlook how observation can be a difficult to master.*

The problem is that testing is not the execution of a single scenario.  The skill of testing is coming up with a series of scenarios which cover a system.  Again, people can come up with a scenario or two, but that doesn't make them skilled testers.

To me, being able to come up with a selection of scenarios to cover a product is not easy - and there are no guarantees.  There is such a vast variety of ways a product can go wrong, that it can be overwhelming.  Consequently we often will see members of the testing community talking about how huge testing can become.

Unfortunately that can become problematic.  I'm currently reading Daniel Kahneman's Thinking Fast And Slow, and we've come to the idea of substitution.

Simply put, when we're asked a very complex question, if it leaves us perplexed we will sometimes answer another related question for which we do have an answer.  In his book, when they asked students in one survey "are you happy?" followed by "how many dates have you been on this month?".  "Are you happy" is a complex question.  Put this way around, they found no correlation.

They then asked another group of students the same questions, but the other way around - "how many dates have you been on this month?" followed by "are you happy?".  This way around they found very strong correlation.  The reason being that when the complex question about happiness came up, they substituted a question they had just answered "how good is my dating life?".

Now remember how I mentioned how testers talk about "the possibilities of what you can test" being huge?  That means when a new system is put in front of us and we're asked "what will you test?", that it's a big complex question.

So no surprises, we tend to perform a substitution of our own, and hence we tend to answer the question initially as if it was one of the following questions ...

  • "What do you control?"
  • "How have you tested other systems?"
  • "Where have you found problems before?"

Now none of those answers is a bad first answer.  The problem is when we're falling into the trap of using one of those without realising.  They are each a very good starting point, but if you don't push further, you'll end up doing shallow testing, and potentially missing something important.

"What do you control?"

I'm going to revisit material from a previous article on Back To Basics, where we looked at registration.

Let's take a look at Facebook registration for a change ...

It's easy to take a look at this and come up with a list of the following which are things we can vary and control ...

  • First name field
  • Surname field
  • Mobile number field
  • Re-enter mobile number
  • New password
  • Birthday drop down lists
  • Female / Male radio buttons
  • Create an account
  • Various hyperlinks

When we look at these items our first approach is to create a list of test scenarios which use different data and permutations.  And that's as I've said is a superb start.  Where it's not great is if once you've exhausted ideas for this to consider you've tested everything.

Because you haven't.

What you've just done is a very competent job of functional testing.  You've chosen items that you know are simplest to control, and you've created scenarios.

What you've omitted are items for which you don't know how to control.  I think an important part of test strategy is being able to recognise "I ought to be able to vary X, but I don't know how" - then asking the question "how can I control X?".  Unless there's explicit instructions, we'll often have a blind spot for the "everything else" which defines a product which is often thrown into the "non-functional testing" basket.

What are some examples of tricky things you might want to control?

  • Platform / Browser you test your system on.  Different browsers react differently - installing other browsers is easy.  And you can set up different virtual machines to simulate other platforms.  Maybe you need an Apple Mac (good luck prying it from your architect's cold, dead fingers) - and maybe some devices such as the latest iPhone or tablet?
  • Screen resolution.  When was the last time you adjusted your screen resolution?
  • Server load.  Does the system work well with multiple users on?  You can use a tool like JMeter to simulate a lot of users on your system - simply it's a tool that allows you to control a lot of concurrent calls to your system and measure response time.
  • Hardware down.  If you've got a load balanced system, do you have a scenario with part of your system down?  It's not an obvious thing to control, but you should ask how you can get someone to do that.

Of course in talking about the above I've provided potential solutions - which was naughty of me.  You need in strategy to be brave enough to say "I need to be able to control X", even when you're not sure how.  Ask your team, and around anyway.  Maybe there's a tool out there, or maybe the team needs to build it into the system as part of testability.  But don't be too quick to say "I can't control X, so I won't bother".

At worst, if you can't find a way to control, then put it as an assumption "we weren't able to test around different X".

"How have you tested on other systems?"

Again a super method to get started, we use our experience on a previous system, and copy and paste it over.  This is great apart from,

  1. Your testing will only ever be as good as your last project (what if instead of good testing, you were instead just very lucky?).
  2. If the current system is almost identical to the last one, that's great.  But more than likely there will be differences - and do you need to account for them?

A good example, if you didn't need mobile device testing on your previous project, you're in danger of thinking that can apply to this project - are you sure?

"Where have you found problems before?"

Once again - this is a helpful method, with a hideous blind spot.  If you use only test areas where you imagine there could be problems, what if there are problems in the system you're just not imagining?  How can you think bigger in your testing approach?

If you only go looking for old bugs, the problem is you might not be taking a different path to find new ones!

A useful guide for strategy

Is there a particular approach you tend to favour?  In the early days, I tended to get stuck on "where have you found problems before".  I now tend to obsess a little on "how have you tested on other systems".

When you're looking at a new piece of work, capture all of your initial ideas and try not to think too much about it.  Once this has dried up, think how you've approach this - has it been as if you were answering,
  • "What do you control?"
  • "How have you tested other systems?"
  • "Where have you found problems before?"

Think about the blind spots associated, and dig deeper.  Try answering the other two substitute questions, and see if it forces you to go deeper.  And of course, try out the Oblique Testing cards to shake things up a bit!

Regarding Observation

* If you look at my book How To Test, there is a chapter on observation.  However dispite over 400 readers, no-one has yet Tweeted me to say "where is Chapter 7?", this is one of the deliberate mistakes in the book -

Dan Billing has accused me of overplaying the "smart arse" card on this one ...

Friday, April 1, 2016

Take warning - Microsoft Office Assistant returns ...

I feel the need to take to the internet to warn you about ...

If you've applied a recent update then you'll have notice that the scourge of previous Microsoft Office version, Mr Clippy ...

Yes, the bane of all our lives is back!!!

But help is on hand - there is a way to turn him off if you're on an Edge or IE browser ...

Just look under Settings on your browser ...

Then you'll notice a new button - called Assistant ...

Select the option Deactivate Microsoft Assistant, then hit apply.

They hit the option Confirm, followed by I'm really, really sure.

And there - blessed silence.

Less than impressed at the ordeal Microsoft has turned this into.  I have some friends at the gym who work for Microsoft in New Zealand, and supposedly you'll only have a limited window to turn this feature off, before it's likely to become permanent - as a revenue stream to show you advertising through the assistant on your browser or any related software.

So get in there and turn the feature off before it's too late!