A Testing Challenge - Project Balto

Probably the film I've most enjoyed this year has to be The Martian, about efforts by NASA and Matt Damon's stranded astronaut on Mars to stay alive.

There was something about it I loved.  Much like Apollo 13, it's about using imagination and knowledge to solve problems and get home.  One scene which interested me was where they were trying to build a rocket against the clock with some relief supplies ... it annoyed me, because I thought of a better way to solve it.

As part of our pre-Christmas test team meeting, I put a similar problem to them, to see how they would solve it.  It was interesting - their solution was similar to mine, but they came up with some different ideas, some of them better than mine.

That's why at the end of this year I've written up that testing challenge, "Project Balto" for Testing Circus.  I encourage you to look at the problem, and talk either with your team or your family about how you'd go about solving it, and do submit your solution.

I'll be going through and judging the responses in February.  More importantly, I'm hoping to write a follow-on piece in Testing Circus which will look at some guidelines for solving such real-life testing problems in the workplace.

Please do contribute, the more responses I get, the better my follow-on piece will be!

Find Project Balto in all it's detail here,

https://www.testingcircus.com/testing-challenge-project-balto/

Peer 101: Why the new Star Wars film WILL be amazing ...

You're probably starting to read this thinking "this is going to be even more hype for the new Star Wars movie isn't it?", well you couldn't be more wrong.  Once again, we'll start looking at critical thinking and some of the psychological and social factors which will sway us when we watch the upcoming new film.

If your Facebook feed is anything like mine, you've probably been bombarded for the last few months with posts and pictures like the ones below ...

The trailers for The Force Awakens have been going for over a year, with every fan dissecting every frame in minute detail.  There has been a huge rush by many of my friends to buy tickets, clamouring to be the first to see it in midnight screenings.  I too will be going out of my way to watch in the first two weeks.

And there will be a common factor, we'll all go to see it and like the picture above find it emotional, and amazing.  But will it be any good though?  Well ... for that you'll have to wait a few months.

This must have left you a bit bemused - surely if you go to see the film this week, then you'll know pretty soon if the film is any good or not.  Surely?

Good luck with that!  There are actually a couple of psychological factors going on, which mean with a good deal of confidence, I can tell say that you and your friends are going to enjoy the movie.  Not just enjoy, but come out positively evangelical.

And no, it's not Jedi mind tricks ...

Although they are a kind of mind trick ...

NOTE

I'm not saying the new Star Wars movie will be great.  I'm not saying that it will be awful either.  What I'm saying is most people's initial reaction will be that it's a superb film.  Even if they maybe change their stance afterwards.  [See "People In Glass Houses" for more]

Group Delusion

Social dynamics have an important effect on our judgement.  Imagine this - a couple of your friends are talking to you, they're really excited because they're going to see the new Star Wars film tonight.  One of their friends has already seen it and really enjoyed it, so they can't wait.

There's a lot of people you know who are likewise keen to see the film, when you enquire it seems like you'll have to wait a few days for a showing that you can attend which isn't sold out already.  The day finally comes, you sit there in the theatre.  It seems that everyone you know has really enjoyed it, and said how phenomenal it is!

We're engineered as social animals to want to conform to a social norm.  We don't want to be different or the "odd one out".  Yes, even "that person" you might know who dresses differently to you ... we often know someone who tries to stand out as an individual, but conforming to the norm of a different social group (that they do want to align to) - we all had a Goth phase didn't we?

THAT PERSON - Robert Smith, the legend

If our social group of peers all like something, we're hardwired to really want to like it ourselves, so as not to be the odd one out.  It's really just a kind of peer pressure.

Denialism

There has been over a year of advertising.  Many fans have said how they've been waiting over 33 years to find out what happens after Return Of The Jedi (from the original trilogy).  Those people who have watched every trailer multiple times, read every article and seen every interview.

That is a huge amount of emotional investment, and instinctively we always try to justify that.  You see this time and again, "the new film has to be good ... because I've waited 33 years to find out what happens next / waited all year watching the trailers".  We don't like to think we've invested either financially or emotionally into something which doesn't pay off.  And so we justify the quality of what's in front of us, by that investment we've put in.  It's not logical, but it does happen - if what's in front of us isn't any good, then we've wasted that investment.  And that's something we're not at all keen on admitting, so we have a haze of denial about it.  [Also see those politicians who think trickle-down economics works, and when challenged just think we need to give more tax breaks to the mega-rich]

So when we are so emotionally committed to something, we tend to judge what's in front of us according to that emotion, and not by it's own merits.

People in glass houses ...

Now, heaven's forbid if you think I'm writing all of this from some moral high ground, "I know psychology, and you don't".  I of course mainly know this because I've been a victim of it myself!

Back in 1989, I won tickets to an advance screening of the new Star Trek movie in London.  Unfortunately we lived over a hundred miles away, but my parents understood my passion, and so we made a day of it.  It took us three hours to drive down, we visited several museums around London, went out for lunch, and finally got to the cinema.

It was an amazing experience, there were several celebrity fans (I remember Caron Keating from Blue Peter), and finally the film!  My brother and I came out telling our parents how superb the film was, and we'd got to see it 2 months before the rest of the country.

That was pretty much 2 months of me saying how superb it was!  Finally it was released, and I took some friends to go to see it at Sheffield University.  I came out feeling a bit embarrassed - it didn't feel like the same film at all.

This was Star Trek V: The Final Frontier - the one directed by William Shatner where "they go to meet God".  It's universally regarded as the worst film in the series ...

How did I get it so wrong?  Well I'd not just gone to see a film - it was the anticipation, the investment, the feeling of being first, and the whole experience of the day.  In the scheme of all that, whether the film was good or not didn't really matter.  You were in a receptive mood to enjoy it, and so you just went with the flow.  [It's no coincidence that film studios also try to wine, dine, entertain critics prior to screening for a similar psychologically coercive effect]

I've seen a similar effect around Star Wars The Phantom Menace - the first of the prequel Star Wars movies.  It was released in the US 3 months before the UK.  Many friends and fans took holidays just to go to America to see it first, and they all said how raved about how great it was.  People who I've known for a while and subsequently rewatched it.  Who now refuse to buy it on DVD and are the ones who say constantly on Facebook how let down / poor the prequels were.

For both myself and those Phantom Menace fans, what happened is we simply watched it a good time "after the hype", when those effects of group delusion and denialism have deteriorated somewhat.  When we rewatch again a few months/years afterward, those phenomenon are not as prevalent, and hence we're viewing for the first time unclouded by those filters!

Rewatch at your peril!  As a ex-Doctor Who producer famously said of how we view (unseen) old films and episodes through rose coloured glasses, "the memory cheats".  Or nostalgia isn't what it used to be ...

Onward to Peer 102 ...

I've introduced a couple of important psychological factors, and how they affect us in everyday life.  Read this blog, and watch and observe those around you over the next few weeks.  It will all be useful experience.

Next time we'll look deeper into how these phenomenon impact us within testing, ways to be mindful, and fight against falling into the relative traps.  Those traps can be far worse than recommending Star Trek V to a group of brand new friends at University (they forgave me).

Further Reading

Your Deceptive Mind, Professor Steven Novella

• Lecture 12 - Culture and Mass Delusions Lecture
• Lecture 20 - Denialism

Questioning skills

In my last piece on memory I was talking about the importance of asking questions.  At every training course I've ever attended, we've always been told "there's no such thing as a stupid question".

Yes, of course ANY question is better than no question at all.  But consider this ... someone is getting frustrated because of a Google search or an SQL database search because they aren't getting the answers they need.  Our first port of call will always be "what are you asking about?".

In an almost yin and yang symbiosis, how you ask a question plants the seeds for the answer you'll receive.  The more I thought about my previous post, the more importance I could see in giving good thought to the questions you need to ask.

For myself, my exposure to peer conferences have made me think a lot about the questions I ask.  I was initially quite well know within Wellington for asking a lot of questions.  This included in my first fortnight being sent to a product demo and being sidelined by the CEO afterwards ... I'd asked so many questions the vendor had felt grilled, when in reality I was there to support them as part of a partner organisation.

I like to consider with questions that I only have a finite number of them, and to try and use them wisely.  I try to check my ego and remove questions which are there more to serve my ego, things like "well, I would never let myself be put into that position myself" or correct a speaker on a trivial point.

Open questions

I've also learned to use more open questions, over ones which can be answered in a quick "yes/no".

I was talking to a graduate the other week, and I very nearly asked them the question "do you know what's expected of you in your sprint team?".  I know the graduate would have answered yes regardless, and the worst thing I could have done would at that point to go "cool ... that's covered then".

Instead I asked them "tell me what you think is expected of you in your sprint team", and this allowed me to see what duties he understood well, as well as to discuss some additional point he might want to be mindful of.

Leading questions

Questions can also be really dangerous if you ask the wrong ones.  There are some questions which are phrased in such ways that people will find themselves automatically responding to them a certain way, even when they are aware it's a lie.

This happened to me last week at the physiotherapist.  I've recently injured my shoulder in the gym, although I've previously dislocated it about 10 years ago in Rugby a couple of times.  This has caused me to need therapy and even an ultrasound scan.

The scan shows some abnormalities, and one explanation is cartilage damage - but that would be most likely if I'd partially dislocated or "subluxed" my shoulder.  My physio asked me "are you sure you didn't feel the joint come out at all?".

Sitting in her office, I found myself going "well it was a few months ago ... and I can't remember properly ,,, I suppose it might have".  This is real dangerous ground when you're trying to recall, because your brain is quite suggestive, and as we've talked about, this can lead to your mind "adding this detail to your memory".

Thankfully before polluting my recollection with a new "made up memory" (see this post for how easy it is), I realised that I have dislocated and subluxed my shoulder several times in the distant past (prior to this injury) - I know intimately how it feels.  If it had happened, I'd have gone home to my wife and said "honey, I've partially dislocated my shoulder again" over "I pulled something at the gym".

Likewise, there are some questions which we're so hardwired to answer a certain way.  From my physio it's "you are still doing your recovery exercises, aren't you?".  When someone asks you a question and adds "aren't you?", it's quite a stretch to reply anything other than "yes".

I have to think very hard when she asks me that question - because I've been given a lot of exercises over the last couple of months, and I'm doing some of them, but not all of them (as some have gone out of vogue as I've recovered).  So I make a point of telling her what I'm doing - occasionally there's something I'm forgetting to do.

Expanding information with the use of the 5 whys ...

I used to work with a lady named Julia Baker at Kiwibank who would tell me that the key to being a good business analyst was asking good questions, and getting people to elaborate their answers.  I asked around some of the business analysts I work with at Datacom about their opinion on "what makes a good question", and they came back with ...

BA's don't dictate the solution, they ask open questions and let others do the talking, steering the conversation as needed.

The main questioning skill I’d use is the toddler technique (as I like to call it) or the five whys.  It’s an actual thing. - but is geared towards problem solving. It can be used for requirements gathering and analysis too though.

People often think in solutions. By asking them why they want something, and then asking why to their answer (and again and again) you’ll eventually get to the root of their requirement and can then build up to a solution. Hence the toddler part. The trick is not being annoying and asking open ended questions that get people to answer why, without you just saying why all the time.

It’ll go a little something like,

• I want a report (Why?)
• To be able to see how many sales we’ve made this month (Why?)
• So I can put it in my sales spreadsheet (Why?)
• So I can track all the sales across all months (Why?)
• So I can report yearly sales to the directors at the end of the year.

It is a pretty lame one, but the key is that the real requirement isn’t a report about just one month, it is giving a different set of data to a group of people. While we can meet that initial requirement, we provide real value by uncovering what is under that final requirement and meeting that instead. Sometimes it is widely different to the solution they came up with first.

Johanna Rothman's take

My previous blog post took about 6 weeks to write, I kept having to explore ideas, perform Google searches, ask around friends and watch Back To The Future.

As well as using it as an opportunity to ask around my team and find their opinion on the subject, I took time to talk to management consultant Johanna Rothman about it.  Her answer together with my conversation with our business analysts helped give me a push.

Here's her response ...

(When it comes to type of questions to ask)  There are these questions from which to choose:

• meta question:  questions about the situation (sometimes considered "questions about other questions")
• closed questions: data-based questions
• open questions: questions that require an extensive answer.
• rhetorical questions: questions that don’t need an answer

Here are questions I would ask myself:

1. Where and when am I?
2. What do I want to have happen?
3. Can I achieve that goal, given my current location/time?
4. If not, what should I do to see what I can achieve?

I particularly love this short ruleset, which links in with my learning of asking questions in peer conferences - is the question relevant, what would the answer give me/the audience, can it be answered given the expertise in the room?


And finally

Sadly rhetorical questions are my downfall.  I'll sometimes have a disagreement with my partner as couples often do.  When your partner is annoyed that you're not answering them, responding with "how do I answer ... you're asking me a rhetorical question".  Whilst that may be the case, it almost always leads to this - so beware, because it seems no-one likes a smart-ass.

Memory 103 - The alternate reality traveller's guidebook

There was a blinding flash of light, the world heaved and wrenched around you, as a portal closes behind you.  It worked, you've jumped reality - but which reality are you in?

You come across a stranger in the street.  What are you going to ask?

Alternate reality travellers have to be really good at asking questions to determine where they are.  Personally I always ask first about the Statue Of Liberty, as I remember an unfortunate reality where they blew it up (the maniacs).

"Ask a stranger" is the time traveller / alternate reality traveller's Google (providing that other stranger is not another traveller).  The questions usually seem bizarre and kind of dumb, but they provide vital information for a traveller in order to get their bearings.

So where am I going with this?  Remember previously we were talking about the psychology of time travel?  Especially our power to travel back through our memories and alter the past.  I also talked last time about how very fallible our memories can be - even when our recall of detail seems good.

The truth is then that we live in a mental alternate reality from each other.  Occasionally as a tester I can name that weird reality as the one where others have defined that "built means done", but I only find this by asking subtly.

I've actually been doing "academic" research into this by re-watching some sci-fi genre, especially Back To The Future.  Marty McFly when he accidentally goes back in time gets his bearings by observation and asking questions.  That unreliable time-machine the Tardis is not much better - it takes the Doctor where he's needed, but doesn't really tell him where he's landed.  He relies a lot on looking around, asking questions, and sometimes getting into trouble and captured to ask authority what's going on.

There's no doubt about it - questioning is the key to bridging that gulf, it's the key to getting information you need about what's going on internally within people.  More important "how they remember/perceive history".

At this point I thought this blog post was going well, however I noticed a little problem here.  I've been working a lot in facilitation this year at work, and as discussed previously this has primarily been about giving others room to express themselves.  Find out how they view testing, and try to shepherd and nurture their view.

But as important as this is, it's also important to note that to recap my previous statement that questioning is the "alternate reality traveller's Google (providing that other stranger is not another traveller)".  But of course that's exactly what other people are - fellow alternate reality travellers.  So in many ways your version of history has a validity which also needs to be shared!

And so as important as questioning is an important part of finding out how another person viewed a previous event.  It's important to follow that up with your version, to make them know there's a grey area.  If you find your viewpoints conflict, then you can talk about it.

A classic on which can come in from a meeting a couple of weeks ago, which everyone only vaguely remembers,

Padma:  Mei you said at the meeting last week you wold fix the problem with the Google account, is it done?
Mei:  No, I said we couldn't reproduce it, but I would try again to see if I could reproduce it - but only after I've finished the high level defects assigned to me.

So obviously there was a meeting a while ago, where this problem was discussed.  Whatever was discussed, Padma has gone away thinking "Mei promised she'd fix it" whilst Mei left thinking "Padma now knows I have other issues that need fixing first before I can even look into this".  Oops.

If human memory is so fallable, why don't we use an audit trails all the time?

This is a good point - writing things down is a really good way to get an auditable trail of consensus and agreement.  The problems is when people try to use it as a communication method - even email is a very slow way to hold a conversation.  It's quite possible to have a conversation which takes place over a month!

This is why we have meetings.  I know it can seem we just have meetings to have meetings sometimes, but the aim of having a meeting typically is as follows

• get everyone involved in an issue into a room to talk.  Include people with knowledge of the area.
• share information and analysis about the issue.
• agree a plan of action to proceed

A key thing as a facilitator is to give people room to talk.  But also when a room either goes quiet or too impassioned, to stop and do a bit of a summary of what's happening.  Always driving towards that "agree action plan" once we have sufficient information in the room, and a consensus is being reached.

Of everything's that discussed, it's this action plan which maybe need to be written down, with the owner.  I always make a point of noting down such actions, repeating it back to the room at the time it's agreed, restating all the actions at an end of a meeting, and finally emailing it to all (with names of who's doing what).  The power of reminding people several times!

In the case of Padme and Mei, at the end of the meeting, maybe an action like,

Google account problem.  Mei has been unable to reproduce it, and will take another look once she's completed other higher priority tasks.  She will update us if it's not possible to take a look at it within by close of this week [Action: Mei]

Even so, the written word itself isn't infallible

As a kid, I always remember Han shooting first in that Star Wars scene with Greedo.  Rewatching it, I'm obviously mistaken!

I attend a lot of peer conferences, and I've been playing around recently with "playing secretary" and keeping notes so I can revisit what was said and explored.

After a recent conference, one of the speakers took to Twitter complaining that they felt personally attacked when one of the questioners called them a moron.  Intrigued I consulted my notes to see what I'd written for the exchange, which was,

"Question: (sounds like) less a team of specialists, but a team of “not my job” morons with no ownership"

So an open and shut case of the speaker making a big deal about nothing?  Maybe ... and maybe not.  It's interesting, because from my memory I actually seem to think the line was "less a team of agilists, more a team of high performing morons".  But I have something different written down, because of course I wasn't word for word copying what people said, but paraphrasing as I went.  Paraphrasing, the whole issues of the holes in our memory we've been exploring - it's all enough for there to make subtle difference in what we remember and what was said.

I also know from giving experience reports - a good experience report is honest, open, and really reveals a level of vulnerability in the speaker.  As such I've had questions asked of me where I felt the questioner was trying to be a bit "high and mighty and judgemental" to score cheap points, which I didn't appreciate.

Though I provided my notes to both the speaker and questioner in that incident, initially they had to resolve it just by talking through what happened, and how it'd made them feel.  Each was really living in an alternate reality where they saw that incident slightly differently.  Only through talking and reaching consensus were they really able to put it into each of their contexts, leave it in the past and move forward.

And so it is with all us alternate reality travellers ...

Coming next...

We've now covered why questioning, discussion and sometimes statements allow us to piece together our past.  But what makes a good question?  That we will explore a little more next time ...

Memory 102: The memory game

Today I'm going to build on my previous article looking into how memory works, exploring some features that you might never have noticed before.  In this article, I really want you to play and explore along with me, and do some of the work yourself.  This is how you'll really get the most out of it!

What I want you to do, is to think back about ten years - to something you used to do quite regularly with a loved one, who possibly is no longer with us.  This event needs to be something you haven't done for a while.  It can be visiting a relative, training, anything.  But it needs to have been a semi-routine.

Last time  I talked about one such occurrence for me - going downstairs for breakfast when I used to visit my in-laws.  But today, I'm going to explore another set of memories ...

I've talked before about my grandmother who died last year.  I have such fond memories of visiting her and my grandad in Stoke, where we'd stay the weekend.

We'd arrive in the afternoon, and she'd be waiting for us, usually dinner was being prepared, and there was a smell of onions and carrots being chopped.  She'd always have plenty of cardboard and tape ready for me, and I'd get busy doing arts and crafts - building from my imagination.  The scissors were kept in the third drawer down next to the living room.  I remember the smooth way the drawer moved on it's runners when opened.

The scissors were heavy and black, made of cast iron, feeling cold in my hand.  They were well used, and flecks of paint were missing all over them.  The rivet which held them together was a little loose, but they worked well enough.  The sellotape was always a pain to find the end, and I can vividly remember the taste of it occasionally when I had to bite it.

As dinner got closer, I remember the hiss-hiss-hiss of the pressure cooker, the rush to tidy the kitchen, help grandad to bring in and set the table, and get the mats and cutlery in place.

In preparing this article, I've been revisiting this set of memories over the last few weeks and putting it under scrutiny.  It's an important one for me.  I suspect you have something very similar about your grandparents.

It seems pretty accurate because of the sheer level of detail in there.  But there's a huge problem - despite what I said, this is not a set of memories.  When I look at it critically and explore it, especially trying to find out different times I visited, I realise I don't have multiple memories - it's a SINGLE memory, a bizarre amalgamation of all my visits.  I've found the same looking at my fond memory of coming down for breakfast and talking to my father-in-law when we used to stay with him.

I suspect your chosen event is the same.  You need to look at the memory yourself critically.  Yes, you can remember an occurrence.  Here's the litmus test - try and separate out in your mind different occurrences of the same situation.

For my own memory, it's like a little helper goes "well if it was winter, I remember grandad coming in with a scuttle of coal and a scrap of worn carpet to stoke the fire ... but in summer we'd sit outside, and we'd got to the end of the garden where some of the trees were named after us".

Here's the thing.  You remember the core event of going to your grandparents (or whatever your chosen event was), and you remember some of the different things that happened in different years or in different seasons.  But your mind creates memories to order stitching together the generic memory of "when you used to visit" with a different aspect you remembered "the Christmas I had the Twikki robot".

Generally for a lot of regular events that we do frequently, we don't use up space remembering them.  We create an archetypal memory which is really a "drag and drop template" of "going to grandma's".  We then fill it with all the details of things we do remember, but we also fill in the gaps when we can't remember something.

I know this, because as detailed as my memory is, there are huge problems with the memories when I think critically about them.  During my time growing up with my grandparents they had the kitchen redone, including moving the scissor drawer.  But that's not represented in the memory at all - it's always by the living room.

And it's always the more recently renovated kitchen that I remember (with those smooth running cupboard drawers).  I remember their earlier kitchen unit was given to us, and we used it as a television stand/desk in my bedroom.  But I have no memory of that unit in their house, although I have memories that correlate to that period..

Likewise - my grandfather is absent from most of the memory that I've shared so far until he sets up the table, and for good reason.  Up until 1982, he worked at Chatterley Whitfield mine, and would appear part way through the afternoon.  But the mine was shut in that year, and he took retirement.

If think the year is pre-1982, then he's not present until later in the afternoon, appearing in his pit gear part way through the day.  And (I kid you not) he appears in the memory with his face all dirty, carrying a hard helmet and pit lamp.  Despite those lamps being given in at the pit for recharging.  However if I know the memory is after 1982, then "bing" like a genie of the lamp, he's present from the moment we arrive.

This just allows us to look a little more at how memory works - and those weird little bugs, which have always been there right under your nose!  We remember archetypal templates of events.  And we remember some of the differences - typically clarified like an episode of Friends as "the one where ...".  Then we just mix and match to bake "complete detailed memories".  The problem is, they might not be as accurate as we believe.

So far, we've old been talking about long term reminiscing.  Surely all this is harmless and inconsequential - surely this doesn't impact us at all at work?

How many meetings have you had this year so far?  Looking at how we've uncovered how we bulk together a lot of similar events under a single template.  How's your memory of individual meetings going?  Do you actually remember them?  Or are you mix and matching your archetypal templates with occasional "the time where ..." incidents (which dwell on the differences) sewn in?

How reliable is your memory for guiding you through all this?  Does it correlate with everyone else?  This should be a pretty scary realisation.  But don't worry, we'll explore this more next time.

Further reading ...

If you're interested in exploring more about this before next time, I recommend,

• Your Deceptive Mind, Professor Steven Novella - particularly Chapter 4 which explores "Flaws and fabrications of memory".  This is an excellent resource, and has inspired a lot of recent posts.
• Memory Is Fiction, Jonah Lehrer
• The Fallibility Of Memory, Craig Good
• This Imperfect Memory, Albert Gareev