A year of blogging and for what?

I'm coming up to a year of blogging  on testing, and it's interesting to reflect back.

I first came up with the idea of this blog really as a place to collect some useful materials.  I've always worked in my companies as a mentor, and imagined this place as a mentoring hub for anywhere I'd work in future, rather than my hard work being locked into the company intranet of somewhere I'd long left.

But as 2011 turned, I moved out of just testing and into test management in my new role.  My new company offered a lot of challenges.  Sometimes I'm frustrated that “we're not there yet”, but we're making progress, all-be-it slower than I'd hoped.

Work has presented me with a few frustrating moments and dead ends.  I've found it useful from tough weeks to sit in front of my machine, and talk about what's gone wrong – but try and stay positive, and talk about how to make things right.

I think an article I personally go back to a lot, is the Kobayashi Maru of Office Relationships.  It was a very tense time at work, and somehow talking about it and breaking it down helped.  I've worked really hard since to build bridges with Pauline, and I'm hoping we have a much better relationship.  Ironically through the bridge building process I've found out just how much pressure she was in, and she's admitted to not always handling it that well.  What could have become a bitter office feud is turning slowly into two mates trying to watch each other's backs – relationships take some building.

Another stand out article I've written has been The Agile Haka, about getting teams to work better, which got published in The Testing Circus, and my General Manager read this month and gave it his seal of approval.  And finally Those Darn Test Estimates, which is more identifying when testing overruns – I refer to those root causes regularly with my Project Managers now.

Those were my favourites, though ironically not the most popular of my posts.  You can never second-guess your readers.

I thought I was going to write this blog to teach.  In fact I found every time I sat down to write, I actually learned.  Just sitting down and thinking of an aspect of work I was unhappy with helped me look at the problem, break it apart, and look for answers.

What else did you expect?  I am a tester after all ...

Why is it so important to be able to socialise after work?

We had a great night last week after work. A departmental meal and karaoke night had been organised for the Friday.  Everyone talked with each other, and often with someone new during the meal. We enjoyed a few drinks, and we got really into the spirit of things becoming quite competitive on the microphone.

It's important to like the people you work with (although I'm usually better friends with people outside of work). But it's important to do these events once in a while because there's an important though subliminal message within these things as long as it's all fun and games.

The office is a place of tension. Project managers want one thing, business owners another, testers and developers something even more different. I discussed a little about office tensions in the Kobayashi Maru ofOffice Relationships – I've worked many different projects in 15 years in the IT business, and it's a similar story everywhere.

But in a social environment, when you get on and even have fun with others, there's an important thing you come away with. No matter what the strains of the 9-til-5, you can get on with each other. Yes it's going to be tense at times and sometimes emotional. But it's nothing personal.  And that's something important to come away with.

The Rugby World Cup – learning to celebrate

The end of an amazing road for New Zealand – hosting the 2011 Rugby World Cup and of course also winning it.

Behind the Rugby World Cup has been an army of volunteers who've given up their time for something they had passion for.  I was among them.

A year ago in September 2010 they asked for volunteers, and I put my name into the ring.  Expecting something glamourous like marshalling at the odd free game, or meeting a celebrity.

In actual fact my volunteering meant I had an office job, working with IT creating photo passes.  As it would turn out, a portion of this has been my actual day job.  So not really glamourous.

But is that a surprise?  When work needs to be done, it needs to be done.  Sometimes it's unglamourous, sometimes dull, sometimes tiring.  But it needs to be done.

The best part after the win by the All Blacks was the victory parade in Wellington.  As volunteers, we got to wear our uniform and walk behind our team through the streets.  It was an amazing experience.

Oddly enough we've been reflecting at work about projects, and how we end the.  Invariably projects complete a little later than liked, and we scurry from one to the next.  We lose our ability to celebrate success in any form.

All told my volunteer work for the World Cup added up to about 7 days.  And yet we walked through the streets with a bit of fanfare, we got a mention of thanks from the Prime Minister himself.

And yet I put many more hours and effort into my work and projects.  But do we find the time to have our own celebration?  Perhaps it's time we did.

[Look very carefully and you'll see yours truly in the volunteer army behind]

Dennis Ritchie dies

Last week Steve Jobs died, and consumers and world leaders stopped to pay tribute.

This week Dennis Ritchie died … and many outside the world of computing will go “who?”.

He was a computer scientist, who not only created the C programming language, but also helped to create the UNIX operating system in the 60s and 70s.

If like me you have ever done programming, you've no doubt done it in C – it's a hugely popular language, and incredibly powerful and flexible one. It's spawned and influenced many of our current languages like C++, C#, Perl and Java. If you've programmed in C, then you've owned C The Programming Language by Kernighan and Ritchie.

The same can be said for the impact of UNIX, itself written in C.

The technologies have made pretty much every type of device and gadget you can imagine today possible, it's used to program mobile phones to XBoxes. You cannot imagine the world of software today without them.

Quite rightly it's been said that the world of computing has lost a titan.

Those darn test estimates …

How long is a piece of string?  I'm tempted to be a wise-ass and say that to project managers when they ask me “how long will it take to test my project?”.

That's actually unfair, experience gives us as testers an idea, based on similar projects, of how long it'll take to test.  But one of the problems is, testing is an activity which has a complex relationship with other factors in a project – we can keep testing and testing, but if development don't start fixing some bugs, we're going to be here forever!

So yes, we can look through the designed features and estimate how long it will take to script, and how long to execute those scripts.

But how long until the product is finished testing?

How long is a piece of string?

Having worked in a test consultancy, there is no doubt about the importance of estimates.  They need, when a project manager looks at them, to be attractive but also realistic, with some contingency.

What we introduced was a list of estimates for testing tasks for “best case”, “probable case” and “worst case”.


                BEST   PROB   WORST
Test Plan         1 2 4
Test Conditions   2      3      5
Test Scripting    4      7     10
Pre-Testing       2      4      6
UAT Execution     4      5      8
Retesting         2      5     10

This gives the test manager some leeway, usually if things go okay it should follow the Probable estimates.  If they book the Best case estimates, be very worried.

What I'm finding is my project managers are taking my estimates, adding the Probable case figures together and times it by an hourly rate to get a budget. I don't know why I'm so surprised ... makes sense, but I'm used to working against time and not $$$.

Unfortunately at the end of the last two projects we've been considerably over that budget.  There seems to be several factors at play which determine which of those estimate paths our testing is going to follow, and it's important to understand and recognise them.

Software Delivered Late

You book in a test contractor to help you test for 6 weeks.  They arrive on week 24 to start analysis and scripting, with some test execution happening in week 26 for 4 weeks.

Then your chief developer tells you there's going to be a 2 week delay getting the build together, it won't be available until week 30 now.

You've made a commitment to your test contractor, and are so obliged to pay him, and possibly find them other work.  If you can't get them to assist elsewhere, then by week 30, you're 4 weeks in and not testing yet.  You've blown over half your budget, and Lord help you if there's any more delays!

We all know developers can often deliver late.  Late software is going to burn up budget.  You need to work with your Project Manager to make them aware of their duty to get software to you as scheduled in order for your budget to be met.

Software Delivered Is Of Poor Quality

Kind of the flip side of late delivered software.  Your vendor has promised that the software delivered has been unit and system tested, and no bugs were found.

You wrote your test plan for acceptance testing, expecting the software to have been extensively tested beforehand, with a certain level of quality.  Your project manager and you are expecting what's delivered to be a candidate for release.  You turn it on, and immediately notice a dozen problems, not able to finish basic use cases.

The developers under duress delivered what they had available to schedule instead of flagging any delays.  Little if any testing has happened, and basic bugs are being discovered only now.  Testing 101 says "more bugs = more fixes = more builds = more retesting".

One thing I try and do with vendors is ask for a release note and end report for testing, detailing what defects were found and what were fixed.  This is a bit of a game of bluff.  If I receive an end report which says “everything was tested, and no defects were raised” I get suspicious.  Very suspicious.

I've also had vendors on conference calls inform me “we're running a build up now, you'll have the install delivered in an hour”.  I pull my project manager to one side when this happens and warn them that maybe that will mean no testing whatsoever has been done …

The Delivery Chain

If you have developers on-site who you can give defects to, they fix, build, test it's possible to get a build almost every day.

If they're off site, only receive defects daily, have to courier builds, you'll be hard pressed to get a build weekly.

If you have two weeks to test, and have a daily build, you'll have 10 opportunities to get it right.

If you have weekly builds it's not likely to happen.  Your second build will have to be perfect – and it usually takes about 3-4 even with an initially high quality piece of software (there are always tweaks needed).

Time erosion

It's so easy to happen.  You have,

• a daily half hour team meeting
• a one hour weekly project progress meeting
• a one hour weekly project technical meeting
• a daily 15 minute end-of-day defect wrap up meeting
• each day you spend half an hour writing a progress report for the concerned business owner

Oh you're giggling there, but we've all been there.  Did you add it all up?  Yes, you're losing about a day a week.  Look at your estimates, did you plan on there being so much leakage?

I'm finding we're increasingly working on projects where there are a large amount of meetings to keep track of progress.  This needn't be a bad thing, and small meetings daily can help set the direction and key priorities of the day/week.  But it's easy for reporting to actually delay any progress being made, and become a sizable and unknown overhead in itself.

And some projects need test management – how do you budget for that?  It's not a solid “task” again more an ongoing overhead.

Requirements?

Requirements?  We didn't have time to write down everything we asked for!

Due to constraints a project has been only broadly defined, but you're required to perform specific testing against it, to a limited timeframe.  Oh and business analysts are too busy to answer your questions, so just get on with it and you know, test!

This is a nightmare position to be in.  You press a button, a message is displayed.  But you have no idea if it's the right message or not.  There are some things you can do – you can check the application didn't die when you pressed the button, and the message made sense in the context of the button.

But if you have vague requirements, you can only vaguely test.  Such projects really feel like they're setting up the test project to fail.  And take the blame.

Another variant of this is you raise about 10 defects against requirements, there's a review, and a business analyst says “oh yes these aren't defects, I asked for these changes by phone from our vendor”.  If things aren't documented, how can anyone keep track of these changes?

Take it easy.  Take it nice and slow.  That's no way to go.  Does your PM know?

Thankfully there's usually a place for these factors in a test plan under risks and assumptions.  But I can't emphasise enough to you the importance to talking them over time and again with your project managers before you embark on any test estimates, so they can understand and more effectively evaluate the risks and the impact to budget.

Testing's Men in Black

At a time when the world is watching the Rugby World Cup in awe of a certain set of men in black, it's interesting to see how this story has been doing the rounds on the internet – thank's to the Testing Club's Rob Lambert

http://www.t3.org/tangledwebs/07/tw0706.html

It's a no doubt apocryphal urban legend about IBM - but a lot of fun to read anyway!  In the 1960s the world of computing had a different emphasis – programming was a much slower business, and there was one shot at delivering software, no patching, it had to be right on release.

IBM supposedly found that programmers who wrote the code were blind to any faults in their software when it came to testing.  Some people though showed a natural aptitude, and thus one of the world's first test teams - “the Black Team” - came into being.

The Black Team were made up of the best-of-the-best when it came to breaking software.  They became a kind of bogey-man to terrify young developers, able to break any software they came across.  This tale here is just a brilliant parable of the supposed lengths they'd go to in order to test software ...

http://www.penzba.co.uk/GreybeardStories/TheBlackTeam.html

Tales of the Black Team go further, telling how team members started to form an identity together, wearing all-black to the white collar IBM offices.  Some even growing Dali-esque moustaches they would twirl sinisterly as they tested.

I'm very dubious – but I absolutely love how software testing, which feels sometimes like a very recent discipline (in New Zealand sometimes it feels not quite respected as a profession at all some days), has managed to pick up this urban legend.

But it also takes me back to one of my first posts here, on “what is software testing”.  The tale of the Black Team is all about a team who go out of their way to break things.  The story where they work out the resonant frequency of a large tape reader so it rocks itself over whilst reading a file is bang-on-the-money for a lot of people who see testers as people who just go out of their way to destroy.

Today in a management meeting it came out that the business owners see testing as a problem.  “The project was all going well until testing got involved”.  As if testers are responsible for the defects they encountered.  I think the reality is much closer to “we managed to delude ourselves that everything was fine until testing gave us a wake up call”.  If testing is done well there's no hiding the truth of where a project lies.

But no- testing is not about breaking things.  It's about proving quality.  And that can be a bogey man of it's own to a complacent project.

Steve Jobs - a legacy of quality ...

Steve Jobs has died today at the age of 56 …

He leaves behind him a successful legacy, with Apple computers now the most lucritive IT company in the world, having ridden out the financial recession where other companies have stumbled.

But is the financial ledger at Apple his only legacy?

I first heard about Apple with a passion back in 2003.  Lots of people in my C&C project were getting Macs.  It was explained to me by one of our developers “you know how on Windows, you try and install software, need to put an update on, has to reboot, won't work, then the update fails and trashes your installation?  On a Mac it just works”.

That seems to be part of the Steve Jobs alchemy – in a world where we're used to software not working first time, where we know we're going to get some quirks until there are a few updates, Apple seemed capable of making products that seemed virtually faultless.  In fact, it would become big news if any issues were found.

Here are what I think are several traits which have made Apple unique,

• They don't work to release deadlines.  They release a product when they're satisfied with it.  Only this week we had complaints that the iPhone 4 was 18 months old, and wasn't it time for a new one to be released?  Apple work to their own timetimes, releasing products when they're ready.
• Diversity.  Or maybe lack of it.  Nokia makes many brands of phone, a new one each month.  Apple only make one phone – the one you want.
• Simplicity.  Apple products are designed to be accessable by everyone.  Hence the uptake of their products includes people outside the normal “gadget freak” band.  So called “silver surfers” (older computer users) find Apple intuitive where Windows leaves them baffled.
• Quality.  As my developer so rightly put it, Apple products “just seem to work”  [It looks easy, but incredibly difficult to achieve]

It's ironic but many companies are obviously trying to chip away at Apple's lead.  They are envious of it's strong position, and hungry to take some of the market, so

• They rush out a product to be first to market.
• They cram their product with as many features as possible, hoping to beat Apple on specification.

But it never works out for them, because in their gold rush to feature add, their product suffers in the quality realm, and word gets out it's a stinker.

We all want a piece of the Apple pie.  I sit now on meetings which talk endlessly about the important of “customer experience”, the Steve Jobs buzz-word.  But when marketing types are talking up the planned “customer experience” they're not trying hard enough to bake quality into their products.  That's not taking on Apple, that's setting yourself up to lose your customers to Apple.

Today Steve Jobs was described as a contemporary Leonardo da Vinci.  I don't think that's right – as his genius wasn't so much in technology as in business.  I described him at work at a modern Henry Ford of computing – who also made products only in black.

But tonight, I think weighing things up, he was perhaps a William Denning of the modern computer age, a champion of quality, although of course, quality with a price tag.