A sample exit report. With no numbers!

Okay - so last time I wrote about an obsession with numbers in testing.

Pretty tough talk Mike ... but in reality?  Normally due to confidentiality I can't share the reports that I normally produce.  However recently we did at Datacom the Oceania Software Testing World Cup.

It was a really great experience, as it allowed several of us testers who don't normally work together to be in a team and learn how different parts of the company test.

We had 3 hours to put an application through it's paces, record defects, and produce an end report on it.

Below is a copy of the report we produced (I've checked the rules, and it doesn't say we can't share this).  You'll notice we link to the results in our defect management system, but I don't produce a list of them all, or even count them.

Instead I try and weave a story of what the main drivers of the project are, the testing that we've covered, and the major issues we found.  You'll notice as well that I use those issues to give the advice that I feel that,

• It could be used as is to demonstrate the look and feel
• It really needs some more connected features to actually be useful and usable.  A major driver from the customer.

Our team name was Quest Aotearoa [QA].  Aotearoa is the Maori word for New Zealand.

Team Quest Aotearoa - Roadmap Test Report

Quest Aotearoa has undertaken to do initial testing of the roadmap application on Android and iPhone.

Business travel is a fraught affair of being lost, trying to manage your appointments, flights, hotels.  Roadmap is designed to be a connected application which allows you to manage and monitor all of this from a single location.  This is the competitive advantage of this application.

Testing has been undertaken on September 8th in Wellington from 7pm to 10pm local time.  Our test machines are,

• LG G3 Android 5.0
• Samsung Galaxy Note 3 Android 4.4
• Samsung A4 Android 4.4
• iPhone iOS 9.1

During this time, Wellington has undergone some severe weather warnings, which has caused disruptions to local flights – something we have used to test.

Testing Scope

As a team we have explored the functionality of the application.  This has included,

• Registration on a number of emails
• Transfering the same registration to different devices
• Logging out of the system, then returning to the application
• Contacting service desk
• Using tutorials
• Using meeting calendar
• Adding flights – both short term and in distant future, together with in the past
• Add hotels – including obscure locations, in the past, leaving before arriving etc
• Mixed items on the timeline, to confirm ordering is as expected
• Removing items from the timeline
• Exploring our location using maps
• Giving feedback
• Using the application after shutdown
• Retrieve your details whilst the phone is in airplane mode

Overview of the application

Issues we have found can be seen under our Lean Testing account.  [This was our bug tracker]

The application as a demonstrator of the aesthetics works really well, and gives users a good idea of how the final product will work.

The display to the user is generally uncluttered, with colour coded systems making it clear the difference between hotels, flights etc.  The application looks consistent across iPhone and numerous Android devices.

However, even as a beta version of software, this application is not yet ready for release.  Mainly because it fails in a number of key ways to deliver reliably on the competitive advantage promised.

Key examples of this include,

• Registrations on Hotmail accounts took 40 minutes to get to the user.  For many new apps, you have about a 5 minute window to impress users before they typically uninstall and move on
• The application lacks key connectivity – for instance you cannot import items from a pre-existing calendar, send a location to Uber, find a local taxi firm etc.  As mentioned we had a severe weather warning, and received warnings from other apps, but not from roadmap
• Local flight NZ463 was looked up on Wellington Airport, and was clearly seen to be late (on the airport site), but the app didn’t tell us this.  This was a very clear failure to deliver one of the key claims of the application
• Roadmap collects together a lot of very sensitive information into one location.  However even when we signed out, we didn’t require a login to get back into that information.
• We had occasional crashes – which are detailed best in Lean Testing

It is our recommendation that this application could be used for basic demonstrations to show the look and feel, but right now need the features discussed addressed before we have a functional MVP which reliably delivers on it’s promises.

Now playing:  Robin Schulz, Sugar

By the way, I thoroughly recommend the Software World Cup.  It's such a great opportunity to reconnect with why you love being a tester.