Developing an approach to responsive design website testing

Since it's introduction in 2011, responsive design has become increasingly important in websites, and with good reason.  Prior to it's use, websites struggled to support the increasing uptake of access from smartphones and tablets.  Some sites would have to develop separate (and thus costly) websites - one for viewing from PCs, and one for viewing for mobiles (typically hosted at m.*).

Responsive design was a methodology of having the same page source, which would scale according to the size of page available - allowing a single source of web content to be supported on difference sizes and devices - from laptop to smartphone.  It basically allows all your information to "fit" horizontally, so if you're on a mobile device, you don't keep having to resize or scroll your screen horizontally <->.

There is a wonderful website you can try this out on here.

Open in full page on your browser, and it should look a little like this ...

Now reduce the size of your browser a little, and you should find that instead of two columns of entries, it reduces to a single column ....

Now take it down even smaller, as if you have the limited number of pixels you get from a mobile, now the labels go above the data fields ...

Pretty cool huh?  But there are also a few potential pitfalls, and this article will talk you through some of them.

Whoo-hoo, mobile testing, I'll have me some of that!

Something we need to be crystal clear about, when we're talking about testing responsive design on mobile devices, we're basically just using them as browsers.  This isn't connected with testing mobile applications which can be installed and work through Google Play or the Apple's App Store, that's a whole other different field of testing (but some people get confused).

Creating a strategy

To get the ball rolling, you need to start setting a strategy - across mobile devices and browsers.  Responsive design uses some newer HTML features, so there are older phones and browsers which really struggle.  So the question has to be - what browsers/devices matter?

When we've done browser testing in the past, we've just tended to install a whole host of browsers onto our machine, and maybe some virtual machines to cover older versions of IE, and just "get started".  Here's the catch though - it's free to install a browser (yes, even Safari).  But mobile devices cost - and if we're talking a high end model, then it costs a lot!  You have to be selective, and your client has to be willing to support the purchase of these devices.

Even "hey guys, does anyone have a phone, I just want you to check this site in testing" is a bit dubious.  You're running your testing strategy from "what you can find around the office".  The other thing is that a mobile phone is a very personal thing - I might check a site for you, but I wouldn't let you take away my phone to look at a problem I'd encountered.

If your client is keen on developing a responsive design site, then they need to be comfortable with renting or at least purchasing some devices.  And here's the thing,

• Going forward, you will have to do some responsive design checking for every release.  It's not just something you bolt on, work for a few week and don't have to worry about anymore.
• New devices are always being released.  This means revising those devices you use about every 6-12 months (a good rule of thumb is every time there is a new version of iOS).

Which devices?

The best answer to this is to talk to your client and ask "what browsers/devices have 5% of traffic or more".

Of course if you are just creating a responsive design, you might not be able to have reliable figures.  In this case there are lots of sources out there.  Mobile test consultancy Synapse provide some useful resources (and I've used Jae before, and he's well worth bringing in).


Apple devices - you can find information about the most popular here.

Android devices - you can find information about the most popular here.

Right now, the jury is still out on Windows Phone 8.1 uptake, as is being able to cross compare device usage (iOS vs Android vs Windows 8.1).

Looking through that list, I'd say at a bare minimum, you'd need to consider the following in your test suite,

• iOS 8.X device
• iOS 7.X device
• KitKat device
• JellyBean device

For these devices, I'd also try and consider lower spec models, especially with smaller screen resolution.  [In responsive design, smaller screen means less real estate, and potential for problems.  As testers we like problems]  That can often mean looking at a smartphone over a tablet.

Beyond that, I'd try and see it I could talk up a purchase of something in Lollipop (it's a low share, but it's the future), and maybe Windows 8.1 (especially as there are some dirt cheap Windows Phones out there right now).

Regarding the browsers on those devices - most people just use the build in browser (until analytics tell you otherwise).

Remember - this is my analysis from the current market - it will change!  Once your site is up, try and get analytics to help profile popular browsers/devices, after all it doesn't matter what other people are using, what matters is what your client's customers are using.

And on that bombshell, just a few weeks after Microsoft announced the death of IE, look who sits at the top of the most popular browsers for reading this blog?

Test Website Access

Well, typically the website you're producing is kept tightly under wraps until it's launched.  Don't forget to have a discussion about that with your web team.  Do you have a VPN you can set up on devices to access your test environment?  You're going to need some form of solution to get your device to see your test pages.

Can't we just get around needing mobile devices, and just use the browser in small mode?

If you make your browser 480 x 800 - isn't that the same as using a browser?

It's to be fair a good first step, but as you'll see below, some of the problems come from the operating system.  Android and iOS have special build in ways to handle certain items like drop down and pop-ups which mean they behave slightly unexpectedly.

So I'm set up - what next?

Okay, so someone approved your devices, you have an accessible test area and you're ready to go ... and?

So what exactly are you going to test?  What it really helps to do now is come up with a way to summarise your application.  To cross browser and cross device test you need to repeat the same elements of testing over and again for every browser and device.

Do you have any clear idea what those elements are for your system?

The following is a guide to my approach ... and remember I looked at some of this a couple of years ago (but just for browsers).

Page checking
Create a map of every page.  Confirm that,

• can every field can be selected and data entered?
• can every check box selected/unselected?
• can every drop down box selected?
• can every button be selected?
• can you select between tabbed pages?
• check for pop up confirmation and error messages

You are basically looking here for something not working, a button missing etc which would prevent you from being able to use a page!

Functional Checking
What are the main basic functions of the website, I need to make a list of them, and repeat on several browsers and devices.

Here's some examples (discussed previously),

• Registration
• Login
• Change my account details

Generally the tests per browser/device don't have to be exhaustive - you are repeating a few examples across multiple browsers after all.  But ideally should include a success and a failure (you always want to check an error message is displayed).

Being able to create a summary overview for testing of your website, is something I hope to explore in the future - so watch this space.

Common Gotcha's

These are the areas I've commonly found problems with on responsive design.  You'll notice that some of these can be countered by good, up-front design ... so if as a tester you find yourself in a design meeting, go in forearmed ...

Landscape to portrait to landscape

A simple test - part fill your page in, turn it on it's side.  Do you lose your data?  Does any page element vanish?

Turn it back to it's original orientation, and check again.   Sometimes the changed orientation causes a page refresh, and things go missing!

Drop downs

The mobile device overrides how drop downs are handled in browsers ...

On the left is the iOS carousel, and on the right the Android selection list.

The problem though occurs if you have long selectable items on your drop downs, For example, consider a list of security questions,

• What was the name of your first girlfriend/boyfriend?
• What was the name of your favourite film?
• What was the name of your junior school?

All these will truncate the questions, so all the user will see is "What was the ...".  There's really no solution for this, but rethinking the approach/redesign.

Pop ups

I've found any kind of required pop ups from a browser can be a little troublesome on mobile devices (they often just go AWOL).

This can include,

• Are you sure you want to continue?
• Read these terms and conditions.
• Select a date from this calendar

Tread carefully - and always check these.

The catch with responsive design

Yeah - there was bound to be one, wasn't there?  This approach significantly reduces risk of problems in responsive websites, but it's no guarantee.  Indeed some mobile phones take a system like Android and tailor particular features, so it's hard and expensive to exhaustively test upfront.  This risk has to be known to both yourself and your client.

Furthermore, because responsive design uses newer features of HTML, it means that much older browsers and devices really don't handle the page very well.  You might want to consider that on old, out of scope browsers/devices that it at least fails gracefully with a "your browser does not support this website" error message.

Some common sense ticket items

Finally some common sense things to consider before we wrap up ...

Health and safety

Some testers are really keen to get into mobile testing.  I actually really find it a pain.  The screens are much smaller, it requires data entry just with your thumbs, you tend to hunch over the device.

This is a recipe for repetitive strain injury.  Try and mix up mobile testing with browser testing, and make sure you're taking regular breaks.

Keep them safe and keep them secret

Make sure they're locked away when not in use, though have a couple of keys to where they're kept with your team.  You just don't want them walking away.  Lock and count them in every night.

You drop it you bought it?

It's inevitable that one is going to get dropped.  What then?  Look into getting them added and listed to your building insurance.  If you can't make it clear what will happen should any damage occur.





Breaking news ...

I ran this article by Stephen Janaway - he's someone whose articles I really respect and look to when learning in the mobile space, and he has a huge amount of experience.

One of the great things about writing an article like this is that it puts down everything you know, and sometimes what you find is there's something you didn't know.  And sure enough ...

So, I'd never really though about using Chrome Developer Tools as an early form of testing - so I can't really explore here, but it might be a topic for a follow-up blog.