This article was originally produced as a series of articles for Testing Circus back in 2012 which can be viewed here. I was extremely proud of the end result, and really thankful for all the effort those who participated gave.
Apart from being published as an article, I also used it with my company at the time, to show the variety of opinions within the testing community and to support the direction I was trying to move testing in. It's an example of how the community can create a positive voice to encourage testers everywhere ...
Background
Without
doubt one of the largest challenges any tester has within their organisation
revolves around building up their testing culture and identity, engaging with
the rest of the business / project about what testing is about, and being able
to have the freedom to apply their skills so they give their company the return
they need.
Without
these things, we're seen as liabilities on projects, and potentially our hard
work not being either understood or appreciated. Worse still it potentially
leads us to perform testing tasks which are deemed important to non-testing
staff, that we feel professionally are not really adding value.
Throughout
August I went around a number of peer testers, asking them the same questions, trying
to collect ideas and opinions from testers across the industry, testing
disciplines and the world. The responses
I got back really opened my eyes to a lot of common issues and vision across
the testing community, and function as the backbone of this series of articles.
Contributors
I would
like to thank the following people for giving their time and passion to this
project, and helping this series to represent the testing community.
Adrian Howard -
'Generalising Specialist' with a strong focus on more Agile ways of doing
things. Twitter id: @adrianh
Ajay Balamurugadas - Senior QA Engineer. A man I've come to respect for his devotion
to learning and teaching testing.
Twitter id: @ajay184f
Bernice Niel Ruhland - Software Testing Manager. I regularly exchange ideas with Bernice, who
is a font of wisdom. She's also a
regular contributor to Testing Circus magazine.
Twitter id: @bruhland2000
Betty Schaar - Sr. consultant and Training
Services Lead at BenchmarkQA. Another
teacher of testers! I've come to know her through Twitter where I recogniser
her wicked sense of humour. I was really
pleased to get Betty onboard as I'd ended up talking to a lot of Agilists, and
her opinions gave a view into the non-Agile testing world, as well as worries
about Agile. Twitter id: @swQualityQueen
Elisabeth Hendrickson - "technically founder and
president of Agilistry Studio, but you can call me whatever you want. Test
Obsessed is a fine title". I think
we all know and love her as the later.
Twitter id: @testobsessed
Geoff Horne - Prinicpal Test Manager, ISQA. A man I was truly honoured to meet at the
Kiwi Workshop on Software Testing on ethics (KWST2), and I was bowled over by
his insight and experience.
Harry Bowen, Jr. - QA Contractor for Userplane. Another tester who I've met and befriended via
Twitter. Twitter id: @Passionfortest
Janet Gregory - Agile Coach and Trainer for
DragonFire Inc. She's the Canadian based
co-author with Lisa Crispin of Agile Testing: A Practical Guide for Testers and
Agile Teams. I need to stop teasing her
about the snow in Canada. Twitter id:
@janetgregoryca
Jari Mikael Laakso
- Head of QA. One of the most
passionate testers I have interacted with, who is always on Twitter trying to
engage with other testers and question, question, question. That's a good tester trait. Twitter id:
@JariLaakso
Simon Knight - Test Lead. He is also the senior editor on the Testing
Planet magazine, and technically my boss there (I'm a co-editor). Twitter id:
@sjpknight
Stephen Janaway - Senior R&D Manager in
Nokia. I have found his insights into
mobile testing essential reading. I am
gutted we used to live in the same town as Stephen in the UK (before I moved to New Zealand), and I
never got to meet him! Twitter id: @stephenjanaway
QUESTION ONE: If you were
asked to provide a testing mission statement for your organisation what would
it be? Do you currently have one? How comfortable are you about measuring up to
the expectation from one?
To start
the ball rolling, Elisabeth provided her elegant and to-the-point catch phrase,
“Empirical evidence trumps speculation. Every. Single. Time”. I like this statement a lot – as often on
projects there is a lot of hearsay and promises that a product is ready, but it
doesn't mean anything until you've actually started to try and use it, and this
is a great starting point for engaging with the rest of the project and
managing expectations.
Bernice
was rightly cautious about a mission statement that attempts a one-size-fits-all approach to testing
“I have thought about writing one but not sure what purpose it would serve. For
me I would rather put that time into assessing risk and testing approaches for
each release we are working on. Each has its own challenges that need to be
addressed. “
However
the question got many thinking, Stephen echoed many people's initial reaction,
“We don't currently have one.”, but then went on to say, “We should have one -
previous groups I have run have had one. I'm comfortable with them - without
something like a mission statement then it's not easy to get everyone behind a
common goal. I think in testing especially it's important to have a common,
shared goal, if only for motivational reasons.”
“If I had to write one now it would be something like 'Taking a pro-active approach to uncovering product risk' - that is,
it would not mention the word testing nor the the words quality or
assurance. “
Indeed,
Betty also ventured her own definition, saying that if she was to write one it
would most likely read, “To contribute to the delivery of 'good enough' software (as
defined by the software stakeholders) that serves our customer's needs
and that is supported by/with 'just
enough' useful documentation for operational needs. “
Having
considered the idea, Betty went on, “I like the idea of having an agreed
upon testing mission statement that can be used as a touchstone for making
sure our methods and actions are aligned and support these
objectives. One caveat though is that since testing is
cross-functional nowadays, this really ought to be bought-into by all groups
that do testing.” Which is a really good
point.
Like
Bernice, Jari saw the potential pitfalls of having a generic testing mission
statement, “My team is testing 4 different products. Each for the same customer,
but the testing is very different. There is no common mission for the team. I
guess this is slightly different for outsourcing companies than product
companies.” Indeed, when I was pooling
experience within my own organisation, I found very different software delivery
lifecycles, lines of supports and consequently impact of risk, which led to
slightly different perceptions of the impact of defects (and hence testing).
Simon
was another tester without one, but would hazard an opinion that it should be
something like "provide rapid feedback to developers and clients by
utilising the most effective automated and manual testing techniques
intelligently and efficiently."
Ajay
also felt that there was no single statement that “everyone is aware of” in his
organisation, but again ventured it should be about “delivering value to the
customer” by “reducing (the number of) unknown customer bugs.”. Empowered by the idea he had this to say, “I
think this testing mission should be part of every tester's goals. Therefore, I
am always working on that direction to accomplish that goal” by communicating
this to his team.
Janet
put excellently that her personal mission statement with clients was about the
need to "be curious, ask questions and collaborate with all team members
to work towards having a shared common understanding of what we are building in
the effort to prevent defects; Evaluate and test the product to find defects,
work with programmers to fix them as quickly as possible so they are not
shipped to the customer." She went
on to say there was a bunch of other things she'd probably want to add, but by
having too long a mission statement it diluted its goals.
Whilst
not a mission statement in itself, Adrian had this acknowledgement of the
importance of testing “what we do have is a passion for great products and
happy customers. Testing helps us make those happen”.
I have
encountered other excellent statements for the mission for testing have come
outside of this questionnaire. I like my
colleague Pam Forrest’s response that testing was a foremost about “ensuring a
product was fit for purpose … that riskier areas which could impact business
and customers have been evaluated within the limits of test system”. Likewise, I found myself nodding along when James Bach says “the purpose of testing is to cast light on the status of the product and its context, in the service of my clients”.
QUESTION TWO: Do you feel the attitude towards what testing
is and what it should deliver has changed during your career? What do you feel
is driving that change? Do you feel this is good or bad for testing?
Stephen
reflected back on his journey with his current company, “within Nokia testing
has become more accepted and is now seen as a critical aspect. Time is given for
testing and resources too; this was not the case when I joined when one manager
remarked to me 'can't we just find some housewives to do the testing?'.”
He
continued to warn about the potential Cassandra syndrome, where testers become
too linked with news the rest of the business is not prepared to believe, “I
believe across the industry testing is becoming more accepted although this is
mainly due to the push from the community rather than a pull from outside. We
need to be careful not to push too much or risk being ignored as the ones who
always shout perceived bad news normally are.”
Adrian
was originally a developer, and “my early encounters with testing folk were not
good. Very bureaucratic. A focus on obeying the letter of the process, rather
than the spirit. For example getting new features that were making customers
happy marked as 'defects' because they weren't in the original spec.
The testing folk 'owned' quality, and that ownership seemed to cause
problems.”
He mused
that such Testing Police who were “often very focused on checklists. If it
passed everything on the checklist it was okay - even when there were glaring
problems elsewhere”.
However
that was in the past and “I now realise that what I had encountered were bad
examples of the testing profession, but without knowledge and experience of 'good' testers I tarred the whole
profession with the brush of badness. I know from conversation with peers that
I wasn't alone in this view. Developers views were being shaped by their
encounters with bad testers and bad testing departments is still a problem I
think.”
Adrian
during his career had become involved initially in XP programming and later Test
Driven Design in Agile. It led to a real
change of understanding, “I started reading more about testing. I was fortunate
enough to work a couple of excellent testers, including those with a strong
exploratory testing focus. I saw people doing much more inclusive testing with
the team as a whole. I saw the value that having people embedded with the team
having a strong testing specialisation brought.
I saw testers as people who helped us build quality in, rather than
people who (poorly) found problems after we'd built stuff.”
Elisabeth
had a similar warning tale from history “when I started in this field there was
a trend toward separating testing and programming. That was a dark, dark time
in our industry. It led to massive amounts of waste and ironically lower
quality at a higher price. Fortunately the rise of Agile has resulted in
organizations re-integrating testing into the development cycle, and often
integrating the testers and programmers within teams. This leads to faster
feedback and a whole lot less duplicated effort, wasted time, and higher
quality.”
Janet
had this to say, “I definitely think there is a difference, but only for those
teams moving to Agile ... What I see more these days is testers trying to be
more proactive trying to collaborate with the programmers and customer's more,
worried more about delivering the right value than adhering to a given set of
requirements. However, it is a slow and hard change for many because of
the radical mindset shift.”
Not
everyone was sure that Agile was the way for all software development to go
Betty had some concerns, “I've been frustrated a bit by the whole agile
movement because of the general lack of focus on documentation (and formal
requirements). All is fine so long as the group that started the work
stays with it forever but that is not a good expectation in this day and age. I
do not advocate for documentation that isn't of value but it's no small task to
determine what is just enough/good enough documentation.”
Given
Betty's background in medical scanners, this is understandable – surely in some
environments detailed requirements and strict gateways between project phases
made sense. I had some difficulty seeing
some of my avionics programs I'd worked on being Agile over V-model.
Harry
too had felt a little burned by an Agile project, where going Agile was, in his
view, seen as an excuse for 'no
documentation'*. “Testing was made difficult without said
documentation as the requirements were never set in stone, but rather evolved
with the slide of Executive opinion, which could change at a moment's notice.
We went full circle on one of our products in 13 months.”
[* Of course Agile is about so
much more than just “no documentation”, it's closer to Betty's definition of
only having documentation which adds value]
Ajay
felt that testing has changed within his organisation, and he felt two powerful
agents for this change was through, the “reputation and credibility of the
testers” and “general awareness (they raised) through testing
war-stories”. Reminding us of our social
obligation to build and communicate our values to the rest of the organisation.
QUESTION THREE: Are there obstacles and expectations around
testing which you find challenging? How do you confront them?
A
colleague from the UK who wished to remain anonymous came forward with what
would be a common theme, “Time. Time. Time. Development slips and gets more
time. The result? Testing gets squeezed. I don't think there'll
ever be an answer to this, you just have to shift your risk analysis
parameters”
Geoff
had this to say, “testing is often seen as the poor cousin on the SDLC &
project teams often consider a project delivered once it is released
to test. Testing in New Zealand is
slowly changing however when budget and timeline are the primary
considerations, often very difficult to translate testing into benefits over
the medium to long term.”
Betty
echoed this, “one aspect I find frustrating, is that the amount of time and the amount of
resources needed for testing is so often way underestimated. We're
always having to justify our needs and often still not given the adequate
time/resources. I have been known
to use the argument that the old rules of thumb for testing resources should be
thrown out (n dev/1 tester) unless we're willing to go with double or
triple the testers to dev ratio.”
Elisabeth
felt herself time and again challenging the prejudice that testers were somehow
the bottom of the project ladder, and people who failed to make it as
developers, “people expect testing to be easy. 'It's just testing,' they say.
But good testing is difficult. It requires an understanding of what we're
building and why we're building it, what can go wrong, and how to ferret out
the bad things before they bite our customers.”
She put
the case for a better valuation of what the abilities testers need “that means
that people who test need to be skilled. They have to understand technology,
the business, analysis, and test design techniques. Testing yields information.
The quality of the information depends very much on the caliber of the person
testing.”
“However, because the product of testing is an
intangible and ephemeral thing, it can be extraordinarily difficult to tell the
difference between a skilled tester and someone who is just going through the
motions. I suppose that's why it is so tempting for so many organizations to
view testing as something that anyone can do with absolutely no training
or skill. I have been fighting this 'anyone
can test' attitude my entire career, so I doubt it will get better any time
soon.”
Repeating
our common theme, “furthermore, organizations tend to baulk if they view the
cost of testing as too high. They try to measure the cost of testing separately
from development, and minimize it. They fail to understand that testing isn't
like insurance where it makes sense to make a risk tradeoff. It's more like
scaffolding where the building just isn't going to go any higher up unless you
have it. Any attempt to minimize the cost of testing is just as likely to
increase the overall cost of quality. It's a systems thing and sometimes
difficult to help executives understand.”
Simon
echoed the feeling about testing being the poor cousin even on some Agile
project and often 'the last to know', “I guess the other thing that tends to
annoy me is not being involved in more planning/scoping/requirements gathering
type meetings. I always see these as areas where test can add value, but nobody
else ever seems to agree”
Bernice
echoed the issue of attempting to complete testing against deadline, “I think
one challenge is time. To manage time, I need to understand priorities plus
perform risk assessment before a testing cycle begins and throughout testing.
This helps identifies the scope of testing which may change as we learn more
and determine how to pull in different tester's strength into the testing.”
Elsewhere
there were several comments about the common misconception that testing is
responsible for quality (and breaking the product). Stephen spoke about challenging the idea
“that testing assures quality. It of course does not, it highlights risk, and
this is the main challenge - carrying out a role which is seen, when executed
correctly, to be raising issues while taking none of the responsibility.
Confrontation is best done through education and experience.”
Adrian
had much the same experience, remembering that “I still encounter the old
testing-department-owns-quality attitude occasionally. Changing that can be
hard since it's often embedded into a larger organisation problem of how
departments are rewarded/graded. This is
made especially hard since it then colours the rest of the organisation's views
on what the value of testing is.”
Harry too
talked about a changing role and relationship between tester and
developer. The responsibility for
quality was not a 'tester thing', it was something they were trying to
challenge, though “the idea that Developers should be responsible for the
quality of their code is a bit foreign and partly not fair to them. They
write, we test. As I pointed out in the presentation, 'Welcome to the
21st century!'. Writing code and
testing it are two different mindsets.”
But just
as developers needed to become more test driven, so too testers needed to
become more technical, “for me at Userplane, I'm testing mostly web
technologies with a sprinkling of mobile. Becoming adept at
Javascript and Selenium as our automation
solution is imperative for success. My thought processes in testing and
managing the process must now include code ability and automation. The
impact is beginning to be felt and they are (so far) very happy with the QA
mindset.”
Simon
was another tester inspired to get more involved in the technical side, “sometimes I think I'd like to try my hand at
writing some production code. I work on a small agile team and really there's
no particularly good reason why I couldn't pitch in other than division of
responsibility. I don't push it too much though because I've got plenty of
other stuff to crack on with usually. Perhaps I just need to get stuck in
rather than waiting for an invitation! “
A few
anonymous colleagues from the UK and Europe echoed a familiar story “the
management and sales teams don't understand testing”. With the perception that“testers are causing
delays on projects.”. Some of this
harked back to a comment that Elisabeth made that the team from developer to
manager need to learn to accept that “it's not done until it's tested”.
Ajay
touched a little on challenging this, also going back to touch upon on
Elisabeth's mission statement of “empirical evidence trumps speculation” by
stating that testers need to build “reputation and credibility … You slowly
build reputation by showing them the proof to start with, then you show the bug
again.” And that he has started to
record and share his testing sessions to emphasise this point.
He is
also driven to save time and effort in testing, going on to explain, “the whole
company was using test cases and then shifted to user stories but in an excel
which took an hr to go through multiple reviews - peer review, development
review, product review. Now, after my
demonstration of using mindmaps, the entire company has thrown away user
stories in excel, started using mindmaps for review, tracking and saved a lot
of time.”
Janet
asked about why we expect everything within testing to be written up, even when
other phases of the project take a lighter approach, “one of the biggest
challenges / obstacles is the need for massive amounts of documentation
required - mostly as a 'Cover Your Ass' response.
I think this stems from a blame culture of many organizations.
It is hard for testers to let go, and realize that maybe we don't need all
those metrics, and documentation.”
QUESTION FOUR: Is there anything
your team does to promote and raise the profile of testing within your
projects?
Stephan
talks about how Nokia encouraged the whole team to be involved with testing,
“we are organised as cross disciplined Agile teams and share the responsibility
for testing when the deadlines come. Each team has (specialised) testers who
will do as much testing as they can in the sprints”, however when push comes to
shove near a deadline, everyone from developers to UX design gets involved in
testing, with the specialised testers “as test advisers to
others … We use a lot of exploratory
testing (ET) and session based testing which help share knowledge and
experiences between the whole team.”
The
result worked, as “distributing (test team) members to each project, and
getting them sitting in the project teams, was the best thing I've seen in the
last 12 years for promoting and raising the profile of testing.”
Betty
echoed the importance of good relationships with the rest of the project,
“fortunately I have been able to generate good rapport with the team
and management pretty much wherever I've been and it's by
repeated project successes that I've been able to garner the
respect for my role. I speak up, point out issues in a way that
is non-threatening to individuals, I highlight where actions taken have
prevented problems, I work on developing rapport and generating
good-will. I use jocularity liberally as I've found humour to be an
invaluable tool for making the work environment fun.”
As a
test consultant Elisabeth observed that, “I don't usually need to raise the
profile of testing, but sometimes I need to help my stakeholders understand
that it cannot be an afterthought. I help them understand that features are not
"done" until they are tested, meaning both checked and explored. And
I help them understand the connection between fast feedback cycles and lowered
risk.”
Geoff
echoed much of this sentiment saying it was important to create “continuous
impression upon leadership to test early, test often, test properly, test
appropriately”.
Harry
picks up on a theme from Ajay about building credibility within your
organisation “on the previous team, we pushed for other teams to allow for us
to do their testing to save them time and resources. We developed a great
reputation for fast and reliable testing. It allowed for the company to
move Mobile to our shop, and consolidate
desktop testing“ within his team.
Ajay
said he helped to advance the culture of testing though introducing innovation
such as mind maps, as well as developing a reputation for “raising good bugs
and good bug advocacy” and engaging with stakeholders and peers about
“conducting testing talks” to convey testing values.
Bernice
talks about engaging with the rest of the organisation by “providing
information on what we are seeing in testing to the developers and business
analyst is a value-add. For example we may ask a Business Analyst if what we
are seeing is what they expected from the functionality. We try to understand
the bigger changes that are in the pipeline to allow us to provide any input
and initial risk assessment. We show that we can add value to the overall
process and not just test at the end of the process. We offer to test code in
the developers sandbox if they would like our testing input earlier. “
Janet
really sums up a lot of answers here.
The core thing being “I encourage all the teams I coach is to make
testing visible. Create testing tasks for all testing activities. Use big
visible charts such a a release test matrix to show progress, as well as extent
of testing. Test results - keep them visible on the continuous integration”.
QUESTION FIVE: What changes have you seen introduced to your
test projects which have made significant impacts? Were these changes met
with initial hostility or scepticism? How was that overcome?
Stephen
feels a significant change came within the “team structure as a result of the
change to the Agile wow. I've also introduced automation (as well as), using
testers to pair test with developers,”.
At Nokia there were “no problems with hostility”.
Stephen
then harks back to a common theme of getting testing visible, and getting
everyone involved, “as mentioned above, changing to exploratory testing and SBT
was a big change. Initial sessions used only testers; we invited everyone but
since it was optional and testing related then only one developer every showed
up. Some bribery was needed (cakes
essentially) to draw people in, and after that we got a steady number of
developers coming to the sessions. They would really add value with their
different view-points and it was very motivating to the testers who organised
the sessions to see them there.”
Betty
talks again about the technical challenges of modern systems, “at one point I
was having to test service oriented archictecture/middleware. That
was new for me and new for the organization that I worked at. We
had to feel our way through to creating a test harness to adequately
test the code without a calling application (since there weren't any
yet anyway).” But salvation came in the
form of relationships, “fortunately the development team was responsive to
helping us create tools to do a good job of testing”.
She also
wasn't afraid to question the expectations of what testing should be
delivering, “we also found that we had to deviate from standard deliverables
mandated by the organization because we were a square peg being
forced into a round hole. At one point, when the standard documentation
wasn't giving me what I needed to do the testing, they asked me to define
what should be contained in the design documents that would be useful. I
have never shied away from digging in on the technical stuff and that
has taken some developers by surprise when I start asking technical
questions. I've seen initial resistance to that line of
questioning but eventually a new level of respect and rapport with developers
when they see how what I am asking for can help overall.”
Elisabeth
makes a good point about challenging where testing should sit and 'who does
testing', “I don't have "test projects." Testing is part of any
project. That by itself is a change for many organizations, and one that
is sometimes met with hostility and scepticism. But when I work with
organizations, part of my mission is to help them integrate testing throughout
so that they don't end up with a huge inventory of unusable source code that
they then have to spend months stabilizing before it's ready for prime time.”
“It takes a whole lot of patience to get people
within an organization to shift their thinking from test-last thinking to
testing throughout. It typically gets easier when they have experience with the
horror show that test-last practices yield, or if they see big improvements
from integrating testing throughout their projects.”
Harry
like many wanted to introduce new ways to his organisation, “I pushed for
exploratory testing to be done as a means of exposing bugs differently.
Skepticism was high so I had to continue writing out test cases and
pursuing the exploratory on my own.”
Again the common theme of reputation and credibility kicks in, as
“ after a few months of exposing things never before encountered for us,
they got the point that we needed to stop doing things 'the way we have always
done it'. We tried for years to get automation brought in to allow for us
to get past regression and allow it to be done via automation. It took
until this March to see the beginnings of that implementation (selenium).
As I pointed out in a presentation I gave last fall, test cases are only
as good as the last bug written against them. Exploratory testing will
allow for us to do code-path/decision point discovery.”
Jari
makes an excellent case for doing the ground work within an organisation and
building the relationships by talking and involving others, “I had to make a
reorganization of a test team once it grew
from 2 to over 60 people (over a
relatively short time). I decided I wanted some team leads/senior assistants in
the team as most of the testers were very junior. A few of those people didn't
have even a year of testing experience, but they had the kind of attitude and
personality I thought would fit the role.”
As many
will sympathise not everything was embraced, and “in the beginning the change
was challenged quite a lot. I spent a large amount of time talking with
individuals, the small teams and the whole team to explain them why I made that
change and why I chose those people. I also made it very clear I welcomed
comments about the changes. After some
time things started calming down and the team was again productive. Some
hiccups here and there, but those are always coming when dealing with people.
All in all, I am very happy how we managed to overcome the change as a team.”
Ajay
talks about the changes he introduced “some of them were challenged but as soon
as the value was demonstrated, others could also highlight the value, people
embraced it.”
Bernice
makes a good point, reiterating Jari's point about engaging with people, “I
find people are open to change if you do not mandate it and if you allow them
to participate in pilot programs to understand testing approaches to adopt.”
Janet
underlined the importance of starting small and winning people over to create a
momentum of change, “my life is about introducing changes, many of them with
significant impacts. Most of them are met with initial hostility /
scepticism. I find that the easiest way to overcome them, is to
show success. I encourage start with something small and easy, see
success. Then move to the next thing. Also, a really important thing to
develop is relationships. Sometimes I introduce a practice to one eager
person(s), get it to work successfully so that there is a frame of
reference, and we can showcase that success to tackle the rest of the team
members.”
QUESTION SIX: Working smarter and not harder, are there any
activities you’ve found your team focusing more on? Consequently are
there any thing you find yourself doing less of?
In a
common theme involved testers questioning how much value detailed test cases and
documentation really delivers, Stephen said that he saw testers “doing more
testing and writing less test cases. In the beginning, using waterfall, we used
to write hundred of tests, taking weeks then run them. Regression efforts took
huge amounts of time as did maintenance of the test cases. Switching to
automation, using dedicated developers in test, then swapping most of the
manual effort to exploratory testing empowered the testers by recognising their
expertise, whilst also removing a large amount of time spent documenting and
maintaining test cases.”
Stephen
had one final concern, “one area that does need to be checked at the moment is
the feeling that you can just automate everything” over understanding the
skills that human testers bring to a project.
Simon
sees Agile being a big part of the way many products are developed in future,
“Agile Test Driven Development tests in a continuous integration environment
are VERY helpful and mean I spend much less time regression testing. Mobile
(development and testing) is a big area of focus for the future.”
Elisabeth
repeat the common themes of more “exploratory testing plus the disciplined
engineering practices associated with Agile including TDD, ATDD, automated
regression” going on to talk about how we need to do less “bug triage, applying
test reduction strategies in an attempt to cut down the regression cycle,
manual scripted testing, test documentation (including traceability matrices).”
Harry
looks back at how his team has evolved and will continue to evolve “we do less
manual (scripted) testing and more exploratory as well as learning/implementing
automation. What passed in the past should continue to pass and it should
not fail. Need to keep moving forward. Once the bug is fixed, test
case passes. If automation picks up too many bugs, someone goofed”.
Betty
examines the ways of working she's finding adding value in order to make
testing for her clients simpler and more focused on adding real value, “I use
things like test checklists, common tests, decision tables over detailed test
cases where I can (and it makes sense to) in order to work smarter. I use
scenario based tests for testing third-party software unless/until issues are
discovered that warrant more detailed functional tests. As a contractor
who is intended to go away, I have felt strongly that I should leave behind
test assets that can be used in the future.
I also test lots of third party software and shouldn't need to be doing
intensive functional sorts of testing so the concept of 'just enough' testing
documentation is my motto.” On a
technical level, “I
have used tools to identify and pull production data, scrub production data or
to generate/manufacture test data. Virtual Machines are a huge
time-saver/help when testing multiple client computer configurations.”
Bernice sees the value of whole-team approaches
“working in teams to approach testing problems allow more testing ideas plus
cross-training for the testers. Spending more time on risk assessment
throughout the project to understand how risk change and to ensure that initial
risks were addressed. Identifying different tools the testers can use based
upon what they are testing. This can include Session-based testing,
mind-mapping, test matrices, testing ideas - the level of documentation is
based upon what is useful and risk level. Plus it should be a living document
as it should change as we learn more through testing.
Betty
has taught testing classes, and sees in modern software that databases have
become more and more the heart of every application, thus “I've been surprised
how many testers do not take the step of doing SQL queries to verify the stored
data and simply trust what the UI shows them. Our testing work has
definitely gotten more technical over the years and that is one aspect that
I've noticed that I have had to grow my skills in order to be better at doing
my job - reading database schemas and related documents, design documents,
writing SQL queries, etc. have all been skills that I've developed along the
way. Long gone are the days of black box testing where we only needed to focus
on inputs and outputs.”
Bernice
picks up on the importance of engaging with ideas from the world-wide testing
community, “we hold a lot of journal clubs and focused discussions with testers
who are interested in exploring certain areas. For example, Michael Bolton's
News Story approach from STAREast is a popular discussion with testers on how
to implement his approach. It is important not to mandate what the testers must
do for all testing assignments. Each should be assessed for the proper testing
approaches or tools. Whereas a session-base test may work well with one testing
assignment, a traditional boundary matrix may work better for another testing
problem”.
I will
leave the final word with Janet, who sees the future of testing quite simply,
“faster feedback - continuous feedback”.
Summary
From all
of this there are a number of key take-home points which come across, many of
which come as no surprise,
- It's important to engage
with stakeholders, and make sure your testing goals and theirs align. If not, how can this be achieved. As Ajay points out time and again, try
to get them to think like testers through education.
- For testers to add value,
testers need to be valued.
Challenge the idea that “anyone
can do testing” by making your testing visible, and building your
credibility.
- Build relationships. Testing is more effective if it pools on
the ideas and experience of the whole project (developers, market managers etc), not just testers.
- Innovate. Try to challenge the idea that “we've
always done testing that way”, especially if testing has always been that
way and delivered little value.
- Pioneer. If documentation is not adding value,
how can it be made to? Or should it
be dropped altogether?
- Time is always going to be a
constraint, and thus it's important to always to do risk analysis. Assess
the risks and limitations in your testing, and give feedback on them to
the project leaders. Help them to
make informed decisions.
- Software is getting more
complex, and as testers we need to not shy away from that complexity.
- Challenge the idea that
testers are responsible for quality.
The whole team is responsible for quality.
