This tutorial suggests that rather than being a document, test strategy is a thought process. The outcome of the thinking might be a short or a long document, but most importantly, the strategy must address the needs of the participants inside the project as well as the customers of the product to be built. It needs to be appropriate to a short agile project or to a 1000 man-year development. It has to have the buy-in of stakeholders but most importantly, it must have value and be communicated.
This tutorial presents a practical definition of a Test Strategy, provides a simple template for creating one and describes a systematic approach to thinking the right way. This will be an interactive session. Bring your test strategy problems with you – we'll try and address them during the day. You will receive a free copy of the Tester's Pocketbook.
Dates & Venues
19 February 2013 – London
09 April 2013 – London
21 May 2013 – London
16 June 2013 – London
10 September 2013 – London
22 October 2013 – London
10 December 2013 – London
There's been a thoughtful debate on the notion of 'test automation' being a rather bad, lazy, misrepresentation as a concept or entity or thing. James Bach and Michael Bolton wrote a paper here. Alan Richardson, in Testing Circus Magazine invites you to join the Anti Test-Automation Brigade. Pretty much, I agree with the sentiments expressed. The term is bad, it exhibits and probably promotes muddled thinking. I will do my best not to use it. Unless I need to reference the range of tools that dominate the testng tools market.
I'll return to this a bit later.
My Dad called himself an engineer (or rather, his company did). In his national service, he joined the Royal Engineers and was very proud of his time in the post-war, peacetime Army. He told me that he spent much of his time going on training courses. When offered the choice of a course on "4.5 ton truck engine maintenance" or two weeks tramping around muddy fields in Germany, unlike many of his comrades who preferred the mud to a classroom or workshop, he chose the courses.
He also did real some engineering - building and dismantling bridges, towing tanks out of ditches and other hearty activities. (Once a tank was bogged down near a churchyard. Dad's idea was to pass a cable around the church to improve the towing angle on the cable. When the recovery truck pulled, the tank was unmoved, but the church started to disintigrate. They stopped the truck and figured out another way to extract the tank, leaving the church intact to great relief). And so on.
So when the time came to selecting a course at university - an engineering degree was the natural choice for me. I thought I would be part of a select, engineering elite. Yes, there were civil, electrical and mechanical engineers, but they were all a happy and respected band of professionals. Then I worked at a large civil engineering consultancy. Everyone was an engineer. And then I noticed, half the male population called themselves engineers of one sort or another. When I got into the software business, I discovered some software people also called themselves engineers. This was a step too far.
Now, my main criteria for the definition of an engineer is someone who works with concepts, designs or materials whose behaviour are underpinned by the laws of physics. Software, in my humble opinion, does not obey any laws of Physics that I know - so it is not an engineering discipline. Discuss. At any rate, it seems nowadays that anyone can call themselves an engineer. Raging against this calumny seems not to be a good use of my time.
The testing tool vendors are not about to change their marketing and rename all their tools on the basis of complaints from a few vociferous individuals. So I suspect we are stuck with the Test automation term for the foreseeable future. I support the view that test automation is a lousy term, but I don't thnk I'm going to get excited about removing or replacing it.
However.
A Bad Term, Well-Scoped?
The view that test automation does not encompass support for all of testing is obvious and if test automation does have a scope, it covers the application of tests (or checks). Some tools can perform logging and reporting, possibly, but let me ignore that for the moment. I am using the teminology of the New Model below.
Let me suggest test automation refers to the 'Applying' activity only. If you add in other utilities that come under the banner of "testing with tool support" - these utilities mostly fall into the Applying aspect of testing. Building environments, preparing test data, generating combinations, analysing results, editors, comparators, analysers, harnesses, frameworks, loggers, cleaner-uppers and tear-downers and so on are all pretty much covered by the Applying activity. So if test automation represents the Applying activity only, it may not be well named, but it could at least be well-scoped.
Perhaps we should rename Test Automation ... Test Application? Or just Application?
More importantly, the nine other activities in the New Model plus the judgements of model adequacy or inadequacy hardly get a mention in the argument which ignores the opportunities to support thinking, collaboration, modelling, test design, record-keeping and so on.
We Need to Curb Our Enthusiasm (or Addiction)
It seems to me that the testing industry - both testers and vendors - are obsessed with test automation to the degree that (with some exceptions) testers are not demanding product to support exploration and modelling activities on the left hand side of the New Model and vendors are not investing in R&D. There are signs this is changing but it has been a slow process so far. Supporting these activities could bring huge benefits to testers. The market for these tools (required by all testers possibly) is also huge. Why are test design tools not being demanded and delivered?
If we label all test tools test automation, it take our eyes of the real prize - tools to support all of testing. Testers and vendors need a broader perspective.
I've been banging on about wanting Test Design tools rather than Test Execution tools for some time. I dug out a talk of 1999 which I gave several times including a variation at STAR East 1999. I am quite pleased - maybe 75% could be repeated today without embarrassment. But I am also rather depressed that I am making almost identical recommendaitons nearly sixteen years later. Click on the image or here to see the slides.
What am I Doing About This?
Firstly, you may know I am working to create a comprehensive tools directory at https://tkbase.com - it covers DevOps, Collaboration and Testing and there are not ten or twenty tool types - there are over 180 tool types with 2414 tools listed.
Second, I am writing a paper on the applicaiton of automation AI and Deep Learning to support the other (exploration, thinking, collaboration and record keeping) activities of testing.
Finally, for the last few months, I've been writing and experimenting with a robot that supports exploration and testing - exploratory testing - if you like. It will support team exploration, pair testing and can also be voice controlled. I'll have something to demonstrate in the next month or so. I'll be talking about experiences with this more fully later in the year. If you are interested in being a pre-alpha tester - do get in touch :O)
What are you doing to further the case for tools that support all of testing?
The Test Management Forum was set up in 2004 and the first meeting took place on 28 January of that year. We have run events every quarter since then, and the 53rd and latest meeting took place at the end of January. Over 2,600 people have attended the London events over the years. Although most Forum attendees come from the UK, the uktmf.com website has approaching 10,000 registered users worldwide and the LinkedIn group 11,800 members.
The TMF continues to be a popular Forum with a global (and not just a UK) influence.
The original aim of the Forum was to bring together more senior practitioners, to network and share knowledge. The core of the Forum meetings are discussion sessions. They are run by an expert in their field and comprise an introductory presentation of 15-30 minutes or so, followed by a facilitated discussion of the issues raised in the talk. Sessions are 75 minutes long and often stir up vigorous debate. The format and ethos of the Forum are unchanged since 2004.
But over the last thirteen years, there have been many changes in the industry and in the Test Management community. I’d like to talk around some of these changes and explain why we decided to bring the TMF up to date with an industry that is far different from that in the early days of the Forum.
In 2004, the Agile movement was still in the early stages. For the next ten years or so, we ran many sessions that explored the changing (sometimes disappearing) role of test managers. I recall having many conversations with people who had been ‘Agiled’ and were concerned their role and contribution to Agile projects was uncertain.
The Test Managers I knew personally reacted in very different ways. Some managers and leaders became testers again, some specialised in test automation or security testing or usability. Others moved closer to their business, became business analysts or consultants. Quite a few were promoted out of testing altogether to become project managers, development leads or heads of solution delivery. Some senior managers retired from IT altogether.
Agile had a big impact, but approaches such as continuous delivery, DevOps, shift- left, shift-right, shift-wherever and analytics are changing the roles of testers in fundamental ways, and I should say, usually for the better. Some testers are falling by the wayside; getting out of testing. The pressures on Test Managers continue, but new doors are opening and opportunities emerging. Some test managers become Scrum Masters, more are finding new roles which I have labelled as Test Masters and in general, these people are performing an Assurance role.
Assurance (as distinct from Quality Assurance) is very much focused on delivery. The assurance role supports the delivers team by acting, at various times, as a process consultant, as a testing expert, as a reviewer, as a project-board level advisor and sometimes as an auditor. This variety of roles is more senior, more comprehensive and more influential than the traditional test manager role. The range of skills and authority required are beyond many test managers – it is not for everybody. But Assurance is a more senior role with a broader influence and is a natural advancement for aspiring test professionals.
With this background, we have decided to re-brand the Test Management Forum to be the Assurance Leadership Forum (or ALF) from April 2017. The ALF has very much the same goals as the TMF, but with a broader organisational, managerial and delivery-focused remit. Test leadership and technical testing issues will figure prominently in the ALF, but the range of topic areas will increase, with an alignment to the aspirations and career progression of senior practitioners.
Mike Jarred, of the Financial Conduct Authority has helped behind the scenes with the Forum for some time. Mike is also one of those managers with a testing background who has advanced beyond testing and into broader software delivery management. He is well placed to take over the Programme Management of the Forum. In December, I moved house to Macclesfield in Cheshire, making it harder for me to host and programme the London-focused Forum events. Mike offered to take the programme role and I gladly took up his offer.
I will continue to be the ‘host’ of the Forum and of course support Mike in putting the events together. My more focused role will allow me to take a more active part in the discussions themselves, for a change! I’ll be asking for expressions of interest in setting up a Northern Forum in the Summer before too long, I'm sure.
In the meantime, I hope you will support the Assurance Leadership Forum, and Mike in leading it.
The existing uktmf.com domain will continue to point to the Forum website, but a new domain ukalf.com points to the same content and will be our preferred url from now on. The LinkedIn group will be also renamed shortly.
Over the next few weeks, we’ll be rewriting some of the content of the website to better define the purpose of the ALF. I expect we’ll have a debate on possible directions in the next Summit. The First ALF Summit will take place on 25 April in London and if you want to contribute to the day, do let us know.
After the Next Generation Testing event in December 2012, Unicom asked me to summarise the thinking behind the proposal for the redistribution of testing. Four minutes 11 seconds. :O)
I’m leaving Twitter, for obvious (to me) reasons, which I'll explain. Essentially, I don't like what's happening to it.
Twitter, YouTube, personal blogs and Mastodon servers are full of accounts and opinions of the calamitous start to Elon Musk's first weeks at the helm of what he hopes will become Twitter 2.0. I've listed some links at the bottom of this post to back up the tale of woe. I don't like Musk, I don't like the sound of his politics, I don't like what he's doing to 'turn the company around'. I'm guessing he'll fail sooner rather than later.
Twitter was bought with borrowed money and now has a debt of around $13bn costing $1bn a year to service. It's revenues are plummeting as advertisers abandon the site. Half of the workforce of 7,500 has been fired (illegally), and whole departments have resigned, refusing to go 'hardcore' (including the HR payroll department, apparently). Of the 3750 or so of employees who were not fired, 75% did not respond to the 'click to be hardcore or be fired' email from Musk.
No one knows how many of the original 7,500 employees are still at the company. It could be just a few hundred who remain. It seems likely that many more people remain at the company because there's no HR staff to terminate these employees. And so on.
What does this mean for Twitter?
The general view expressed by experts, Twitter-watchers and ex-employees is that when (not if) Twitter has some infrastructure failures, there may not be enough (or any) people with the skills required to restore the service. Twitter will always have had hackers trying to penetrate the site, but, given the vulnerability of the service they'll be trying extra hard to bring it down. Forever.
It's when they deploy larger software or infrastructure upgrades when the fun will really start.
Musk has also admitted, if Twitter can't get it's revenues up it may have to file for bankruptcy. It really is that bad.
So, I'm leaving (not leaving) Twitter
I won't close my account because, you never know, maybe it'll turn a corner or something and become both viable and an attractive, friendly place to be. But I'm not holding my breath.
Introducing Mastodon
Mastodon seems to be the main game in town for people wishing to change platforms. Compared to Twitter, it is still small – around 8million accounts according to https://bitcoinhackers.org/@mastodonusercount – but growing fast as most people leaving the 'bird site' need a new home and land on the federated, decentralised service named after the ancient ancestor of elephants.
Lots of blogs and YouTube videos explaining what Mastodon is and how you use it have appeared over the last few weeks. You must choose and register on a specific server (often called an instance), but that server is one of (today) about 3,500. At the time of writing, around 200 servers are being added per day. You can think of a Mastodon server a bit like an email server. You have to toot (or post) rather than Tweet through your home server, but it can connect with every other Mastodon user or server on the planet. (Unless they are blacklisted – and some are).
I have set up a Mastodon Server
It's not as easy to find people you know, and of course, it's early days and most people aren't on Mastodon yet. But it's growing steadily as people join, experiment and learn how to use the service.
Mastodon accounts are like Twitter except, like email, you have to specify a service that you are registered with. For example, my Mastodon account is @paul@tester.social – a bit like an email address. Click on my address to see my profile.
I've been using Mastodon for about a month now, and I've found and followed Mastodon accounts of 70-75 testing-involved people I follow on the bird site. That's nearly a quarter of the 325 I follow (and I follow quite a few non-testing, jokey and celebrity accounts). So I'll risk saying, 25% or more tech-savvy people have made the move already. If you look at who the people you know follow, you will see names you recognise. It's not so hard to get started. And enthusiastic people who follow you on Twitter will be looking for you, when they join.
Why did I set up a Mastodon server?
Good question. On the one hand, I like to try new products and technologies – I have been running my own hardware at data centres since the early 2000s. At one point I had a half rack and eleven servers. Nowadays, I have three larger servers hosting about twenty virtual machines. If you want to know more about my set up, drop me a line or comment below.
I've hosted mail and web servers, Drupal and Wordpress sites, and experimented with Python-based data science services, MQTT, lots of test tools, and for a while, I even ran a Bitcoin node to see what was involved. So I thought I'd have a play with Mastodon. I used this video to guide me through the installation process. A bit daunting but with open-source software, you have to invest time to figure things out that aren't explained in documentation or videos.
tester.social is hosted on a 4 cpu, 16Gb memory, 1 Tb data virtual machine and uses Cloudflare R2 as a cloud-based object store for media uploads etc. From what I've seen of other site setups it would easily support 10,000 or more registered users. But, I'm going to monitor it closely to see how it copes of course.
It's an experiment, but one we will support for the next year at least. If it takes off and we get lots of users, we may have to think carefully about hiring technical and moderation staff and/or limiting registrations. But that's a long way off for now.
Be aware that when you register, you will be asked to explain in a few words what you want to see on the site, and what you'll be posting about. Your account will be reviewed and you'll get an email providing a confirmation and welcome message. This is purely to dissuade and filter out phoney accounts and bots.
We will commit to the Mastodon Server Covenant and hopefully be registered with the Mastodon server community which today numbers just over 3,000. Nearly 2,000 servers have been set up since Musk took over the bird site. Mastodon is growing rapidly.
I don't know anyone on Mastodon, how do I find people I know?
If you are on Twitter, keep an eye out for people you know announcing their move to Mastodon – follow them, and see who they follow. And follow them. And see who they follow that you know and follow them and...
If you are not on Twitter, follow me. See who I follow and follow those you know. You'll get the hang of it pretty quickly.
I don't want to leave twitter yet, but want to experiment
Firstly, you can join any Mastodon service and use a cross-poster to copy your toots to tweets and vice versa. All new post on either service will be mirrored to the other. I found this article useful to learn what help is out there to migrate from Twitter to Mastodon: https://www.ianbrown.tech/2022/11/03/the-great-twitter-migration/.
For example, there are tools to scan your Twitter follows and followers for Mastodon handles, and you can import these lists to get you started. I used Movetodon to follow all my Twitter follows who had Mastodon accounts. Over time, I'll use it again and again to catch people who move in the future.
I registered with https://moa.party/ – it synchronises my posts on tester.social and Twitter in both directions – so I only have to post once. I post on tester.social and the tweet that appears on bird site isn't marked as 'posted by a bot' – so that works for me.
I found Martin Fowler's Exploring Mastodon blog posts on the subject very useful. He talks about his first month of using the service. Which is where I am at the moment. Sort of.
A few FAQs answered. Sort of
What if I don't like the service I register with or prefer another service? You can always transfer your account to another Mastodon service. Your follows and followers will be transferred and your posts, blocks and so on. You have to create a new account on the target service and may have to change your account name if your current name is taken on the new service. A redirect from old to new account is part of the service.
I hate advertising – can I avoid it? Mastodon servers do not display advertisements. Of course, companies might post commercials, but if you complain, it might be taken down and the poster be advised to leave the service and blocked in extremis.
What are toots and boosts? A toot is equivalent to a Tweet, and a boost is the same as a Retweet.
Are there apps for Mastodon? Yes of course – you can see the IOS and Android apps here. there are also a number of 3rd party apps. Don't ask me what they do – I haven't explored.
Why is the web interface better than apps? If you use the web interface, you can turn on what is called Advanced Web Interface in your preferences. In this version, you can view and pin multiple columns at the same time – each column having your toot window, home timeline, local timeline, federated timeline in parallel. You can set up what appears in each column in your preferences.
What are toot window, home timeline, local timeline, federated timeline? The toot window is where you post new messages. The three timelines are:
Home: like on Twitter, it shows all the posts of all the people you follow on all Instances Local: it shows all the posts of the members of your Instance Federated: it shows all the posts of the members of your Instance and also the posts of people on other Instances that are followed by people of your Instance.
If you want more help – and I'm sure you do, try this site: https://mastodon.help/ – it seems to cover all of the basics and quite a lot of the advanced stuff too.
If you join tester.social and need help; have a question?
If you need help mention @questions in a post and it'll reach us. We'll try to answer as soon as we can.
Test Strategy is a Thought Process, Not a Document
Test strategy is one of those nebulous terms for an activity that everyone has to undertake for their development projects. The problem with test strategy is that there isn't a single agreed definition of what it is. Some people treat Test Strategy as a document: "A high-level description of the test levels to be performed and the testing within those levels for an organization or programme". So - a strategy could range from one to hundreds of pages of text. All we need is a document template!
Not quite.
Over the last twenty years, almost all of the Test Strategy documents Paul has reviewed have been "copy and edits" of documents for previous projects. All that changed was the names and the dates. And all of these document were read by … no one.
We suggest that rather than being a document, test strategy is a thought process. The outcome of the thinking might be a short or a long document, but really, the strategy must address the needs of the participants inside the project as well as the customers of the product to be built. It needs to be appropriate to a short agile project or to a 1000 man-year development. It has to have the buy-in of stakeholders but most importantly, it must have value and be communicated. That's quite an ambitious goal. but we think it is achievable.
A Strategy Answers Questions
Before your organisation, your team or you as an individual test a system, you need some questions answered. These questions are pretty fundamental and yet, we have met many teams who could not answer them - even when they were in the thick of the testing. Here is a sample of the type of questions to be answered?
Who are your testing stakeholders? What do they want from testing? Why do they want it? In what format? How often?
What is the scope of testing? What is out of scope? How will you manage scope?
How much testing is enough? What models should be used to guide the testing? How will you assess coverage?
and so on...
One way of looking at strategy is to regard it as a way of answering these questions. But not every question can be answered up-front. Some questions cannot be answered now, but perhaps later when information come to light. So the strategy must provide guidance for answering these questions and making decisions. It does this in three ways:
Some decisions can be made now, directly and with confidence.
Some decisions cannot be made now, but the strategy will provide a process, method or mechanism for reaching a decision.
Some decisions cannot be made now, and no method could be formulated, but the strategy can provide some guiding principles to be followed in reaching a decision.
In this way, a strategy written early in a project, before all the information required is available, can flex and support a project that is heading into unknown territory.
Is Agile Test Strategy an oxymoron? We don't think so. If you look at the three levels of decision making above, structured or waterfall projects aim to answer every questin ahead of time. So, in these projects, most of the quesitons can be directly answered so the emphasis is on 'type 1' decisions. In Agile or less certain projects, the lack of knowledge, or the need to remain flexible demands that the emphasis will shift more towards 'type 2' or 'type 3' decisions.
The Trick is to Ask the Right Questions
The challenge in test strategy is not providing perfect answers to these questions. The secret to success is in asking the right questions in the first place. Over twenty years, we have found oruselves asking the same questions again and again. The Test Axioms are an attempt to arrange these questions into a more usable structure. The Axioms represent context-neutral aspects of testing to think about. The 16 axioms are organised into three Axiom groups: Stakeholder, Design and Delivery. Each Axiom has 5-10 questions associated with the, the Axioms therefore provide a checklist of questions to ask, a a structure for your strategy document, should you coose to write one.
Our Test Strategy workshop
We have created a workshop titled 'Test Strategy in a Day' that you might find useful. In this workshop, we present a practical definition of a Test Strategy, provide a simple template for creating one and describe a systematic approach to thinking the right way. It's an interactive session where we invite you to bring your test strategy problems with you - we'll try and address them during the day.
Test strategy is a thought process, not a document
The goal is to acquire and communicate knowledge of achievement
How a valuable test strategy can be formulated and communicated
See below the four presentations given by Paul at the World Conference on Next Generation Testing held in Bangalore, India between 8th and 12th July 2013.
In London, on 18 May I presented a keynote to the Testing and Finance conference. I've been asked for the slides of that talk, so I have uploaded them here. The talk was mostly based on two articles that I originally wrote for Atlassian, and you can find the text of those articles in the blog here: http://gerrardconsulting.com/index.php?q=node/602
Note that the presentation introduces some broader suggestions about influences on the future of testing and testers, including the increasing adoption of continuous delivery.
It's a rather poor, but sadly typical, misrepresentation or let's be generous miunderstanding of the "state of the industry". The opening comment gives you the gist.
"If you work in software quality assurance (QA), it’s time to find a new job."
Apparently DevOps is the 'next generation of agile development ... and eliminates the need for QA as a separate entity'. OK maybe DevOps doesn't mandate or even require independent test teams or testers so much. But it does not say testing is not required. Whatever.
There then follows a rather 'common argument' - I'd say eccentric - view of DevOps at the centre of a Venn diagram. He then references somene elses' view that suggests DevOps QA is meant to prevent defects rather than find them but with all due respect(!) both are wrong. Ah, now we get to the meat. Nearly.
The next paragraph conflates Continuous Delivery (CD), Continuous Integration and the 'measurement of quality'. Whatever that is.
"You cannot have any human interaction if you want to run CD."
Really?
"The developers now own the responsibility rather than a separate entity within the organization"
Right. (Nods sagely)
"DevOps entails the use of vendors and tools such as BUGtrack, JIRA and GitHub ..."
"To run a proper DevOps operation, you cannot have QA at all"
That's that then. But there's more!
"So, what will happen to all of the people who work in QA? One of the happiest jobs in the United States might not be happy for long as more and more organizations move to DevOps and they become more and more redundant."
Happy? Er, what? (Oh by the way, redundant is a boolean IMHO).
Then we have some interesting statistics from a website http://www.onetonline.org/link/summary/15-1199.01 I can't say I know the site or the source of data well. But it is entirely clear that the range of activities of ISTQB qualified testers have healthy futures. In the nomenclature of the labels for each activitiy the outlook is 'Bright' or 'Green'. I would have said, at least in a DevOps context that their prospects were less than optimal, but according to the author's source, prospects are blooming. Hey ho. Quote a source that contradicts one's main thesis. Way to go!
And here we have it. The definitive statistics that prove DevOps is on the rise and QA jobs are collapsing!
"For QA, the trend definitely does not look good."
So. That's that. The end of QA. Of Testing. Of our voice of reason in a world of madness.
Or is it? Really?
I followed the link to the Google stats. I suggest you do the same. I entered 'Software Testing Jobs' as a search term to be added and compared on the graph and... voila! REDEMPTION!!!
Try it yourself, add a search term to the analysis. Here is the graph I obtained. I suggest you do the same. Here's is my graph:
Now, our American cousins tend to call testers and testing - QA. We can forgive them, I'm sure. But I know the term testers is more than popular in IT circles over there. So think on this:
The ratio of Testers v DevOps jobs is around five to one. Thats testers to ALL JOBS IN DEVOPS IS FIVE TO ONE.
ARE WE CLEAR?
So. A conclusion.
Don't pay attention to blogs of people with agendas or who are clearly stupid.
Think carefully about the apparent sense but clear nonsense that people put on blogs.
Be confident that testing, QA or whatever you call it is as important now as it was forty years ago and always will be.
It's just that the people who do testing might not be called testers. Forever.
