First published 09/02/2012
Atlassian kindly asked me to write a blog post on testing for their website. I wrote a longer, two-part article that appears here and here. I have combined the two articles into a single blog post here.
Testing is Dead?
Recently, there has been a spate of predictions of doom and gloom in our business. Conference talks have had titles such as ‘Test is Dead’ and ‘Death to the Testing Phase’. ‘Testing has contributed little to quality improvement in the last ten years’, and even being a tester is a ‘bad thing’ – are all keynote themes that circulated at conferences, blogs and YouTube in late 2011.
My own company has been predicting the demise of the ‘plain old functional tester’ for years and we’ve predicted both good and bad outcomes of the technology and economic change that is going on right now. In July I posted a blog, ‘Testing is in a Mess’ where I suggested that there's complacency, self-delusion and over capacity in the testing business; there is too little agreement about what testing is, what it’s for or how it should be done.
There are some significant forces at play in the IT industry and I think the testing community, at least the testers in what might be called the more developed ‘testing nations’ will be coming under extreme pressure.
The Forces and Factors that will Squeeze Testers
The growth of testing as a career
Over the last twenty years or so there has been a dramatic growth in the number of people who test and call themselves testers and test managers. When I started in the testing business in 1992 in the UK, few people called themselves a tester, let alone thought of themselves as having a career in testing. Now, there are tens of thousands in the UK alone. Twenty years ago, there were perhaps five companies offering testing services. There must be ten times that number now and there are hundreds, perhaps thousands of freelancers who specialise and make a living in testing. Beyond this of course, the large system integration and outsourcing firms have significant testing practices with hundreds or even thousands of staff, many offshored of course.
It’s not that more testing happens. I think it’s because the people who do it are now recruited into teams, having managers who plan, resource and control sizable budgets in software projects to perform project test stages. Many ‘career testers’ have never done anything else.
Lack of advance in the discipline
The sources and sheer volume of testing knowledge have exploded. There are countless papers, articles, blogs and books available now, and there are many conferences, forums, meet-ups and training courses available too. But, even though the volume of information is huge, most of it is not new. As a frequent conference goer over 20 years, it depresses me that the innovation one sees in conferences, for example, tends to be focused on the testers’ struggle to keep pace with and test new technologies rather than insights and inventions that move the tester’s discipline forward.
Nowadays, much more attention is paid to the management of testing, testers and stakeholders expectations and decision making. But if you consider the argument that perhaps test management is a non-discipline, that is there is no such thing as test management, there’s just management, and you take the management away from test management – what’s left? Mostly challenges in test logistics – or just logistics – and that’s just another management discipline?
Advances(?) in Automation
What about the fantastic advances in automation? Let’s look at the two biggest types of test automation.
Test execution robots are still, well, just robots. The advances in these have traced the increased complexity of the products used to build and deliver functionality. From green-screen to client/server to GUI to Web, to SOA, the test automation engineer of 1970 (once they got over the shock of reincarnation) would quickly recognise the patterns of test automation used today. Of course, the automation frameworks are helping to make test automation somewhat more productive, but one could argue that people have been building their own custom frameworks for years and years and they should have been mainstream long ago.
The test management tools that are out there are fantastic. Integrated test case management, scheduling, logging and incident management and reporting. Except that the fundamental purpose of these tools is basic record-keeping and collaboration. Big deal. The number of companies who continue to use Excel as their prime test management tools shows just how limited the test management tools are in what they do. Most organisations get away without test management products altogether because these products support the clerical test activities and logistics, but do little or nothing to support the intellectual effort of testers.
The Emergence/Dominance of Certification
The test certification schemes have gone global it seems. Dorothy Graham and I had an idea for a ‘Foundation’ certification in 1997 and we presented a one page syllabus proposal to an ad-hoc meeting at the Star WEST conference in San Jose to gauge interest. There wasn’t much. So we came back to the UK, engaged with ISEB (not part of BCS in those days) and I became the founding Chair of the initial ISEB Testing Board. About ten or so UK folk kicked off the development of the Foundation scheme which had its first outing in late 1998.
As Dorothy says on her blog (http://dorothygraham.blogspot.com/2011/02/part-2-bit-of-history-about-istqb.html), the Foundation met its main objective of “removing the bottom layer of ignorance” about software testing. Fourteen years and 150,000 certificate awards later, it does the same. Except that for many people it’s all they need (and may ever need) to get a job in the industry.
The Agile Juggernaut
Agile is here to stay. Increasingly, developers seem to take test, Test-Driven and Behaviour-Driven Development and Specification by Example more seriously. Continuous Integration and Delivery is the heartbeat, the test, life-support and early warning system. The demands for better testing in development are being met. A growing number of developers have known no other way.
It seems likely that if this trend continues, we’ll get better, stable software sooner and much of the checking done late by system testers will not be required. Will this reduce the need for system testers? You bet.
Some Agile projects don’t use testers – the testers perform a ‘test assurance’ role instead. The demand for unskilled testers reduces and the need for a smaller number of smarter testers with an involvement spread over multiple projects increases. Again – fewer testers are required.
What is the Squeeze?
The forces above are squeezing testers from the ‘low-value’ unskilled, downstream role to upstream, business-savvy, workflow-oriented, UX (user experience)-aware testing specialists with new tools. Developers are absorbing a lot of checking that is automated. Some business analysts are taking their chance and absorbing test disciplines into analysis and are taking over the acceptance process too.
If a 3 day certification is all you need to be a professional tester, no wonder employers think testing is a commodity, so will outsource it when they can.
Stakeholders know that avoiding defects is better than finding them. Old-style testing is effective but happens at the end. Stakeholders will say, “Let’s take requirements more seriously; force developers to test and outsource the paperwork”.
Smart testers need to understand they are in the information business, that testing is being re-distributed in projects and if they are not alert, agile even, they will be squeezed out. Needless to say, the under-skilled testers, relying on clerical skills to get by will be squeezed out.
A Methodological Shift
There seems to be a methodological shift from staged, structured projects to iterative and Agile and now, towards ‘continuous delivery’. Just as companies seem to be coming to terms with Agile – it’s all going to change again. They are now being invited to consider continuous ‘Specification by Example’ approaches. Specification by example promotes a continual process of specification, exampling, test-first, and continuous integration.
But where does the tester fit in this environment?
The Industry Changes its Mind – Again
So far, I’ve suggested there were four forces that were pushing testers out of the door of software projects (and into the real world, perhaps). Now, I want to highlight the industry changes that seem to be on the way, that impact on development and delivery and hence on testing and testers. After the negative push, here’s the pull. These changes offer new opportunities and improve testers’ prospects.
Recent reports (IBM’s ‘The Essential CIO’ 2011 study and Forrester’s ‘Top 10 Technology Trends to Watch’) put Business Intelligence, adoption of cloud platforms and mobile computing as the top three areas for change and increased business value (whatever that means).
Once more, the industry is in upheaval and is set for a period of dramatic change. I will focus on adoption of the cloud for platforms in general and for Software as a Service (SaaS) in particular and the stampede towards mobile computing. I’m going to talk about internet- (not just web-) based systems rather than high integrity or embedded systems, of course.
The obvious reason for moving to the cloud for Infrastructure as a Service (IaaS) and regardless of the subtleties of capex v opex costs, the cost advantage of moving to cloud-based platforms is clear. “Some of this advantage is due to purchasing power through volume, some through more efficient management practices, and, dare one say it, because these businesses are managed as profitable enterprises with a strong attention to cost” (http://www.cio.com/article/484429/Capex_vs._Opex_Most_People_Miss_the_Point_About_Cloud_Economics). So, it looks like it’s going to happen.
Moving towards IaaS will save some money. The IT Director can glory in the permanent cost savings for a year – and then what? Business will want to take advantage of the flexibility that the move to the cloud offers.
The drift from desktop to laptop to mobile devices gathers pace. Mobile devices coupled with cloud-based services have been called the ‘App Internet’. It seems that many websites will cease to be and might be replaced by dedicated low-cost or free Apps that provide simple user interfaces. New businesses with new business models focusing on mobile are springing up all the time. These businesses are agile by nature and Agile by method. The pull of the App internet and Agile approaches is irresistible.
The Move to SaaS and Mobile (App) Internet
I’m not the biggest fan of blue sky forecasters, and I’m never entirely sure how they build their forecasts with an accuracy of more than one significant digit, but according to Forrester’s report “Sizing the Cloud”, the market for SaaS will grow from $21bn in 2011 to $93bn in 2016 and represent 26% of all packaged software. (
http://forrester.com/rb/Research/sizing_cloud/q/id/58161/t/2).
Now 26% of all packaged software doesn’t sound so dramatic, but wait a minute. To re-architect an installed base of software and create new applications from scratch to make that percentage will be a monumental effort. A lot of this software will be used by corporates who have systems spanning the (probably private) cloud and legacy systems and the challenges of integration, security, performance and reliability will be daunting.
The Impact on Development, Delivery and Testing
Much of the software development activity in the next five years or so will be driven by the need for system users and service vendors to move to new business models based on new architectures. One reason SaaS is attractive to software vendors is that the marketing channel and the service channel are virtually same and the route to market is so simple that tiny boutique software shops can compete on the same playing field as the huge ISVs. The ISVs need to move pretty darned quick not to be left with expensive, inflexible, unmarketable on-premise products so are scrambling to make their products cloud-ready. Expect there to be some consolidation in some market sectors.
SaaS works as an enabler for very rapid deployment of new functionality and deployment onto a range of devices. A bright idea in marketing being deployed as new functionality in the afternoon seems to be feasible and some companies seem to be succeeding with ‘continuous delivery’. This is the promise of SaaS.
Many small companies (and switched-on business units in large companies) have worked with continuous delivery for years however. The emergence of the cloud and SaaS and of maturing Agile, specification by example, continuous integration, automated testing and continuous delivery methods means that many more companies can take this approach.
What Does this Mean for Software Practitioners?
Businesses like Amazon and Google and the like have operated a continuous delivery model for years. The ‘Testing is Dead’ meme can be traced to an Alberto Savoia talk at Google’s GTAC conference (
http://www.youtube.com/watch?v=X1jWe5rOu3g). Developers who test (with tools) ship code to thousands of internal users who ‘test’ and then the software goes live (as a Beta, often). Some products take off; some, like Wave, don’t. The focus of Alberto’s talk is that software development and testing is often about testing ideas in the market.
Google may have a unique approach, I don’t know. But most organisations will have to come to terms with the new architectures and a more streamlined approach to development. The push and pull of these forces are forcing a rethink of how software available through the internet is created, delivered and managed. The impacts on testing are significant. Perhaps testing and the role of testers can at last can mature to what they should be?
Some Predictions
Well, after the whirlwind tour of what hot and what’s not in the testing game, what exactly is going to happen? People like predictions so I’ve consulted my magic bones and here are my ten predictions for the next five years. As predictions go, some are quite dramatic. But in some companies in some contexts, these predictions will come true. I’m just offering some food for through.
Our vision, captured in our Pocketbook (http://businessstorymethod.com) is that requirements will be captured as epic stories, and implementable stories will example and test those requirements to become ‘trusted’, with a behaviour-driven development approach and an emphasis on fully and always automated checking. It seems to us that this approach could span (and satisfy) the purist Agilists but allow many more companies used to structured approaches to adopt Agile methods whilst satisfying their need to have up-front requirements.
Here are my predictions:
- 50% of in-house testers will be reassigned, possibly let go. The industry is over staffed by unqualified testers using unsystematic, manual methods. Lay them off and/or replace them with cheaper resource is an easy call for a CIO to make.
- Business test planning will become part of up-front analysis. It seems obvious, but why for all these years has one team captured requirements and another team planned the test to demonstrate they are met. Make one (possibly hybrid) group responsible.
- Specification by example will become the new buzzword on people’s cv. For no other reason that SBE incorporates so many buzzwordy Agile practices – Test-First, Test-Driven, Behaviour-Driven, Acceptance-Test Driven, Story-Testing, Agile Acceptance testing) – it will be attractive to employers and practitioners. With care, it might actually work too.
- Developers will adopt behaviour-driven-development and new tools. The promise of test code being automatically generated and executed compared to writing one’s own tests is so attractive to developers they’ll try it – and like it. Who writes the tests though?
- Some system tests and most acceptance tests will be business-model driven. If Business Stories, with scenarios to example the functionality, supported by models of user workflows are created by business analysts, those stories can drive both developer tests and end to end system and acceptance tests. So why not?
- Business models plus stories will increasingly become ‘contractual’. For too long, suppliers have used the wriggle-room of sloppy requirements to excuse their poor performance and high charges for late, inevitable changes to specification. Customers will write more focused compact requirements, validated and illustrated with concrete examples to improve the target and reduce the room for error. Contract plus requirements plus stories and examples will provide the ‘trusted specification’.
- System tests will be generated from stories or be outsourced. Business story scenarios provide the basic blocks for system test cases. Test detailing to create automated or manual test procedures is a mechanical activity that can be outsourced.
- Manual scripted system test execution will be outsourced (in the cloud). The cloud is here. Testers are everywhere. At some point, customers will lose their inhibition and take advantage of the cloud+crowd. So, plain old scripted functional testers are under threat. What about those folk who focus more on exploratory testing? Well, I think they are under threat too. If most exploration is done in the cloud, then why not give some testing to the crowd too?
- 50% of acceptance tests will be automated in a CI environment for all time. Acceptance moves from a cumbersome, large manual test at the end to a front-end requirements validation exercise with stories plus automated execution of those stories. Some manual tests, overseen by business analysts will always remain.
- Tools that manage requirements, stories, workflows, prototyping, behaviour-driven development, system and acceptance testing emerge.
Where do testers fit? You will have to pick your way through the changes above to find your niche. Needless to say you will need more than basic ‘certification level’ skills. Expect to move towards a specialism or be reassigned and/or outsourced. Business analysis, test automation, test assurance, non-functional testing or test leadership beckon.
Whither the Test Manager?
You are test manager or a test lead now. Where will you be in five years? In six months? It seems to me there are five broad choices for you to take (other than getting out of testing and IT altogether).
- Providing testing and assurance skills to business: moving up the food chain towards your stakeholders, your role could be to provide advice to business leaders wishing to take control of their IT projects. As an independent agent, you understand business concerns and communicate them to projects. You advise and cajole project leadership, review their performance and achievement and interpret outputs and advise your stakeholders.
- Managing Requirements knowledge: In this role, you take control of the knowledge required to define and build systems. Your critical skills demand clarity and precision in requirements and the examples that illustrate features in use. You help business and developers to decide when requirements can be trusted to the degree that software can reasonably be built and tested. You manage the requirements and glossary and dictionary of usage of business concepts and data items. You provide a business impact analysis service.
- Testmaster – Providing an assurance function to teams, projects and stakeholders: A similar role to 1 above – but for more Agile-oriented environments. You are a specialist test and assurance practitioner that keeps Agile projects honest. You work closely with on-site customers and product owners. You help projects to recognise and react to risk, coach and mentor the team and manage their testing activities and maybe do some testing yourself.
- Managing the information flow to/from the CI process: in a Specification by Example environment, if requirements are validated with business stories and these stories are used directly to generate automated tests which are run on a CI environment, the information flows between analysts, developers, testers and the CI system is critical. You define and oversee the processes used to manage the information flow between these key groups and the CI system that provides the control mechanism for change, testing and delivery.
- Managing outsourced/offshore teams: In this case, you relinquish your onsite test team and manage the transfer of work to an outsourced or offshore supplier. You are expert in the management of information flow to/from your software and testing suppliers. You manage the relationship with the outsourced test team, monitor their performance and assure the outputs and analyses from them.
Close
The recent history and the current state of the testing business, the pressures that drive the testers out of testing and the pull of testing into development and analysis will force a dramatic re-distribution of test activity in some or perhaps most organisations.
But don’t forget, these pressures on testing and predictions are generalisations based on personal experiences and views. Consider these ideas and think about them – your job might depend on it. Use them at your own risk.
Tags: #testingisdead #redistributionoftesting
Paul Gerrard
My linkedin profile is here
My Mastodon Account
