First published 10/06/2014
At the Unicom NextGen Testing show on 26th June, (http://www.next-generation-testing.com/) I'll be pitching some ideas for where the testing world is going – in around 15 minutes. I thought I'd lay the groundwork for that short talk with a blog that sets out what I've been working on for the past few months. These things might not all be new in the industry, but I think they will become increasingly important to the testing community.
There are four areas I've been working on in between travelling, conferences and teaching.
My argument is this: for the investment of time and effort required, learning how to write some simple code in some language will give you a skill that you might be able to use to write your own tools, but more importantly, the confidence and vocabulary to have more insightful discussions with developers. Oh, and by the way, it will probably make you a better tester because you will have some insider knowledge on how programmers work (although some seem to disagree with that statement).
Anyway, I have taken the notion further and proposed a roadmap or framework for a programming training course for testers. Check this out: http://gerrardconsulting.com/?q=node/642
In creating the materials, I initially thought I'd crank out a ton of powerpoint and some sample Python code and walk the class through examples. But I changed tack almost immediately. I decided to write a Python programming primer in the Pocketbook format and extract content from the book to create the course. I'd be left with a course and a book (that I could give away) to support it. But then almost immediately, I realised two things:
I have called this Lean Python and the link to the book website is here: http://www.apress.com/gb/book/9781484223840#otherversion=9781484223857
Now, it quickly occurred to me that I really did not know where all this Big Data was coming from. There were hints from here and there, but it subsequently became apparent that the real tidal wave that is coming is the Internet of Things (also modestly known as the Internet of Everything)
So I started looking into IoT and IoE and how we might possibly test it. I have just completed the second article in a series on Testing the Internet of Everything for the Tea Time with Testers magazine. In parallel with each article, I'm presenting a webinar to tell the story behind each article.
In the articles, I'm exploring what the IoT and IoE are and what we need to start thinking about. I approach this from the point of view of a society that embraces the technology. Then look more closely at the risks we face and finally how we as the IT community in general and the testing community in particular should respond. I'm hopeful that I'll get some kinf of IoE Test Strategy framework out of the exercise.
The first article in the series appears in the March-April edition of the magazine (downloadable here) and is titled, “The Internet of Everything – What is it and how will it affect you”.
There is a video of an accompanying webinar here: The Internet of Everything – What is it and how will it affect you.
It seems like the software development business is changing. It is 'Shifting Left' but this change is not being led by testers. The DevOps, Continuous Delivery, Behaviour-Driven Development advocates are winning their battles and testers may be left out in the cold.
Because the shift-left movement is gathering momentum, Big Data and the Internet of Everything are on the way, I now believe that we need a New Model of Testing. I'm working on this right now. I have presented draftsof the model to audiences in the UK, Finland, Poland and Romainia and the feedback has been really positive.
You can see a rather lengthy introduction to the idea on the EuroSTAR website here. The article is titled: The Pleasure of Exploring, Developing and Testing. I hope you find it interesting and useful. I will publish a blog with the New Model for Testing soon. Watch this space.
First published 10/04/2014
A question from Amanda in Louiville, Kentucky USA.
“What's the acceptable involvement of a QA analyst in the requirements process? Is it acceptable to communicate with users or should the QA analyst work exclusively with the business team when interpreting requirements and filling gaps?
As testers, we sometimes must make dreaded assumptions and it often helps to have an awareness of the users' experiences and expectations.”
“Interesting question, Amanda. Firstly, I want to park the ‘acceptable’ part of your question. I’ll come back to it, I promise.
Let me suggest firstly, that collaboration and consensus between users, BAs, developers and testers is helpful in almost all circumstances. You may have heard the phrase ‘three amigos’ in Agile circles to describe user/BA, developer and tester collaboration. What Agile has reminded us of most strongly is that regular and rapid feedback is what keeps momentum going in knowledge based (i.e. software development) projects.
In collaborative teams, knowledge gets shared fast and ‘dreaded assumptions’ don’t turn into disasters. I can think of no circumstance where a tester should not be allowed to ask awkward questions relating to requirements like ‘did you really mean this...?’, ‘what happens if...?’, ‘Can you explain this anomaly?’, ‘If I assume this..., am I correct?’. Mostly, these questions can be prefaced with another.
‘Can I ask a stupid question?’ reduces the chance of a defensive or negative response. You get the idea, I’m sure.
Where there is uncertainty, people make assumptions unless they are encouraged to ask questions and challenge other people’s thinking – to get to the bottom of problems. If you (as a tester) make assumptions, it’s likely that your developers will too (and different assumptions, for sure). Needless to say the users, all along, may be assuming something entirely different. Assume makes an ass of u and me (heard that one before?)
So – collaboration is a very positive thing.
Now, you ask whether it is ‘acceptable’ for testers to talk direct to users. When might it not be “acceptable”? I can think of two situations at least. (There are probably more).
One would be where you as a tester work for a system supplier and the users and BAs work for your customer. Potentially, because of commercial/contractual constraints you might not be allowed to communicate directly. There is a risk (on both sides) that a private agreement between people who work for the supplier and customer might undermine or conflict with an existing contract. The formal channels of communication must be followed. It is a less efficient way of working, but sometimes you just have to abide with commercial rules. Large, government or high-integrity projects often follow this pattern.
Another situation may be this. The BA perceives their role to be the interface between end users and a software project team. No one is allowed to talk direct to users because private agreements can cause mayhem if only some parties are aware of them. The BA is accountable to users and the rest of the project team for changes to requirements. There may be good reasons for this, but if you all work for the same organisation what doesn’t help is a ‘middle man’ who adds no value but distorts (unknowingly, accidentally or deliberately) the question from a tester and the response from a user.
Now, a good (IMHO) BA would see it as perfectly natural to allow testers (and other project participants) to ask questions of users directly, but it is also reasonable for them to be present, to assess consequences, to facilitate discussion, to capture changed requirements and disseminate them. That’s pretty much their job. A tester asking awkward questions is teasing out value and reducing uncertainty – a good thing. Who would argue with that?
But some BAs feel they ‘own’ the relationship with users. They get terribly precious about it and feel threatened and get defensive if other people intervene. In this case, the ‘not acceptable’ situation arises. I have to say, this situation reflects a rather dysfunctional relationship, not a good one. It isn’t helpful, puts barriers in the way of collaboration, introduces noise and error into the flow of information, causes delays and causes uncertainty. All together a very bad thing!
Having said all that, with this rather long reply, I’ve overran some quota or other, I’m sure. The questions I would ask, ‘unacceptable to whom?’ and ‘why?’ Are BAs defending a sensible arrangement or are they being a pain in the assumption?”
First published 01/06/2016
A recent study from the University of Oxford makes for interesting reading:
If programmers have a 50/50 chance of being replaced by robots, we should think seriously on how the same might happen to testers.
Machine Learning in testing is an intriguing prospect but not imminent. However, the next generation of testing tools will look a lot different from the ones we use today.
For the past thirty years or so we have placed emphasis on test automation and checking. In the New Model for Testing, I call this 'Applying'. We have paid much less attention to the other nine - yes, nine - test activities. As a consequence, we have simple robots to run tests, but nothing much to help us to create good tests for those robots to run.
In this paper, I am attempting to identify the capabilities of the tools we need in the future.
The tools we use in testing today are limited by the approaches and processes we employ. Traditional testing is document-centric and aims to reuse plans as records of tester activity. That approach and many of our tools are stuck in the past. Bureaucratic test management tools have been one automation pillar (or millstone). The other pillar – test automation tools – derive from an obsession with the mechanical, purely technical execution activity and is bounded by an assertion that many vendors still promote – that testing is just bashing keys or touchscreens which tools can do just as well.
The pressure to modernise our approaches, to speed up testing and reduce the cost and dependency on less-skilled labour means we need some new ideas. I have suggested a refined approach using a Surveying metaphor. This metaphor enables us to think differently on how we use tools to support knowledge acquisition.
The Survey metaphor requires new collaborative tools that can capture information as it is gathered with little distraction or friction. But they can also prompt the user to ask questions, to document their thoughts, concerns, observations and ideas for tests. In this vision, automated tools get a new role – supportive of tester thinking, but not replacing it.
Your pair in the exploration and testing of systems might soon be a robot. Like a human partner, they will capture the knowledge you impart. Over time they will learn how to support and challenge you and help you to navigate through your exploration or Surveying activity. Eventually, your partner will suggest ideas that rival your own. But that is still some way off.
To download the full paper, go to the Tools Knowledge Base.
First published 09/05/2013
But there are also some significant forces at play in the IT industry and I think the testing community, will be coming under extreme pressure. I summarise this change as ‘redistributed testing’: users, analysts, developers and testers will redistribute responsibility for testing by, wait for it, collaborating more effectively. Testers probably won’t drive this transition, and they may be caught out if they ignore the winds of change.
In this article, I’ll suggest what we need from the leaders in our industry, the market and our organisations. Of course, some responsibility will fall on your shoulders. Whether you are a manager or technical specialist, there will be an opportunity for you to lead the change.
SaaS works as an enabler for very rapid deployment of new functionality and deployment onto a range of devices. A bright idea in marketing in the morning can be deployed as new functionality in the afternoon and an increasing number of companies are succeeding with ‘continuous delivery’. This is the promise of SaaS.
Most organisations will have to come to terms with the new architectures and a more streamlined approach to development. The push and pull of these forces will make you rethink how software available through the Internet is created, delivered and managed. The impacts on testing are significant. If you take an optimistic view, testing and the role of testers can perhaps, at last, mature to what they should be.
In many ways, in promoting the testing discipline as some of us have done for more than twenty years, we have been too successful. There is now a sizable testing industry. We have certification schemes, but the schemes that were a step forwards fifteen years ago, haven’t advanced. As a consequence, there are many thousands of professional testers, certified only to a foundation level who have not developed their skills much beyond test script writing, execution and incident logging. Much of what these people do are basically ‘checking’ as Michael Bolton has called it.
Most checking could be automated and some could be avoided. In the meantime, we have seen (at last) developer testing begin to improve through their adoption of test-driven and behaviour-driven approaches. Of course, most of the testing they do is checking at a unit level. But this is similar to what many POFTs spend much of their time doing manually. Given that most companies are looking to save money, it’s easy to see why many organisations see an opportunity to reduce the number of POFTs if they get their developers to incorporate automated checking into their work through TDD and BDD approaches.
As the developers have adopted the disciplines and (mostly free) tools of TDD and BDD, the testers have not advanced so far. I would say, that test innovation tends to be focused on the testers’ struggle to keep pace with new technologies rather than insights and inventions that move the testers’ discipline forward. Most testing is still manual, and the automated tests created by test teams (usually with expensive, proprietary tools) might be better done by developers anyway.
In the test management space, one can argue that test management is a non-discipline, that is, there is no such thing as test management, there’s just management. If you take the management away from test management – what’s left? Mostly challenges in test logistics – or just logistics – and that’s just another management discipline isn’t it?
What about the fantastic advances in automation? Well, test execution robots are still, well, just robots. The advances in these have tracked the technologies used to build and deliver functionality – but pretty much that’s all. Today’s patterns of test automation are pretty much the same as those used twenty or more years ago. Free test automation frameworks are becoming more commonly used, especially for unit testing. Free BDD tools have emerged in the last few years, and these are still developer focused but expect them to mature in the next few years. Tools to perform end-to-end functional tests are still mostly proprietary, expensive and difficult to succeed with.
The test management tools that are out there are sophisticated, but they perform only the most basic record keeping. Most people still use Excel and survive without test management products that only support the clerical test activities and logistics and do little to support the intellectual effort of testers.
The test certification schemes have gone global. As Dorothy Graham says on her blog the Foundation met its main objective of “removing the bottom layer of ignorance” about software testing. Fifteen years and 150,000+ certificate awards later, it does no more than that. For many people, it seems that this ‘bottom layer of knowledge’ is all they may ever need to get a job in the industry. The industry has been dumbed-down.
But, continuous delivery is a machine that consumes requirements. For the rapid output of continuous delivery to be acceptable, the quality of requirement going into that machine must be very high. We argue that requirements must be trusted, but not perfect.
The core of the redistribution idea is that the checking that occupies much of the time of testing teams (who usually get involved late in projects) can be better done by developers. Relieving the testers of this burden gives them time to get involved earlier and to improve the definition of software before it is built. Our proposal is that testers apply their critical skills to the creation of examples that illustrate the behaviour of software in use in the requirements phase. Examples (we use the term business stories) provide feedback to stakeholders and business analysts to validate business rules defined in requirements. The outcome of this is what we call trusted requirements.
In the Business Story Pocketbook, we define a trusted requirement as “… one that, at this moment in time, is believed to accurately represent the users’ need and is sufficiently detailed to be developed and tested.” Trusted requirements are specified collaboratively with stakeholders, business analysts, developers and testers involved.
Developers, on receipt of validated requirements and business stories can use the stories to drive their TDD approach. Some (if not all) of these automated checks form the bulk of regression tests that are implemented in a Continuous Integration regime. These checks can then be trusted to signal a broken build. As software evolves, requirements change; stories and automated checks change too. This approach, sometimes-called Specification by Example depends on accurate specifications (enforced by test automation) for the lifetime of the software product. Later (and fewer) system testers have reduced time to focus on the more subtle types of problem, end to end and user experience testing.
The deal is this: testers get involved earlier to create scenarios that validate requirements, and that developers can automate. Improving the quality of requirements means the target is more stable, developers produce better code, protected by regression tests. Test teams, relieved of much of the checking and re-testing are smaller and can concentrate on the more subtle aspects of testing.
With regards to the late testing in continuously delivering environments, testers are required to perform some form of ‘health check’ prior to deployment, but the days of teams spending weeks to do this are diminishing fast. We need fewer, much smarter testers working up-front and in the short time between deployment and release.
The pace of technical change is so high that the old way of testing just won’t cut it. Some teams are discovering they can deliver without testers at all. The challenge of testing is perceived (rightly or wrongly) to be one of speed and cost (even though it’s more subtle than that of course). Testers aren’t being asked to address this challenge because it seems more prone to a technical solution and POFTs are not technical.
But the opportunities are there: to get involved earlier in the requirements phase; to support developers in their testing and automation; to refocus testing away from manual checking towards exploratory testing; to report progress and achievement against business goals and risks, rather than test cases and bug reports.
Some of the clichés of testing need to be swept away. The old thinking is no longer relevant and may be career limiting. To change will take some courage, persistence and leadership.
Developers write code; testers test because developers can’t: this mentality has got to go. Testing can no longer be thought of as distinct from development. The vast majority of checking can be implemented and managed by development. One potential role of a tester is to create functional tests for developers to implement. The developers, being fluent in test automation, implement lower level functional and structural tests using the same test automation. Where developers need coaching in test design, then testers should be prepared to provide it.
Testers don’t own testing: testing is part of everyone’s job from stakeholder, to users, to business analysts, developers and operations staff. The role of a tester could be that of ‘Testmaster’. A testmaster provides assurance that testing is done well through test strategy, coaching, mentoring and where appropriate, audit and review.
Testing doesn’t just apply to existing software, at the end: testing is an information provision service. Test activity and design is driven by a project’s need to measure achievement, to explore the capabilities, strengths and weaknesses so decisions can be made. The discipline of test applies to all artefacts of a project, from business plans, goals, risks, requirements and design. We coined the term ‘Project Intelligence’ some years ago to identify the information testers provide.
Testing is about measuring achievement, rather than quality: Testing has much more to say to stakeholders when its output describes achievement against some meaningful goal, than alignment to a fallible, out of date, untrusted document. The Agile community have learnt that demonstrating value is much more powerful than reporting test pass/fails. They haven’t figured how to do it of course, but the pressure to align Agile projects with business goals and risks is very pronounced.
Henry Kissinger said, “A leader does not deserve the name unless he is willing occasionally to stand alone”. You might have to stand alone for a while to get your view across. Dwight D Eisenhower gave this definition: “Leadership is the art of getting someone else to do something you want done because he wants to do it”.
Getting that someone else to want to do it might yet be your biggest challenge.
First published 23/05/2016
If you are looking to learn Python, the most amazing programming language ever created (IMHO)...
You could do a lot worse than buy a copy of my little Python Pocketbook, Lean Python.
Here's a review of the book:
If you want to buy from Amazon - that's fine, but it will be faster (and save me a few quid) to buy direct from me. Just use the contact form to place an order.
First published 21/07/2015
This year, I'm running a half-day tutorial on the Tuesday 3 November at EuroSTAR in Maastricht. Visit the conference page here..
I've created a short elevator pitch to promote the session.
First published 18/05/2012
In London, on 18 May I presented a keynote to the Testing and Finance conference. I've been asked for the slides of that talk, so I have uploaded them here. The talk was mostly based on two articles that I originally wrote for Atlassian, and you can find the text of those articles in the blog here: http://gerrardconsulting.com/index.php?q=node/602
Note that the presentation introduces some broader suggestions about influences on the future of testing and testers, including the increasing adoption of continuous delivery.