Live Specifications: From Requirements to Automated Tests and Back

First published 08/01/2013

Many thanks to the Eurostar folk allowing me to present the webinar, “Live Specifications: From Requirements to Automated Tests and Back”. This talk describes how we think companies can implement continuous delivery and live specifications using the Behaviour-Driven Development approach and redistributed testing.

There were some interesting questions posed at the time some of which I answered, but several I didn't. I've finally got around to writing some notes against each. See below...

Q: How to convince client to agile?

Well, the reasons for going Agile are well understood now. The critical arguments (from my point of view) are:

Q: BDD works very well for testing stories and component testing, e.g. web services, but when considering end-to-end scenario testing, BDD gets very messy. What do you suggest when testing scenario based end-to-end tests?

True enough. The current BDD tools focus very much at a story or feature-level testing and automation but do not support the ‘end to end’ testing required to test integration and consistency between features.

The BDD approach needs to align more with the increasing use of workflow and story-boarding approaches used by teams building non-trivial systems. Essentially, feature test scenarios need to be mapped to steps in a workflow so they become an executable end to end test procedure. Now, this is relatively easy with manual tests run by intelligent testers. But there are some specific challenges with automating these end to end tests.

Existing BDD tools can usually work with GUI automation tools, but more often, they are used with unit test frameworks. In general, they fit more naturally with developers’ feature testing than system or user acceptance testing. In order to create end to end tests:

Q: Does a BDD dictionary replace an acceptance test plan ?

In a word, no. I’m not sure I understand the question. In the Business Story Method and supporting technology, the dictionary has several core features:

Q: Could you name again the BDD tools you mentioned as an example at the beginning of the webinar?

There is a comprehensive description of BDD on the http://behaviourdriven.org website. The section on Implementations provides a list of the common tools, although I think the list is growing steadily. http://behaviourdriven.org/Implementations

The tools I mentioned in the talk were Cucumber, Specflow and FitNesse, I believe.

Q: I have tried to google DeFOSPAM but got no hits.

It’s there for me 8O). Try http://gerrardconsulting.com/index.php?q=node/604 – this text is an extract from the pocketbook.

Q: In BDD, when scenarios become more and more, how to organize them effectively?

It’s an obvious problem with stories, once you create non-trivial systems using them. We have heard tales of companies managing 20,000 stories in separate feature files in a single directory, and that’s a scary prospect. One way with existing ‘file-based’ products would be to create a consistent file naming convention and directory structure. But this is really cumbersome.

The Relish website http://relishapp.com offers a simple repository service for your stories (and other documentation) and is free for ‘open’ projects. Expect other services like this to spring up.

Q: In regards to production vs going live, how to you avoid customers being exposed to changes in production?

Increasingly, the continuous delivery model implies that software and changes are delivered into a production (or production-like) environment for testing prior to go live. There are several approaches that can be used including:

In all cases, a reliable and rapid roll-back facility is required to avoid the wrath of users affected by a faulty release. These approaches are discussed in Jez Humble and David Farley’s excellent book, “Continuous Delivery”.

Q: So you suggest testers to be more like business analysts and to help product owners to refine requirements, right?

I’ve suggested that business analysts and testers should look to using a critical thinking discipline to prepare stories and scenarios to illustrate features, derived from requirements. Scenarios can be used to drive reviews of requirements by exampling them. The DeFOSPAM approach is a simple method of deriving stories and scenarios for this purpose.

Now, testers are not alone in having critical thinking skills or a touch of scepticism. It may be that business analysts perform the story-generation and requirements validation role. But the scenarios can also define a set of required acceptance tests for features. In this case, the testers might be better placed to create them.

Our recommendation is that stories and scenarios are created to achieve this dual goal. By so doing, requirements are improved, developers get concrete examples of features in use and a minimal set of test cases they can choose to automate through a BDD approach. In principle, all feature checks could be provided as scenarios for developers to automate so that later testers can be relieved of some or all of this error-prone chore.

There is an opportunity for testers to perform early story and scenario preparation to reduce the amount of later manual checking. The time saved may mean fewer testers are needed. What better incentive is there to get involved early?

Q: What if the requirements are potential needs that users might require? No one will know until it is out in the real world.

If a requirement describes a ‘nice to have’, sham, vanity or useless feature and it is not challenged at the requirements stage, then it is possible that it will be developed and delivered, tested and deployed – and never used. This is more likely in a structured or outsourced development. In an agile team, one would have thought this is much less likely.

Regardless of approach however, the task of creating a feature description and scenarios is like a paper prototype. The “as a.. I want … so that …” construct drives out an owner, a need and a purpose for the feature. It challenges the stakeholder motives: “who will use it? What does it do? And why?” The chances are this highlights a dud feature, or at least challenges its viability.

If a feature makes it through this stage, then the “given … when … then …” scenarios provide an opportunity challenge the feature using real-world concrete examples. The sceptical analyst, developer or tester can use the scenarios to ask, “what if…?” and suggest some good, bad, contradictory, anomalous or just plain meaningless consequences.

The act of story creation can be viewed as a ‘thought experiment’ to speculate on how the feature may stand up to real world operation. It is, in effect, a proxy or prototype, much cheaper to test than a delivered feature for acceptance.

Q: What if you have a vague requirement? What if requirements change over time?

In the case of a vague requirement and the developer has nothing better to work from, then the developer is likely to guess at the requirement, or make unsafe assumptions, or invent solutions that bear little relation to the real need. Perhaps they know the business domain and deliver something useful. But the risk of getting it completely wrong is probably too high to bear.

Creating stories to illustrate a feature specified in a requirement triggers some difficult questions for stakeholders. The DeFOSPAM mnemonic represents seven key questions:

The DeFOSPAM process forces the stakeholder to articulate a requirement more clearly.

To the question, “What if the requirement changes over time?” one has to ask in response, “Why does it change?”

In some cases, the business need changes. Well, there’s not much we can do about that one except negotiate the changes and cost of change with stakeholders. More often, requirements change because the perception of the need changes. One could look at a software project as a learning experience for stakeholders. They might propose some vague requirements initially, and wait for the developers to deliver some software for the stakeholders to evaluate. Having some experience of a solution in use, the stakeholders then say, “that’s great, but what I really want is…”

This process works, but my, it is frustrating for everyone on the team, especially the developers and testers whose time is wasted. How can we shortcut this process? By giving stakeholders examples of the proposed system in use through a prototype, wireframes and examples. Creating stories and scenarios coupled with steps in a workflow, with perhaps wireframes to give them a feel for what features might look like can trigger the same learning process, but at a much lower cost.

So, the answer is really to accelerate the learning process by providing rapid, meaningful feedback through example and improve the requirements and associated stories so they are trusted. Trusted does not mean perfect. In the Business Story Pocketbook (p 31), “A trusted requirement is one that, at this moment in time, is believed to accurately reflect the users’ need, and sufficiently detailed to be developed and tested”.

Q: What is the best tool for requirement management? What are the most important features a tool should have to be used in SDLC?

There are many, many RM tools out there, and it would be unreasonable to promote any one of them, and requirements can comprise many different models as well as the most common textual description. But from the ‘Live Specification’ point of view and covering functional requirements only (and excluding the obvious needs to be easy to use, flexible etc.) these features seem to be key:

Q: How will the business see “more testing” in the initial phase? How to convince the business that tester will now start “testing” requirements and less integration tests (after the development team delivers)?

I’ve tried to explain above how the exampling and validation of requirements using business stories has some significant benefits. The trade-off is that it requires a change of emphasis in the behaviours of analysts, developers and testers and collaboration between all three. But you know what? Productive Agile teams have reminded us that software development is most effective when it is treated as a collaborative process.

The change required is that Testers need to collaborate with both business analysts (to be involved earlier and contribute their critical skills to creating stories and scenarios) and with developers (to provide coherent scenarios for automation of feature checks).

The payoffs are significant:

There are an increasing number of companies moving towards continuous delivery, live specification, BDD and redistributed testing etc. To my knowledge, there are no objective, comparative studies or metrics demonstrating benefits and value for money of these approaches. I doubt there ever will be. But a growing number of practitioners are acquiring positive experiences of these approaches and environments. Well-known software gurus and fast-growing service companies are promoting them with vigour.

I have to say, with this momentum and backing, you should expect these methods to proliferate over the next few years.

 

Tags: #BDD #redistributedtesting #behaviour-drivendevelopment #continuousdelivery #continuousintegration #continuoustesting #redistributingtesting

Paul Gerrard My linkedin profile is here My Mastodon Account