Paul Gerrard

First published 25/11/2011

Some time ago, Tim Cuthbertson blogged “how I Replaced Cucumber With 65 Lines of Python.” I recently commented on the post and I've expanded on those comments a little here.

I share Tim's frustrations with Cucumber. I think the reuse aspect of the step definitions has some value, but that value is limited. I've heard of several sites having literally thousands of feature files and step definitions and no way to manage them systematically. A bit of a nightmare perhaps.

To address the 'specification by example isn't enough' challenge – SBE isn't enough and I demonstrate/discuss that here. Although some trivial requirements can be SBE – most can't, so you need a separate requirement statement to supplement scenarios/examples to fully describe the requirement.

This doesn't sound very Agile, but I'm not talking Agile here necessarily. I understand that some teams can live with minimalist stories and the spec is the code. I'm talking about teams that require an accurate definition of the requirement and want to drive the creation of tests from stories and scenarios. This need could apply to all project styles and not just Agile.

Gojko Adzic talks about the need for 'Key Examples' in his Specification by Example book. When I spoke to Gojko not too long ago and suggested more specification content beyond examples was usually required – he agreed. If this is true, that doesn't mean that we need bloated requirements documents. The level of detail in a requirement (as captured by a BA) can be quite compact, because the precision of a business rule doesn't need heavy explanation – the scenarios and tabulated examples (if needed) do that for us.

Successful execution of 'key examples' are a necessary but not usually sufficient acceptance criteria. Developers definitely need more tests to cover edge cases, for example. (User) acceptance requires end to end tests and probably combinations of examples in sequence to fully satisfy the business users. (Although these types of tests are likely to be manual rather than automated).

Some time ago, we wrote (a small amount of) code to generate Python unitttest code directly from stories, scenarios and example tables and it works fine. (All we need are different language templates to generate xunit code in other programming languages). The test code may be xunit format – but the story/scenarios define the content of the tests. xUnit code could drive any style of test in theory. We're also experimenting with generating Robot Framework code and HTML Fitnesse tables directly. All seems feasible to me and in principle, all that's required is a template to generate the correctly formatted output. Additional setup/teardown code and fixtures are held in pre-existing code.

Our SaaS product SP.QA can be used to capture requirements and the range of stories/scenarios that example them. Since the requirements and stories are managed in a database and the test code is generated, the developers (or testers) only need to manage a single class or function for each story to implement the tests.

This has the distinct advantage that BAs can treat epic stories/sagas as requirements and build their hierarchical view. Stories that identify features and scenarios that example them can be refined over time. When they are 'trusted' the test code can be generated.

We're offering a commercial product, but I think even in the open source domain, the days of plain text story writing are numbered. We think all requirements tools will move in this direction. In the future, requirements tools will capture requirements and stories/scenarios that example those requirements so that they can be used to drive requirements reviews, to be a starting point for acceptance tests and be used to generate test code for developers and system testers.

When business analysts and users get their hands on these tools, then BDD will really take off.

Tags: #BDD #Python #Cucumber #xUnit #Fitnesse #RobotTestFramework

Paul Gerrard My linkedin profile is here My Mastodon Account