Paul Gerrard

My experiences, opinions in the Test Engineering business. I am republishing/rewriting old blogs from time to time.

First published 18/05/2012

In London, on 18 May I presented a keynote to the Testing and Finance conference. I've been asked for the slides of that talk, so I have uploaded them here. The talk was mostly based on two articles that I originally wrote for Atlassian, and you can find the text of those articles in the blog here: http://gerrardconsulting.com/index.php?q=node/602

Note that the presentation introduces some broader suggestions about influences on the future of testing and testers, including the increasing adoption of continuous delivery.



Tags: #redistributionoftesting #futureoftesting

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 15/07/2013

See below the four presentations given by Paul at the World Conference on Next Generation Testing held in Bangalore, India between 8th and 12th July 2013.

Tags: #nextgentesting

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 12/04/2012

 

In this approach, the technique involves taking a requirement and identifying the feature(s) it describes. For each feature, a story summary and a series of scenarios are created and these are used to feedback examples to stakeholders. In a very crude way, you could regard the walkthrough of scenarios and examples as a ‘paper-based’ unit or component test of each feature.

What do you mean ‘unit or component test’?

A story identifies a feature and includes a set of examples that represent its required behaviour. It is not a formal component specification, but it summarizes the context in which a feature will be used and a set of business test cases. Taken in isolation, the story and scenarios, when compared with the requirement from which it is derived, provide a means of validating the understanding of the requirement.

Rather than test the functionality of a feature (which is done later, by developers usually), the story and scenarios test the requirement itself. When anomalies are cleared up, what remains are a clarified requirement and a story that identifies a feature that provides a set of clarifying examples. The definition of the identified feature is clarified and validated, in isolation – just like a component can be tested and trusted, in isolation.

The scenarios are limited in scope to a single feature but taken together, a set of stories validates the overall consistency and completeness of a requirement with respect to the feature(s) it describes.

Creating Stories and Scenarios – DeFOSPAM

DeFOSPAM is the mnemonic the Business Story Method™ uses to summarise the seven steps used to create a set of stories and scenarios for a requirement that allow us to comprehensively validate our understanding of that requirement.
  • Definitions
  • Features
  • Outcomes
  • Scenarios
  • Prediction
  • Ambiguity
  • Missing
The number of features and stories created for a requirement are obviously dependent on the scope of a requirement. A 100 word requirement might describe a single system feature and a few scenarios might be sufficient. A requirement that spans several pages of text might describe multiple features and require many stories and tens of scenarios to fully describe. We recommend you try and keep it simple by splitting complex requirements.

D – Definitions

If agreement of terminology, or feature descriptions cannot be attained, perhaps this is a sign that stakeholders do not actually agree on other things? These could be the goals of the business, the methods or processes to be used by the business, the outputs of the project or the system features required. A simple terminology check may expose symptoms of serious flaws in the foundation of the project itself. How powerful is that?

On the one hand, the need for definitions of terms used in requirements and stories arises as they are written and should be picked up by the author as they write. However, it is not uncommon for authors to be blind to the need for definitions as they might be using language and terminology that is very familiar to them. Scanning requirements and stories to identify the terms and concepts that need definition or clarification by subject matter experts is critical.

Getting the terminology right is a very high priority. All subsequent communication and documentation may be tainted by poor or absent definitions. The stories and scenarios created to example requirements must obviously use the same terminology as the requirement so it is critical to gain agreement early on.

Identify your sources of definitions. These could be an agreed language dictionary, source texts (books, standards etc.) and a company glossary. The company glossary is likely to be incomplete or less precise than required. The purpose of the definitions activity is to identify the terms needing definition, to capture and agree terminology and to check that the language used in requirements and stories is consistent.

On first sight of a requirement text, underline the nouns and verbs and check that these refer to agreed terminology or that a definition of those terms is required.

  • What do the nouns and verbs actually mean? Highlight the source of definitions used. Note where definitions are absent or conflict.
  • Where a term is defined, ask stakeholders – is this the correct, agreed definition? Call these ‘verified terms’.
  • Propose definitions where no known definition exists. Mark them as ‘not verified by the business’. Provide a list of unverified terms to your stakeholders for them to refine and agree.
When you start the process of creating a glossary, progress will be slow. But as terms are defined and agreed, progress will accelerate rapidly. A glossary can be viewed as a simple list of definitions, but it can be much more powerful than that. It’s really important to view the glossary as a way of making requirements both more consistent and compact – and not treat glossary maintenance as just an administrative chore.
  • A definition can sometimes describe a complex business concept. Quite often in requirements documents, there is huge scope for misinterpretation of these concepts, and explanations of various facets of these concepts appear scattered throughout requirements documents. A good glossary makes for more compact requirements.
  • Glossary entries don’t have to be ‘just’ definitions of terminology. In some circumstances, business rules can be codified and defined in the glossary. A simple business rule could be the validation rule for a piece of business data, for example a product code. But it could be something much more complex, such as the rule for processing invoices and posting entries into a sales ledger.
Glossary entries that describe business rules might refer to features identified in the requirements elsewhere. The glossary (and index of usage of glossary entries) can therefore provide a cross-reference of where a rule is used and associated system feature is used.

F – Features – One Story per Feature

Users and business analysts usually think and document requirements in terms of features. A feature is something the proposed system needs to do for its user and helps the user to meet a goal or supports a critical step towards that goal.

Features play an important part in how business users, wishing to achieve some goal, think. When visualising what they want of a system, they naturally think of features. Their thoughts traverse some kind of workflow where they use different features at each step in the workflow. ‘… I’ll use the search screen to find my book, then I’ll add it to the shopping cart and then I’ll confirm the order and pay’.

Each of the phrases, ‘search screen’, ‘shopping cart’, ‘confirm the order’ and ‘pay’ sound like different features. Each could be implemented as a page on a web site perhaps, and often features are eventually implemented as screen transactions. But features could also be processes that the system undertakes without human intervention or unseen by the user. Examples would be periodic reports, automated notifications sent via email, postings to ledgers triggered by, but not seen by, user activity. Features are often invoked by users in sequence, but features can also collaborate to achieve some higher goal.

When reading a requirement and looking for features, sometimes the features are not well defined. In this case, the best thing is to create a story summary for each and move on to the scenarios to see how the stories develop.

Things to look for:

  • Named features – the users and analysts might have already decided what features they wish to see in the system. Examples could be ‘Order entry’, ‘Search Screen’, ‘Status Report’.
  • Phrases like, ‘the system will {verb} {object}’. Common verbs are capture, add, update, delete, process, authorise and so on. Object could be any entity the system manages or processes for example, customer, order, product, person, invoice and so on. Features are often named after these verb-object phrases.
  • Does the requirement describe a single large or complex feature or can distinct sub-features be identified? Obviously larger requirements are likely to have several features in scope.
  • Are the features you identify the same features used in a different context or are they actually distinct? For example, a feature used to create addresses might be used in several places such as adding people, organisations and customers.

O – One Scenario per Outcome

More than anything else, a requirement should identify and describe outcomes. An outcome is the required behaviour of the system when one or more situations or scenarios are encountered. We identify each outcome by looking for requirements statements that usually have two positive forms:
  • Active statements that suggest that, ‘…the system will…’
  • Passive statements that suggest that ‘…valid values will …’ or ‘…invalid values will be rejected…’ and so on.
Active statements tend to focus on behaviours that process data, complete transactions successfully and have positive outcomes. Passive statements tend mostly to deal with data or state information in the system.

There is also a negative form of requirement. In this case, the requirement might state, ‘…the system will not…’. What will the system not do? Usually, these requirements refer to situations where the system will not accept or proceed with invalid data or where a feature or a behaviour is prohibited or turned off, either by the user or the state of some data. In almost every case, these ‘negative requirements’ can be transformed into positive requirements, for example, ‘not accept’ could be worded as ‘reject’ or possibly even as ‘do nothing’.

You might list the outcomes that you can identify and use that list as a starting point for scenarios. Obviously, each unique outcome must be triggered by a different scenario. You know that there must be at least one scenario per outcome.

There are several types of outcome, of which some are observable but some are not.

Outputs might refer to web pages being displayed, query results being shown or printed, messages being shown or hard copy reports being produced. Outputs refer to behaviour that is directly observable through the user interface and result in human-readable content that is visible or available on some storage format or media (disk files or paper printouts).

Outcomes often relate to changes of state of the system or data in it (for example, updates in a database). Often, these outcomes are not observable through the user interface but can be exposed by looking into the database or system logs perhaps. Sometimes outcomes are messages or commands sent to other features, sub-systems or systems across technical interfaces.

Often an outcome that is not observable is accompanied by a message or display that informs the user what has happened. Bear in mind, that it is possible that an outcome or output can be ‘nothing’. Literally nothing happens. A typical example here would be the system’s reaction to a hacking attempt or selection of a disabled menu option/feature.

Things to look out for:

  • Words (usually verbs) associated with actions or consequences. Words like capture, update, add, delete, create, calculate, measure, count, save and so on.
  • Words (verbs and nouns) associated with output, results or presentation of information. Words like print, display, message, warning, notify and advise.

S – Scenarios – One Scenario per Requirements Decision

We need to capture scenarios for each decision or combination of decisions that we can associate with a feature.

Often, the most common or main success scenario is described in detail and might be called the ‘main success’ or ‘default’ scenario. (In the context of use cases, the ‘main success scenario’ is the normal case. Variations to this are called extensions or scenarios). The main success scenarios might also be called the normal case, the straight-through or plain-vanilla scenario. Other scenarios can represent the exceptions and variations.

Scenarios might be split into those which the system deals with and processes, and those which the system rejects because of invalid or unacceptable data or particular circumstances that do not allow the feature to perform its normal function. These might be referred to as negative, error, input validation or exception condition cases.

The requirement might present the business or technical rules that govern the use of input or stored data or the state of some aspect of the system. For example, a rule might state that a numeric value must lie within a range of values to be treated as valid or invalid, or the way a value is treated depends on which band(s) of values it lies in. These generalised rules might refer to non-numeric items of data being classified in various ways that are treated differently.

A scenario might refer to an item of data being treated differently, depending on its value. A validation check of input data would fall into this category (different error messages might be given depending on the value, perhaps). But it might also refer to a set of input and stored data values in combination. A number of statements describing the different valid and invalid combinations might be stated in text or even presented in a decision-table.

Things to look out for:

  • Phrases starting (or including) the words ‘if’, ‘or’, ‘when’, ‘else’, ‘either’, ‘alternatively’.
  • Look for statements of choice where alternatives are set out.
  • Where a scenario in the requirement describes numeric values and ranges, what scenarios (normal, extreme, edge and exceptional) should the feature be able to deal with?

P – Prediction

Each distinct scenario in a requirement setting out a situation that the feature must deal with, should also describe the required outcome associated with that scenario. The required outcome completes the definition of a scenario-behaviour statement. In some cases, the outcome is stated in the same sentence as the scenario. Sometimes a table of outcomes is presented, and the scenarios that trigger each outcome are presented in the same table.

A perfect requirement enables the reader to predict the behaviour of the system’s features in all circumstances. The rules defined in the requirements, because they generalise, should cover all of the circumstances (scenarios) that the feature must deal with. The outcome for each scenario will be predictable.

Now, of course, it is completely unrealistic to expect a requirement to predict the behaviour in all possible situations because most situations are either not applicable or apply to the system as a whole, rather than a single feature.

However, where scenarios are identifiable, the need must be to predict and associate an outcome with those scenarios.

When you consider the outcomes identified on the Outcomes stage, you might find it difficult to identify the conditions that cause them. Sometimes, outcomes are assumed or a default outcome may be stated but not associated with scenarios in the requirements text. These ‘hanging’ outcomes might be important but might never be implemented up by a developer. Unless, that is, you focus explicitly on finding these hanging outcomes.

Things to look out for:

  • Are all outcomes for the scenarios you have identified predictable from the text?
  • If you cannot predict an outcome try inventing your own outcomes – perhaps a realistic one and perhaps an absurd one and keep a note of these. The purpose of this is to force the stakeholder to make a choice and to provide clarification.

A – Ambiguity

The Definitions phase is intended to combat the use of ambiguous or undefined terminology. The other major area to be addressed is ambiguity in the language used to describe outcomes.

Ambiguity strikes in two places. Scenarios identified from different parts of the requirements appear to be identical but have different or undefined outcomes. Or two scenarios appear to have the same outcomes, but perhaps should have different outcomes to be sensible.

There are several possible anomalies to look out for:

  • Different outcomes imply different scenarios but it appears you can obtain the same outcome with multiple scenarios. Is something missing from the requirement?
  • It is possible to derive two different outcomes for the same scenario. The requirement is ambiguous.
In general, the easiest way to highlight these problems to stakeholders is to present the scenarios/outcome combinations as you see them and point out their inconsistency or duplication.

Look out for:

  • Different scenarios that appear to have identical outcomes but where common sense says they should differ.
  • Identical scenarios that have different outcomes.

M – Missing

If we have gone through all the previous steps and tabulated all of our glossary definitions, features, scenarios and corresponding outcomes we perform a simple set of checks as follows:
  • Are all terms, in particular nouns and verbs defined in the glossary?
  • Are there any features missing from our list that should be described in the requirements? For example, we have create, read and update features, but not delete feature
  • Are there scenarios missing? We have some but not all combinations of conditions identified in our table
  • Do we need more scenarios to adequately cover the functionality of a feature?
  • Are outcomes for all of our scenarios present and correct?
  • Are there any outcomes that are not on our list that we think should be?

Workshops

Story-driven requirement validation, being based on the DeFOSPAM checklist is easily managed in a workshop format in three phases. Requirements might be clustered or chunked into selected groups to be reviewed. It is helpful if all of the terms requiring definition, clarification or agreement are distributed ahead of the session, so the stakeholders responsible for the definitions can prepare ahead of the meeting.

At the workshop or review meeting, each requirement is considered in turn and the discussion of the stories derived from it is performed:

  • Firstly, consider the definitions to be captured and agreed. The group need to consider the definition of every term individually but also in the context of other related terms as a self-consistent group.
  • For each feature, the scenarios are considered one by one. Where there are suspected omissions, ambiguities, these are discussed and corrected as necessary.
  • The scenarios for a feature are considered as a set: have enough scenarios been documented to understand the requirement? Do they provide enough spread? Do they cover the critical situations?
  • The requirement, stories and scenarios are considered as a whole: is the requirement clear, complete and correct? Are all features identified and all requirements addressed or described by one or more stories? Can the requirements and stories be trusted to proceed to development?
Usually, discrepancies in requirements, definitions and stories can be resolved quickly in the meeting and agreed to.

The text of this post has been extracted from The Business Story Pocketbook written by Paul Gerrard and Susan Windsor.

Tags: #DeFOSPAM #businessstory #requirementsvalidation

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 28/11/2012

Last Friday, I was happy to present to the Skillsmatter Agile Testing and BDD Exchange at 'The Crypt'. Thanks to Gojko Adzic for inviting me and chairing the day.

You can see the video of my talk here. Afraid Skillsmatter don't allow embedding the video.

The significance of the talk is that, although I've been writing about my frustration with testing (Testing is in a Mess) and the Redistribution of Testing for 18 months or so, last week's talk is the first that makes a coherent argument for redistribution of testing in non-Agile environments.

I positioned the talk as a sales pitch to corporates but being presented to Agile devotees, so please excuse the (I'm Agile, honestly guvnor) caveats at the beginning.

Tags: #agile #businessstorymethod #BusinessStoryManager #BDD #redistributionoftesting

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 01/07/2015

At the end of May I spent a very pleasant few days in Minsk, Belarus at the SQA Days conference. Many thanks to Vladislav Orlikov for inviting me and giving me the opportunity to visit a new country and meet a lot of very nice and smart people.

Vlad sent me a link to the video of my Keynote and you can see it below. The background noise is mostly the sound of the simultaneous translation, which diminishes a few minutes into the video.

How to Test the Internet of Everything (in English) from Vlad Orlikov on Vimeo.

Vlad also used a PDF I sent him to print 500 copies of The Tester's Pocketbook to give to every attendee. Books are highly regarded in Belarus and for 24 hours I was extremely popular and had to sign (I estimate) 150-200 of them!

Tags: #IOE #IOT #InternetofEverything #InternetofThings

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 17/03/2016

The Software Tools Market is Changing

  • The Digital revolution means software must be aligned much more precisely with business
  • Customers get almost daily mobile app updates; they want the same for their business applications
  • Continuous delivery, DevOps, Shift-Left and pervasive automation are the means to succeed
  • New processes, disciplines and the tumbling walls between silos mean tools are essential
  • The automation challenge has moved from selection and implementation of 1-3 tools for each discipline to selection and implementation of 20-30 tools for a team in a DevOps regime
  • Tools need to be integrated technically, but the integration of team disciplines is just as important.

The Need for a Tools Directory

For as long as the web has existed, there have been websites that provide lists of references to tools that support, for example, test automation. These web pages and sites have been set up by individuals, wishing to share their knowledge of software tools for their own communities.

It's a burdensome task to create and maintain these lists. Vendors move webpages around, they rename tools, they merge and split tool functionality, they add new tools and new vendors and tools are popping up all the time. It's really hard to maintain the accuracy of lists like these. If you look around the various websites that provide such lists, this is what you tend to find:

  • Listings do not provide much detail beyond simple categorization, e.g. 'Web' or 'Mobile' test tools
  • Invariably, the lists are incomplete. Common tools are listed; less well-known tools are often missing
  • Most listings are dominated by proprietary tools. Open source tools are less well-represented, although some 'free tools' listings do exist, they are still incomplete
  • Many tools have functionality that spans multiple categories. Some are available in proprietary, some are open source and deployed on workstations, servers or SaaS platforms. Tools might be listed in multiple categories, but usually not
  • Tools listings often only provide a link to a vendor web page for the tool and little else. Forums, training, supporting service companies or contractors are not usually listed and cannot be searched
  • Tools cannot be compared with respect to functionality, licensing, platforms or integrations
  • There are no tools usage statistics available; we have to rely on vendor marketing to gauge their popularity
  • Not enough information, too much advertising.

The Tools Knowledge Base (tkbase.com)

The Tools Knowledge Base is a free-to-use service providing information on tools, vendors and the consultants and service companies that support them.

  • A searchable directory of tools: Our focus is (broadly) DevOps, SDET, Test and collaboration. Each tool record stores limited data but links to the vendor or developer web page. This basic information and the content of the tools web page are downloaded and indexed nightly by our search engine
  • We don't replicate tool web pages: We only collect the minimal amount of information that allows us to index the tools information for searching
  • A sophisticated search engine: The Whoosh! search facility can be used to find tools matching your search criteria. It's not Google but has most of the features that Google search provides
  • A hierarchical tool type/features list: Every tool can be properly profiled and compared (currently 313 features)
  • Resources: Content such as images, videos, scripts or training content or links to other web content can be uploaded and associated with every tool or company
  • Questions & Answers: Every tool and every resource can be discussed. Registered users can post questions and answers; the notification system will send messages to users/owners of that tool
  • 18,664 (as of 17 March 2016) searchable posts: We download the content of blogs from 302 bloggerseach night. These posts are indexed and searchable. We do not store the blog posts, we only index
  • Embeddable Content: We offer a range of APIs allowing conferences, service companies and consultants to access and share our data on their own websites
  • Partnerships: Conferences, trade magazines, websites and domain experts who support us.

What do I do Now?

If you are tools user: we'd like you to register, and identify the tools you use. your tools chain will appear on your profile. If a tool you use is not in TKB, then we invite you to create it. (If you want to embed the tools you use as a list in your website - we have an API for that).

If you are a tools expert or tools service provider: please see above, plus... We are looking for people who are knowledgeable enough to review or possibly edit the features listings for the tools you know well. The features hierarchy will grow and evolve over time - help us to perfect it. We will list you as a service provider on the tools you know best. It's the least we can do.

If you are a tool vendor: we ask you to search for the tools you offer and check they are in the system and properly described. If your tools exist in the system and you want to manage the information we hold, that's fine - let us know and we can make you the administrator (after a couple of quick checks). If you want, we can maintain the data on your behalf, for a small fee. Do get in touch. Alternatively, nominate a tools expert (see above) and we'll invite them to keep your details correct.

If You Own/Contribute to an Open Source Project: We make exactly the same offer as we make to the vendors. you are free to edit the information for your tools in the same way. you might already be represented on a site like GitHub - we're just offering an extra publicity channel - it might help you reach a broader audience. People looking for tools often start their search with proprietary products but rarely see free tools listed side by side. Now is your chance.

If you are a blogger: Search for one of your recent blog posts and if you find it - your blog is already indexed in the system. If not, you can suggest the blog and register it yourself. Please note we are looking for blogs that cover our scope (DevOps, Testing, Collaboration). Company blogs are acceptable if they focus on these topic areas. Some general technology blogs might be acceptable, at our discretion. Offensive or blatantly commercial posts are not acceptable.

If your blog is indexed, let us know, we will give you credit for it on your profile.

If you maintain your own online tools listing: please get in touch. We believe we already have a more comprehensive list (in our scope) than anyone else. We offer to provide you with ad free, embeddable tool type listings for your existing site. Join us as a partner and tools expert and help us to improve the data in our system to improve the listings on your site. We currently have 183 tool type listings. All are available for free. We'll give you your own lists and track the usage data and share that with you too. Drop me a line and I can explain what we can do for you.

If you want a tools listing on your own site: Get in touch. We make the same offer as above.

Can I advertise my tool or service on your site? Of course you can. We offer competitive CPM (cost per thousand impressions) rates. Our users are tools-focused so your ads will be seen by the right people. We will of course provide monthly statistics for your perusal.

Where is the Knowledge Base?

https://tkbase.com

Please take a look and let me know what you think.



Tags: #testing #collaboration #tkb #ToolsKnowledgeBase #DevOps #SDET

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 22/06/2021

It's a common question. I need to lead, but do I have the ability to do that?

There's lots of talk about 'born leaders' but what if I'm not a born leader? Can I still succeed?

In this video, I suggest leadership can be learned and enhanced with practice and support.



Tags: #ALF

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 22/05/2014

Some weeks back I presented a webinar about the Internet of Everything, with special focus on what the concept is and how it is going to affect us. 

This was the first of the webinars based on the article series that I am writing for Tea-time with Testers further articles and webinars will be appearing soon.



Tags: #IOE #IOT #InternetofEverything

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 08/01/2013

Many thanks to the Eurostar folk allowing me to present the webinar, “Live Specifications: From Requirements to Automated Tests and Back”. This talk describes how we think companies can implement continuous delivery and live specifications using the Behaviour-Driven Development approach and redistributed testing.

There were some interesting questions posed at the time some of which I answered, but several I didn't. I've finally got around to writing some notes against each. See below...

Q: How to convince client to agile?

Well, the reasons for going Agile are well understood now. The critical arguments (from my point of view) are:
  • Breaking larger projects into smaller ones reduces the complexity and risks of managing delivery – if a small project fails – it fails fast and cheaply.
  • Agile encourages the use of autonomous multi-skilled teams who communicate well. Decision making delays are much reduced, feedback is fast so problems are corrected quicker and more cheaply.
  • An on-site customer (often known as the product owner) provides day to day direction for projects, so technical people don’t go off track, and get feedback on ‘the right thing to do’ quickly.
  • Breaking complex functionality into stories allows stories to be prioritised by business value (by the on-site customer). As a consequence, the team always delivers value as early as possible and are not distracted by lower value activities.
  • The morale of Agile teams is usually much higher than other organisations because all members are involved, well-informed and accountable for progress towards delivery.
  • Agile does not necessarily reduce the amount of rework, but brings it forward so the consequences of defects are minimised. Further, refactoring is usually an explicit task for developers in Agile teams, so the reword due to redesign and discovery of better ways to do things is visible and manageable.

Q: BDD works very well for testing stories and component testing, e.g. web services, but when considering end-to-end scenario testing, BDD gets very messy. What do you suggest when testing scenario based end-to-end tests?

True enough. The current BDD tools focus very much at a story or feature-level testing and automation but do not support the ‘end to end’ testing required to test integration and consistency between features.

The BDD approach needs to align more with the increasing use of workflow and story-boarding approaches used by teams building non-trivial systems. Essentially, feature test scenarios need to be mapped to steps in a workflow so they become an executable end to end test procedure. Now, this is relatively easy with manual tests run by intelligent testers. But there are some specific challenges with automating these end to end tests.

Existing BDD tools can usually work with GUI automation tools, but more often, they are used with unit test frameworks. In general, they fit more naturally with developers’ feature testing than system or user acceptance testing. In order to create end to end tests:

  • Each feature/scenario test requires a context to be defined by the controlling workflow OR the feature/scenario needs keywords/macros that can be called to prepare the feature for use in different scenarios. It’s not yet clear which approach is best (or even viable, without too much manual effort).
  • Existing keyword-driven test tools and frameworks map nicely to BDD. It seems to me that the keyword driven approach for navigation through (potentially complex) workflows is the most likely path to success with end to end automation. Watch out for BDD capabilities in existing frameworks e.g. Robot Framework has a BDD capability.
  • In our tool, Business Story Manager, we already have the capability to create process-paths or workflows with steps mapped to features and scenarios. We have ‘integration to Robot Framework’ on our roadmap. Other integrations will follow.
  • Overall however, we expect most automated testing to focus on feature-based checking, rather than complex end to end testing with obscure paths.

Q: Does a BDD dictionary replace an acceptance test plan ?

In a word, no. I’m not sure I understand the question. In the Business Story Method and supporting technology, the dictionary has several core features:
  • To capture terms (and concepts) that are relevant to an application and requirements. These may be business or technical terms, abbreviations or any concept, even business rules.
  • To index the usage of these terms in requirements, stories and tests so they can be traced and coverage analyses made.
  • We extend this in our tool to cover data item names in scenario outlines. Where scenarios contain <placeholders> for data provided as tabulated examples we can trace the usage of these data items and in effect create a simple, but effective data dictionary for them.
  • Overall – the goals of the dictionary are to promote a ubiquitous language, support traceability, impact analysis and provide requirements and story coverage measures that are based on the language used in the business domain.

Q: Could you name again the BDD tools you mentioned as an example at the beginning of the webinar?

There is a comprehensive description of BDD on the http://behaviourdriven.org website. The section on Implementations provides a list of the common tools, although I think the list is growing steadily. http://behaviourdriven.org/Implementations

The tools I mentioned in the talk were Cucumber, Specflow and FitNesse, I believe.

Q: I have tried to google DeFOSPAM but got no hits.

It’s there for me 8O). Try http://gerrardconsulting.com/index.php?q=node/604 – this text is an extract from the pocketbook.

Q: In BDD, when scenarios become more and more, how to organize them effectively?

It’s an obvious problem with stories, once you create non-trivial systems using them. We have heard tales of companies managing 20,000 stories in separate feature files in a single directory, and that’s a scary prospect. One way with existing ‘file-based’ products would be to create a consistent file naming convention and directory structure. But this is really cumbersome.

The Relish website http://relishapp.com offers a simple repository service for your stories (and other documentation) and is free for ‘open’ projects. Expect other services like this to spring up.

Q: In regards to production vs going live, how to you avoid customers being exposed to changes in production?

Increasingly, the continuous delivery model implies that software and changes are delivered into a production (or production-like) environment for testing prior to go live. There are several approaches that can be used including:

  • Feature toggling (http://en.wikipedia.org/wiki/Feature_toggle) essentially, a flag in the software, can be set to turn features (that may not be complete) on or off at will.
  • So called ‘dark releases’ – whereby features are released to production but are not advertised or accessible through the normal user interface, but could be accessed by testers using special urls and passwords, for example. Sometimes called ‘back doors’, these are useful for testers but are also of interest to hackers J
  • A ‘blue-green’ release implements two parallel environments in production. One is live and the other is for evaluation. When the evaluation version has been checked out, routers can be flipped to reroute traffic to the new version (and flipped back in case of problems).
  • Limited releases and Canary releases are where a software version is released to only a subset of users in a particular country, region or network or to a subset of production servers for evaluation. Users may or may not be aware of such trials. If a problem occurs, only a subset of your users are affected (and a rapid rollback can be effected).
In all cases, a reliable and rapid roll-back facility is required to avoid the wrath of users affected by a faulty release. These approaches are discussed in Jez Humble and David Farley’s excellent book, “Continuous Delivery”.

Q: So you suggest testers to be more like business analysts and to help product owners to refine requirements, right?

I’ve suggested that business analysts and testers should look to using a critical thinking discipline to prepare stories and scenarios to illustrate features, derived from requirements. Scenarios can be used to drive reviews of requirements by exampling them. The DeFOSPAM approach is a simple method of deriving stories and scenarios for this purpose.

Now, testers are not alone in having critical thinking skills or a touch of scepticism. It may be that business analysts perform the story-generation and requirements validation role. But the scenarios can also define a set of required acceptance tests for features. In this case, the testers might be better placed to create them.

Our recommendation is that stories and scenarios are created to achieve this dual goal. By so doing, requirements are improved, developers get concrete examples of features in use and a minimal set of test cases they can choose to automate through a BDD approach. In principle, all feature checks could be provided as scenarios for developers to automate so that later testers can be relieved of some or all of this error-prone chore.

There is an opportunity for testers to perform early story and scenario preparation to reduce the amount of later manual checking. The time saved may mean fewer testers are needed. What better incentive is there to get involved early?

Q: What if the requirements are potential needs that users might require? No one will know until it is out in the real world.

If a requirement describes a ‘nice to have’, sham, vanity or useless feature and it is not challenged at the requirements stage, then it is possible that it will be developed and delivered, tested and deployed – and never used. This is more likely in a structured or outsourced development. In an agile team, one would have thought this is much less likely.

Regardless of approach however, the task of creating a feature description and scenarios is like a paper prototype. The “as a.. I want … so that …” construct drives out an owner, a need and a purpose for the feature. It challenges the stakeholder motives: “who will use it? What does it do? And why?” The chances are this highlights a dud feature, or at least challenges its viability.

If a feature makes it through this stage, then the “given … when … then …” scenarios provide an opportunity challenge the feature using real-world concrete examples. The sceptical analyst, developer or tester can use the scenarios to ask, “what if…?” and suggest some good, bad, contradictory, anomalous or just plain meaningless consequences.

The act of story creation can be viewed as a ‘thought experiment’ to speculate on how the feature may stand up to real world operation. It is, in effect, a proxy or prototype, much cheaper to test than a delivered feature for acceptance.

Q: What if you have a vague requirement? What if requirements change over time?

In the case of a vague requirement and the developer has nothing better to work from, then the developer is likely to guess at the requirement, or make unsafe assumptions, or invent solutions that bear little relation to the real need. Perhaps they know the business domain and deliver something useful. But the risk of getting it completely wrong is probably too high to bear.

Creating stories to illustrate a feature specified in a requirement triggers some difficult questions for stakeholders. The DeFOSPAM mnemonic represents seven key questions:

  • What do the words of the requirement mean? (Definition)
  • What features are being specified? (Features)
  • What outcomes can be identified? (Outcomes)
  • What situations or scenarios must the feature deal with? (Scenarios)
  • Does the requirement predict sensible outcomes? (Prediction)
  • Is the text of the requirement ambiguous? (Ambiguity)
  • Has anything been left out of the requirement? (Missing)
The DeFOSPAM process forces the stakeholder to articulate a requirement more clearly.

To the question, “What if the requirement changes over time?” one has to ask in response, “Why does it change?”

In some cases, the business need changes. Well, there’s not much we can do about that one except negotiate the changes and cost of change with stakeholders. More often, requirements change because the perception of the need changes. One could look at a software project as a learning experience for stakeholders. They might propose some vague requirements initially, and wait for the developers to deliver some software for the stakeholders to evaluate. Having some experience of a solution in use, the stakeholders then say, “that’s great, but what I really want is…”

This process works, but my, it is frustrating for everyone on the team, especially the developers and testers whose time is wasted. How can we shortcut this process? By giving stakeholders examples of the proposed system in use through a prototype, wireframes and examples. Creating stories and scenarios coupled with steps in a workflow, with perhaps wireframes to give them a feel for what features might look like can trigger the same learning process, but at a much lower cost.

So, the answer is really to accelerate the learning process by providing rapid, meaningful feedback through example and improve the requirements and associated stories so they are trusted. Trusted does not mean perfect. In the Business Story Pocketbook (p 31), “A trusted requirement is one that, at this moment in time, is believed to accurately reflect the users’ need, and sufficiently detailed to be developed and tested”.

Q: What is the best tool for requirement management? What are the most important features a tool should have to be used in SDLC?

There are many, many RM tools out there, and it would be unreasonable to promote any one of them, and requirements can comprise many different models as well as the most common textual description. But from the ‘Live Specification’ point of view and covering functional requirements only (and excluding the obvious needs to be easy to use, flexible etc.) these features seem to be key:
  • The functional requirement encompasses a ‘statement of need or business rule’ (the traditional requirement) plus associated features, scenarios and examples. Think of this as ‘specification AND example’.
  • A dictionary of terminology, abbreviations, synonyms, antonyms, data names, business rules and an index of use throughout the requirements content.
  • Traceability from business goals through requirements, features scenarios and examples to allow business impact analysis. A change at any level can be traced up or down the hierarchy.
  • Reporting of requirements content in a format understandable to stakeholders, at any level of the hierarchy so that a review of goals versus requirements, requirements versus features/scenarios can be supported.
  • Integration of features and scenarios with a test execution management, test automation framework or BDD tool so that requirements content drives testing and development, and test status can be reflected up the hierarchy from scenarios through features, requirements and business goals.
  • Needless to say, accessibility to everyone in a business or project team, version and change history, access control.

Q: How will the business see “more testing” in the initial phase? How to convince the business that tester will now start “testing” requirements and less integration tests (after the development team delivers)?

I’ve tried to explain above how the exampling and validation of requirements using business stories has some significant benefits. The trade-off is that it requires a change of emphasis in the behaviours of analysts, developers and testers and collaboration between all three. But you know what? Productive Agile teams have reminded us that software development is most effective when it is treated as a collaborative process.

The change required is that Testers need to collaborate with both business analysts (to be involved earlier and contribute their critical skills to creating stories and scenarios) and with developers (to provide coherent scenarios for automation of feature checks).

The payoffs are significant:

  • Business stories provide early feedback so vague, ambiguous and incomplete requirements can be trusted to reflect the true need.
  • Business stories and scenarios provide clear ‘feature acceptance criteria’ but can also be a covering set of feature checks that can be automated by professional developers.
  • Automated feature checks support both the Behaviour- and Test-Driven Development approaches so development is less error-prone; they also provide candidate regression tests to be used for the lifetime of the feature.
  • Automated feature checking reduces the need for labour intensive and error-prone, expensive (and boring) manual checking so test teams can focus on the more subtle end to end and obscure tests.
  • Delivery timescales become more reliable and the quality of delivered software is higher.
  • Using tools that provide the traceability, better business impact analysis becomes possible – changes can be properly evaluated – at last.
  • New and meaningful test coverage measures based on the language and terminology used by stakeholders become viable.
There are an increasing number of companies moving towards continuous delivery, live specification, BDD and redistributed testing etc. To my knowledge, there are no objective, comparative studies or metrics demonstrating benefits and value for money of these approaches. I doubt there ever will be. But a growing number of practitioners are acquiring positive experiences of these approaches and environments. Well-known software gurus and fast-growing service companies are promoting them with vigour.

I have to say, with this momentum and backing, you should expect these methods to proliferate over the next few years.

 

Tags: #BDD #redistributedtesting #behaviour-drivendevelopment #continuousdelivery #continuousintegration #continuoustesting #redistributingtesting

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account