Paul Gerrard

First published 31/08/2012

In response to Kim Ming Leung's comment A Different way to Apply SBE to Structured Methods and with reference to some of the ideas on his blog post here: The New V-Model I decided to write a longer response and blog.

I prefer to use the term “Measurement” and have applied it to Business goal, Business process, User story and program levels"

The given/when/then construct is appropriate where examples of a process or feature is required to illustrate and challenge the understanding of the feature or process in context. We recommend its use in requirements validation of identified features before implementation, in addition to the testing of software as per Behaviour-Driven Development. The approach can be used to validate business processes to some extent, in that pre-conditions, steps and post-conditions provide an example and therefore a test case of a stage in a business process.

Higher level business goals are defined and measured in different ways, however.

In the early 2000s I coined the term 'Project Intelligence' to label the information that testers and a full test process gathered, analysed and disseminated. You can download the 2004 paper that sets out these ideas here: Managing Projects With Intelligence. The notion of testing is expanded to include not only software testing (at all levels) but also the measurement (and that is the correct term in this context) of achievement in general. That is, I suggest that the disciplines of testing can be applied to the highest level business goals, intermediate goals and achievements, and in fact, any output, outcome or business benefit. So this notion fits nicely with your proposal to use the term "measurement rather than testing".

"Different “test detailing” methods are applied at different level"

Given your description of detailing, I agree. Testers (if they are to measure at every technical and business level) need to learn how to measure in a plethora of dimensions. Business measures come in all shapes and sizes from financial metrics and ratios to all kinds of physical measures to measures of less tangible outcomes or achievements:

• Financial measures might appear simple, but the formulae, data derivation and calculations required might not be. A certain level of financial accounting nous will be required.
• Physical measures relate to 'almost everything else' that is measurable. This could be a headcount, products sold, manufactured and delivered in a period, time to manufacture, deliver or repair, commissioned floorspace, employees trained and certified, systems delivered, tested or implemented. There is no limit to what can be measured and the variations of goal and achievement targets can be set and measured are endless.
• What about intangibles? How are they measured? There's the theory that anything can be measured (Tom Gilb's law). I interpret this as, "if I choose to assign a measure to that 'intangible concept', then that's good enough for me". In so far as stakeholders can regard such targets as meaningful, then I guess these things can be measured. But this is, in my opinion, more an art than a science. There's a lot more subjectivity that is comfortable for some. But if you are the measurer, and not the stakeholder, then perhaps it's not your problem to worry about. Maybe you don't have the domain knowledge or experience to judge what a good measure is? Who knows?

Given this extended role of testing (or measurement), the people who test today have an opportunity to expand their remit to measuring at a much higher level, and their involvement and influence in projects could be much more significant. Approaches such as benefits realisation, results-based and performance management require good measurement to work, but their advocate rarely, if ever, explain how measurements are made. There is a presumption that numbers just 'fall from the air' perhaps. The challenge of measurement falls most severely on systems testers. I propose that the hard-won lessons of system testing can be applied at all levels in business programmes and maybe the systems testers should take this role on.

"I regard the exploration and verification capabilities desirable side effects but the ultimate goal is to agree with users the Measurements (Acceptance criteria) as Specifications."

I think by this you mean the internal testing of systems at a unit, integration or system level is of little interest to stakeholders but that the definition of criteria for acceptance and the acceptance (testing) process itself is of most importance. I think definitely that stakeholders usually are not interested in what goes on inside software development projects (and most software testing falls into this category), but they are interested in the process that triggers acceptance and payments to suppliers in particular. But there is a large variation in views of what acceptance criteria are and whether they can ever be objective. I think the traditional 'software testing' view that a feature can be deemed accepted if a set of tests derived from its specification can be staged and run successfully is inadequate and not meaningful.

Most stakeholders find it hard to be specific enough for us (testers) to stage tests. I don't think we have made anywhere near enough progress towards defining acceptance criteria that truly cover all aspects of a system to allow us to test them 'objectively'. Such criteria must cover the functionality, of course, but there are so many intangible aspects to the comfort, fit, confidence, sense of certainty that a 'good' system engenders that our attempts to define and measure usability, performance, security in technical terms miss the mark. I know we can define acceptance criteria in these technical, measurable, objective ways, but they aren't satisfactory even through they are “rational” and “objective”.

But you know what? I don't think it matters.

Many studies have shown that business people, especially at the highest level, make their biggest decisions based on experience, intuition and gut feel. Yes, they ask for the data, analyses and fancy charts. Sometimes they understand the data. But when it comes to making the call, as often as not, the guy at the top will take the opinions of his peers and gauge the consensus in his community to make that final decision. Our pained deliberations over the rationality or irrationality of our measurements don't really figure at all in their estimation.

What we really need is to be regarded as a peer and have our say in that consensus.

Tags: #projectintelligence #BSM #resultsbasedmanagement #benefitsrealization

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account