Paul Gerrard

First published 06/11/2009

This talk was presented at SQSTEST in 2006. It sets out an alternative way of thinking about 'process improvement'. My argument is that we should focus on results, then define the changes we need to make. It draws on Results-Chain theory and the change management approach of John Kotter.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #softwaresuccessimprovement

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 16/12/2012

A couple of weeks ago I gave a talk that included a couple of slides that focused on the idea of Specification by Example and how it cannot be relied upon to fully define the functionality of a software solution. I thought I'd summarise it here while the thought was fresh in my mind and also because Martin Fowler recently re-posted a blog originally published some time ago.

Martin provides a broader perspective and significantly, he says 'Specification By Example only works in the context of a working relationship where both sides are collaborating and not fighting'. Quite. He quotes a Cedric Beust post that critiques TDD (and Agile projects in general) that promote the use of tests as specifications.

Clearly, SBE can work nicely in an Agile environment where the scenarios are there to capture some key examples of the feature in use. The more general business rules to be implemented in code are (presumably) discussed and captured elsewhere – specifically in the code and exampled in tests. The examples and automated tests based on the conversations are retained to provide evidence that the software 'works' and stays working after changes elsewhere. One obvious, valuable outcome of SBE, Behaviour-Driven or Test-Driven approaches is a set of automated tests that are a quite effective anti-regression measure for use in projects that practice a continuous delivery approach. But what about non-Agile? Can SBE work in all contexts?

The questions is, “can examples alone be trusted to fully describe some system behaviour?” The answer is occasionally yes, but usually – no. Here's an example of why not.

The table below shows some scenarios associated with a feature. Call it SBE, BDD or just a shorthand for some TDD tests. Whatever.

given a, b, c are real numbers
when a=<a>
  and b=<b>
  and c=<c>
then r1=<r1> and r2=<r2>


| a |  b | c | r1 | r2    |
| 1 | -2 | 1 |  1 |  1    |
| 1 |  3 | 2 |  1 |  2    |
| 1 |  3 | 2 |  1 |  2    |
|12 |-28 |15 | 1.5|  0.833|

It doesn't give much away does it? “Do you know what it is yet?” (as Rolf Harris might ask).

Now, I could keep giving you new examples that are correct from the point of view of the requirement (that I'm not yet sharing with you). Maybe you'd spot the pattern of the inputs and outputs and guess that a b c are the coefficients of a quadratic and r1 r2 are the roots. Aha. The programmer could easily implement the code as follows:

r1 = (-b + sqrt(bb – 4ac))/(2a)
r2 = (-b – sqrt(bb – 4ac))/(2a)

Sorted. But is it...?

Suppose I then gave you an example that could NOT be processed by the quadratic formulae? The example below would probably cause an exception in the code:

| a |  b | c |  r1 | r2  |
| 4 |  3 | 2 | ... | ... |

You can't take square roots of negative numbers. So you could argue that there's a validation rule (not yet specified) that rejects inputs that cause this exception and change the code accordingly. But in fact, one CAN derive the square root of negative numbers. They are just called 'complex numbers' that's all. (Mathematicians can be a bit slippery). Have we got it right yet? We'd have to look at the expected outcomes in the examples provided, and generate them in code and hope for the best. Whatever. That's enough maths for one day.

The principle must be that examples on their own do not provide enough information to formulate a general solution. It is always possible to code a solution that will satisfy the examples provided. But that is not a solution – it is mimicry. A coded table of pre-defined answers can mimic the general solution. But the very next example, when used to test our solution will fail if the example is not in our coded table. Our model of the solution is incomplete – it's wrong. In fact, to be certain we have the perfect solution we would effectively need an infinite number of examples that when tested generate the required outcomes. Specification by Example ALONE cannot provide a complete specification.

Where does this leave us? Specifications are models of systems. All models are wrong (or at least incomplete) but some are useful. But having a specification is a necessary (but probably not sufficient) condition for building a solution.

Perhaps Specification by Example is mis-named. It should called be Specification AND Example.

The question remains, “How much software is out there that just mimics a solution?”

Tags: #specificationbyexample #SBE

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/05/2011

We can help you meet the challenges in the selection and management of your current and prospective external/internal suppliers and partners. We can help your supplier management by:

• Evaluating supplier strengths and weaknesses
• Identifying any major risks associated with your supplier services
• Developing contract schedules including specific acceptance criteria
• Refining and improving the commercial relationship with your suppliers
• Developing plans and techniques for the performance tracking and management of your suppliers

If you’d like to know more, please contact us directly.

Tags: #SupplierSelection #SupplierSelectionManagement

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 27/05/2011

Susan Windsor's Test Assurance introduction, presented at the Unicom Next Generation conference on 19 May. To date, Test Assurance has been a rather specialised discipline that we've been involved in for the past 10-12 years or so. But it now seems like it's becoming a hot topic. This is a broad overview of what it's all about. Download the presentation

Tags: #testassurance

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/11/2009

Is it possible to define a set of axioms that provide a framework for software testing that all the variations of test approach currently being advocated align with or obey? In this respect, an axiom would be an uncontested principle; something self-evidently and so obviously true and not requiring proof. What would such test axioms look like?

This paper summarises some preliminary work on defining a set of Test Axioms. Some applications of the axioms that would appear useful are suggested for future development. It is also suggested the work of practitioners and researchers is on very shaky ground unless we refine and agree these Axioms. This is a work in progress.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #testaxioms #thinkingtools

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/11/2009

Some foundation/visionary work to define ERP specific test methods and tools has already been performed by the author. However, the approach needs more research, rigor and proving in a commercial environment. Academic and commercial partners are sought to refine, develop and prove these methods and tools. An overview of the value of reporting test progress with reference to risk.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #risk-basedtesting #sap #erp

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 06/11/2009

As I am a member of the ISEB Testing Certificate board I should immediately say that these comments, observations etc. should not be considered the views of that august body.

The ISEB Testing Certificate Scheme, at the Foundation level, has been in operation for a couple of years now. Over two thousand testers, developers, managers and a significant number of users (often roped into testing) have taken and passed the Foundation exam and presumably display their certificate on their office wall. Like many others, I don’t have a wall I can call my own, so it takes pride of place on the wall of my study at home. As 2000 draws to a close, I’m looking back at my personal experience as an exam candidate who was thorough with the clep study guide, an accredited trainer and as one of those responsible for the syllabus (but not in that order!).

When was the last time you sat a formal exam? When I took the Foundation paper, I had not sat an exam for 19 years. Although I was confident of passing, like everyone else, I had pre-exam nerves. I guess these are not to be feared if they are under control, but it was clear from the comments of the other victims, that most folk enjoyed their hour of pressure…once it was over. After the exam, there’s a frustrating time to wait for the result and when it comes, there’s a nervous moment as you open the envelope. If you’ve passed, then the strain of a three-day intensive course and exam seem worthwhile after all.

I’ve presented our own course many times now, in public and on client sites. There are now few surprises in the questions, complaints and occasional praise I receive. What are other people’s reactions to the scheme? The most common comment relates to the sheer quantity of material to be assimilated in (usually) less than three days. I know what they mean. During the course, I speak around 20,000 words per day. Trust me – we taped and later transcribed a course I gave last year to build into our TBT version. The range of topics is extremely broad. To cover all of the foundation topics thoroughly would probably take three or four weeks worth of courses. For candidates having little experience, haven’t read a book on testing, or who left college many years ago, it’s a real challenge to take it all in.

Is it just too much? In my opinion, the quantity of material to be covered isn’t necessarily a problem. But the large range of topics covered means that inevitably, some are less useful than others. Few ISEB candidates are developers, so the white box test sections often cause grief to non-technical students (and the trainer). However, there is a strong argument for including such topics. If we teach what developers should do, testers can argue the case for better upstream testing practices when they return to work.

Is the training useful? With few exceptions, people who take the course come away with many, many ideas they can use on their return to the real world. I have received emails months after a course telling me of lessons being applied. Typical success stories are “…my boss finally understands what I’m trying to do as a tester”, “I’m now confident that our testing is thorough”, “…the developers show us more respect and we get on better”. For testers with all ranges of experience, it seems to be useful.

Is the qualification worthwhile? I think so. As an employer of testers, I’m confident that if a job applicant has the qualification they will be able to at least understand what I’m talking about. They should know how to be productive in any testing project and be able to argue a good case for doing testing to their peers. An independent consultant friend of mine told me that having the qualification actually got her an assignment recently. Someone willing to use his or her own money to fund their training and qualification is worth taking seriously, don’t you think? Yes, it’s worthwhile.

Is the scheme successful? Well, it will be when it’s finished. We all hope the Practitioner scheme gets going soon. In the meantime, there are strong signs that the scheme is penetrating other countries as diverse as Ireland, Holland, Scandinavia, Australia with others on the way. It’s going the right way.

Paul Gerrard, January 2001

Tags: #iseb #certification

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 25/11/2009

Ref Michael B's testing v checking interesting but...

comparison with scripted – only talks about execution

thought process leading to scripts – similar toexploration and more productive?

scripted: super-clow motion

load our expextaction bullet – kerching

before we pull the trigger – are explorers at a disadvantage?

benefits of planned scripted

is MB comparing expert explorer with dumb test runer – i.e. a tool?

could we get a valid comparison of scripted v exploratory?

Tags: #ALF

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/05/2011

It’s our experience that a test approach should to be tailored to meet the specific needs of an organisation and/or a project.

There is no 'one-size fits all'.

Testing needs to take into account organisational culture, people, suppliers and method of working which together combine to make a unique situation. No two organisations are exactly the same. For a project, testing has to account for stakeholder goals, business objectives, technical environment and development approach being deployed. No two projects, even for the same organisation, are the same. We apply our extensive knowledge and experience along with industry good practice to define:

• An organisation wide testing framework to be used as a model for your individual projects, taking account of both waterfall and agile methods
• A project specific testing approach that meets your quality goals and takes account of your budget and timescale demands

We will deliver a testing approach that is ‘tailored fit’ to meet your needs. If you’d like to know more, please contact us directly or buy the Testers Pocketbook.

*** If you are interested in an on-site workshop, see our 'Test Strategy in a Day' workshop ***

Tags: #TestApproach #TestStrategy

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 06/11/2009

This paper, written by Paul Gerrard was presented to the EuroSTAR Conference in Edinburgh, November 1997. This is the most popular and downloaded paper on this site, which tells you that perhaps the art of GUI testing hasn't moved on that much in the last ten years?

Download Testing GUI Applications.

Tags: #gui #guiapplications

Paul Gerrard My linkedin profile is here My Mastodon Account