Paul Gerrard

My experiences, opinions in the Test Engineering business. I am republishing/rewriting old blogs from time to time.

First published 01/05/2007

At last week's Test Management Forum, Susan Windsor introduced a lively session on estimation – from the top down. All good stuff. But during the discussion, I was reminded of a funny story (well I thought it was funny at the time).

Maybe twenty years ago (my memory isn’t as good as it used to be), I was working at a telecoms company as a development team leader. Around 7pm one evening, I was sat opposite my old friend Hugh. The office was quiet, we were the only people still there. He was tidying up some documentation, I was trying to get some stubborn bug fixed (I’m guessing here). Anyway. Along came the IT director. He was going home and he paused at our desks to say hello, how’s it going etc.

Hugh gave him a brief review of progress and said in closing, “we go live a week on Friday – two weeks early”. Our IT director was pleased but then highly perplexed. His response was, “this project is seriously ahead of schedule”. Off he went scratching his head. As the lift doors closed, Hugh and I burst out laughing. This situation had never arisen before. What a problem to dump on him! How would he deal with this challenge? What could he possibly tell the business? It could be the end of his career! Delivering early? Unheard of!

It’s a true story, honestly. But what it also reminded me of was that if estimation is an approximate process, our errors in estimation in the long run (over or under estimation) expressed as a percentage under or over, should balance statistically around a mean value of zero, and that mean would represent the average actual time or cost it took for our projects to deliver.

Statistically, if we are dealing with a project that is delayed (or advanced!) by unpredictable, unplanned events, we should be overestimating as much as we under estimate, shouldn’t we? But clearly this isn’t the case. Overestimation, and delivering early is a situation so rare, it’s almost unheard of. Why is this? Here's a stab at a few reasons why we consistently 'underestimate'.

First, (and possibly foremost) is we don't underestiate at all. Our estimates are reasonably accurate, but consistently we get squeezed to fit with pre-defined timescales or budgets. We ask for six people for eight weeks, but we get four people for four weeks. How does this happen? If we've been honest in our estimates, surely we should negotiate a scope reduction if our bid for resources or time is rejected? Whether we descope a selection of tests or not, when the time comes to deliver, our testing is unfinished. Of course, go live is a bumpy period – production is where the remaining bugs are encountered and fixed in a desperate phase of recovery. To achieve a reasonable level of stability takes as long as we predicted. We just delivered too early.

Secondly, we are forced to estimate optimistically. Breakthroughs, which are few and far between are assumed to be certainties. Of course, the last project, which was so troublesome, was an anomaly and it will always be better next time. Of course, this is nonsense. One definition of madness is to expect a different outcome from the same situation and inputs.

Thirdly, our estimates are irrelevant. Unless the project can deliver in some mysterious predetermined time and cost contraints, it won't happen at all. Where the vested interests of individuals dominate, it could conceivably be better for a supplier to overcommit, and live with a loss-making, troublesome post-go live situation. In the same vein, the customer may actually decide to proceed with a no-hoper project because certain individuals' reputation, credibility and perhaps jobs depend on the go live dates. Remarkable as it may seem, individuals within customer and supplier companies may actually collude to stage a doomed project that doesn't benefit the customer and loses the supplier money. Just call me cynical.

Assuming project teams aren't actually incompetent, it's reasonable to assume that project execution is never 'wrong' – execution just takes as long as it takes. There are only errors in estimation. Unfortunately, estimators are suppressed, overruled, pressured into aligning their activities with imposed budgets and timescales, and they appear to have been wrong.

Tags: #estimation

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 03/05/2006

I coach rowing, so I'll use this as an analogy. Consider the crew of rowers in a racing eight. The coach’s intention is to get all eight athletes rowing in harmony, with the same movement with balance, poise and control. In theory, if everyone does the same thing, the boat will move smoothly, and everyone can apply the power of their legs, trunk and arms to moving the boat as quickly as possible (and win races). Of course, one could just show the crew a video of some Olympic champions and say, 'do what they do', 'exactly', 'now'. But how dumb is that? Each person is an individual, having different physical shape and size, physiology, ambition, personality, attitudes and skill levels. Each athlete has to be coached individually to bring them up to the 'gold standard'. But it's harder than that, too. It's not as if each athlete responds to the same coaching messages. The coach has to find the right message to get the right response from each individual. For example, to get rowers to protect their lower backs, they must 'sit up' in the boat. Some rowers respond to 'sit up' others to 'keep your head high', 'be arrogant' and so on. That's just the way it is with people.

In the same way, when we want people to adopt a new way of working – a new 'process', we have to recognise that to get the required level of process adherence and consistency, (i.e. changed behaviours) every individual faces a different set of challenges. For each individual, it's a personal challenge. To get each individual to overcome their innate resistance to change, improve their skill levels, adjust their attitudes, and overall, change their behaviour, we have to recognise that each individual needs individual coaching, encouragement and support.

Typical 'process' improvement attempts start with refined processes, some training, a bit of practice, a pilot, then a roll-out. But where is the personal support in all this? To ask a group of individuals to adopt a new process (any process) by showing them the process and saying 'do it', is like asking a village footbal team to 'play like Brazil'.

Tags: #Rowing #TesterDevelopment #TestProcessImprovement

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 06/11/2009

Tools for Testing Web Based Systems

Selecting and Implementing a CAST Tool

Tools Selection and Implementation (PDF) What can test execution tools do for you? The main stages of tool selection and implementation, and a warning: Success may not mean what you think!

Registered users can downaload the paper from the link below. If you aren't registered, you can register here.

Tags: #testautomation #cast

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 29/01/2010

Last week I presented a talk called “Advancing Testing Using Axioms” at the First IIR Testing Forum in Helsinki, Finland.

Test Axioms have been formulated as a context-neutral set of rules for testing systems. Because they represent the critical thinking processes required to test any system, there are clear opportunities to advance the practice of testing using them.

The talk introduces “The First Equation of Testing” and discusses opportunities to use the Axioms to support test strategy development, test assessment and improvement and suggests that a tester skills framework could be an interesting by-product of the Axioms. Finally, “The Quantum Theory of Testing” is introduced.

Go to the web page for this talk.

Tags: #testaxioms #futures #advancingtesting

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 25/02/2008

There's been a lively discussion on axioms of testing and the subject of schools came up in that conversation. I'm not a member of any particular school and if people like to be part of one – good for them. I think discussion of schools is a distraction and doesn't help the axioms debate at all. I do suggest that axioms are context- and school- independent – so with respect to schools of testing, I had better explain my position here.

My good friend Neil Thompson has eloquently illustrated an obvious problem of being a member of a school. The dialogue reminded me of a Monty Python sketch – I can't think which one but “Dialectical Materialsm” came into it somewhere. I think it was an argument between Marx, Engels and other philosopers.

Anyway, Neil's hilarious example is very well pitched. I'd like to set out my position with respect to schools in this post.

I'm a consultant. I don't do targeted marketing so I don't choose my clients, my clients choose me. In the last 18 months or so, I've had widely varying engagements including: a Test Assurance Manager role on a $200m SAP project with an oil company, consulted for a £1bn+ Government Infrastructure/safety-related project, a medium-sized financial services company whose business is supported by systems developed and supported by a consortium of small companies and a software house providing custom software solutions to banks.

Each organisation and project represents a different challenge for testing. There is a huge spread of organisational cultures, busines and technical environments and of course, scale. My projects are a tiny sample of the huge variation in contexts for which test approaches must be designed, but it's easy to say:

  • It is quite obvious that no single approach or pre-meditated combination of test methods, tools, processes, heuristics etc. can support all contexts.
  • Since all software projects are unique, the contexts are unique so off-the-shelf approaches must be customised or designed from scratch.
There are some fairly well-defined approaches to testing that have been promoted and used over the years, and one can identify some stereotypes in the way that Bret Pettichord has in his useful talk 'Schools of Software Testing'. It seems to me that Bret's characterisation of schools must be at the same time, a characterisation of approaches. The ethos of a school defines the approach they promote – and vice-versa. It's not obvious whether schools predate their approaches or the approaches predate the school. I'm not sure it matters – but it varies.

But I think the difference in approaches reflect primarily a difference in emphasis. The Agile Manifesto articulated the values and preferences of that community very clearly but there are a range of agile approaches and all have merit.

Which leaves us exacly where?

The so-called schools of testing appear to me to limit its members' thinking. For the members of a school, the ethos of the schools represents a set of preferences for the types of projects and contexts that they would choose to work in. Being a member of a school also says something about its members when they market their services or are invited to join a project: “I prefer (or possibly demand) to work in this way and am qualified to do so (by some means)”.

In this respect, for individuals or organisations who align themselves with schools, the school ethos also represents a brand.

I am not a partner with any one test tool vendor because I value my independence. I do not limit myself to working only one way because my client projects don't come neatly packaged as one type or another. I have never known a project be adequately supported by one uncustomised, pre-packaged approach. Some people need to belong to a school or to be categorised or branded. I don't.

So, I'm not interested in testing schools, but I fully respect the wishes of people who want to be part of one.



Tags: #ALF

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 06/11/2009

This paper, written by Paul Gerrard was presented to the EuroSTAR Conference in London, November 1995 and won the prize for 'Best Presentation'.

Client/Server (C/S) technology is being taken up at an incredible rate. Almost every development organisation has incorporated C/S as part of their IT strategy. It appears that C/S will be the dominant architecture taking IT into the next millennium. Although C/S technology is gaining acceptance rapidly and development organisations get better at building such systems, performance issues remain as an outstanding risk when a system meets its functional requirements.

This paper sets out the reasons why system performance is a risk to the success of C/S projects. A process has been outlined which the authors have used to plan, prepare and execute automated performance tests. The principles involved in organising a performance test have been set out and an overview of the tools and techniques that can be used for testing two and three-tier C/S systems presented.

In planning, preparing and executing performance tests, there are several aspects of the task which can cause difficulties. The problems that are encountered most often relate to the stability of the software and the test environment. Unfortunately, testers are often required to work with a shared environment with software that is imperfect or unfinished. These issues are discussed and some practical guidelines are proposed.

Download the paper from here.



Tags: #performancetesting #client/server

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 13/07/2011

It seems to me that, to date, perceptions of exploration in communities that don't practice it have always been that it is appropriate only for document- and planning-free contexts. It's not been helped by the emphasis that is placed on these contexts by the folk who practice and advocate exploration. Needless to say, the certification schemes have made the same assumption and promote the same misconception.

But I'm sure that things will change soon. Agile is mainstream and a generation of developers, product owners and testers who might have known no other approach are influencing at a more senior level. Story-based approaches to define requirements or to test existing requirements 'by example' and drive acceptance (as well as being a source of tests for developers) are gaining followers steadily. The whole notion of story telling/writing and exampling is to ask and answer 'what if' questions of requirements, of systems, of thinking.

There's always been an appetite for less test documentation but rarely the trust – at least in testers (and managers) brought up in process and standards-based environments. I think the structured story formats that are gaining popularity in Agile environments are attracting stakeholders, users, business analysts, developers – and testers. Stories are not new – users normally tell stories to communicate their initial views on requirements to business analysts. Business analysts have always known the value of stories in eliciting, confirming and challenging the thinking around requirements.

The 'just-enough' formality of business stories provides an ideal communication medium between users/business analysts and testers. Business analysts and users understand stories in business language. The structure of scenarios (given/when/then etc.) maps directly to the (pre-conditions/steps/post-conditions) view of a formal test case. But this format also provides a concise way of capturing exploratory tests.

The conciseness and universality of stories might provide the 'just enough' documentation that allows folk who need documentation to explore with confidence.

I'll introduce some ideas for exploratory test capture in my next post.

Tags: #exploratorytesting #businessstories #BusinessStoryManager

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 28/05/2007

As a matter of record, I wanted to post a note on my involvement with the testing certification scheme best known in the UK (and many other countries) as the ISEB Testing Certificate Scheme. I want to post some other messages commenting on the ISEB, ISTQB and perhaps other schemes too, so a bit of background might be useful.

In 1997, a small group of people in the UK started to discuss the possibility of establishing a testing certification scheme. At that time, Dorothy Graham and I were probably the most prominent. There was some interest in the US too, I recall, and I briefly set up a page on the Evolutif website promoting the idea of a joint European/US scheme, and asking for expressions of interest in starting a group to formulate a structure, a syllabus, an examination and so on. Not very much came of that, but Dot and I in particular, drafted an outline syllabus which was just a list of topics, about a page long.

The Europe/US collaboration didn't seem to be going anywhere so we decided to start it in the UK only to begin with. At the same time, we had been talking to people at ISEB who seemed interested in administering the certification scheme itself. At that time ISEB was a certifying organisation having charitable status, independent of the British Computer Society (BCS). That year, ISEB decided to merge into the BCS. ISEB still had it's own identity and brand, but was a subsidiary of BCS from then on.

ISEB, having experience of running several schemes for several years (whereas we had no experience at all) suggested we form a certification 'board' with a chair, terms of reference and constitution. The first meeting of the new board took place on 14th January 1998. I became the first Chair of the board. I still have the Terms of Reference for the board, dated 17 May 1998. Here are the objectives of the scheme and the board extracted from that document:

Objectives of the Qualification • To gain recognition for testing as an essential and professional software engineering specialisation by industry. • Through the BCS Professional Development Scheme and the Industry Structure Model, provide a standard framework for the development of testers' careers. • To enable professionally qualified testers to be recognised by employers, customers and peers, and raise the profile of testers. • To promote consistent and good testing practice within all software engineering disciplines. • To identify testing topics that are relevant and of value to industry • To enable software suppliers to hire certified testers and thereby gain commercial advantage over their competitors by advertising their tester recruitment policy. • To provide an opportunity for testers or those with an interest in testing to acquire an industry recognised qualification in the subject.

Objectives of the Certification Board The Certification Board aims to deliver a syllabus and administrative infrastructure for a qualification in software testing which is useful and commercially viable. • To be useful it must be sufficiently relevant, practical, thorough and quality-oriented so it will be recognised by IT employers (whether in-house developers or commercial software suppliers) to differentiate amongst prospective and current staff; it will then be viewed as an essential qualification to attain by those staff. • To be commercially viable it must be brought to the attention of all of its potential customers and must seem to them to represent good value for money at a price that meets ISEB's financial objectives.

The Syllabus evolved and was agreed by the summer. The first course and examination took place on 20-22 October 1998, and the successful candidates were formally awarded their certificates at the December 1998 SIGIST meeting in London. In the same month, I resigned as Chair but remained on the board. I subsequently submitted my own training materials for accreditation.

Since the scheme started, over 36,000 Foundation examinations have been taken with a pass rate of about 90%. Since 2002 more than 2,500 Practitioner exams have been taken, with a relatively modest pass rate of approximately 60%.

The International Software Testing Qualificaton Board (ISTQB) was established in 2002. This group aims to establish a truly international scheme and now has regional boards in 33 countries. ISEB have used the ISTQB Foundation syllabus since 2004, but continue to use their own Practitioner syllabus. ISTQB are developing a new Practitioner level syllabus to be launched soon, but ISEB have already publicised their intention to launch their own Practitioner syllabus too. It's not clear yet what the current ISEB accredited training providers will do with TWO schemes. It isn't obvious what the market will think of two schemes either.

Interesting times lie ahead.

Tags: #istqb #iseb #TesterCertification

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 05/11/2010

I attended the Unicom “Next Generation Testing Conference” this week and there was lots of good discussion on the difference between Agile and Waterfall.

Afterwards (isn’t it always afterwards!) I thought of something I wished I’d raised at the time, so I thought I’d write a blog by way of follow up. Now I don’t do blogs, although I’ve been meaning to for ages. So this is a new, and hopefully pleasant, experience for me.

I’ve always been concerned about how to contract with a supplier in an agile way. Supplier management is a specific area of interest for me. I’ve listened to many presentations and case studies on this, but frankly haven’t been convinced yet. However, I’ve had one of those light bulb moments. We spend so much time talking about how to improve the delivery and predictability of systems and yet our industry has a bunch of suppliers whose business depends upon us not getting requirements right!

This isn’t their fault though as the purchasing process most companies go through encourages and rewards this behaviour. In a competitive bidding process where price is sensitive, all bidders are seeking to give a keen price for their interpretation of the requirements, knowing that requirements are typically inconsistent and complete. This allows them to bid low, maybe at cost or even lower, and then make their profit on the re-work. I’d always thought re-work was around 30% but Tom Gilb gave figures that show it can be much higher than that, so that’s a good profit incentive isn’t it.

So, here we all are, seeking to reduce spend putting pressure on daily rates, etc. We’re looking at the wrong dimension here! There is a much bigger prize!

Can we reduce the cost of purchasing, reduce the cost of re-work and meet our goals of predictable system delivery. Now, (and some of you will be thinking finally!) I’m convinced agile can help achieve this, but is any customer out there enlightened enough to realise this? Or is it more important to them to maintaining the status quo and avoid change?

Tags: #agile #contracts #suppliers #test

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 05/10/2010

The first four articles in this series have set out the main approaches to combating regression in changing software systems. From a business and technical viewpoint, we have considered both pre-change regression prevention (impact analysis) and post-change regression detection (regression testing). In this final article of the series, we’ll consider three emerging approaches that promise to reduce the regression threat and present some considerations of an effective anti-regression strategy with a recap of the main messages of the article series.

Three Approaches: Test, Behaviour and Acceptance Test-Driven Development

There is an increasing amount of discussion on development approaches based on the test-driven model. Ten years or so ago, before lightweight (later named Agile) approaches became widely publicized, test-driven development (TDD) was rare. Some TTD happened, but mostly in high integrity environments where component development and testing was driven by the need to meet formal functional and structural test coverage targets.

Over the course of the last ten years however, the notion of developers creating automated tests typically based on stories and discussions with on-site customers is becoming more common. The leaders in the Agile community are tending to preach behaviour- (BDD) and even acceptance test-driven development (ATDD) to improve and make accessible the test assets in Agile projects. They are also an attempt to move the Agile emphasis from coding to delivery of stakeholder value.

The advocates of these approaches (see for example testdriven.net, gojko.net, behaviour-driven.org, ATDD in Practice) would say that the approaches are different and of course, in some respects they are. But from the point of view of our discussion of anti-regression approaches, the relevance is this:

  1. Regression testing performed by developers is probably the most efficient way to demonstrate functional equivalence of software (given the limited scope of unit testing).
  2. The test-driven paradigm ensures that regression test assets are acquired and maintained in synchrony with the code – so are accurate and constantly reusable.
  3. The existence of a set of trusted regression tests means that the programmer is protected (to some degree) from introducing regressions when they change code (to enhance, fix bugs in or refactor code).
  4. Programmers, once they commit to the test-first approach tend to find their design and coding activities more predictable and less stressful.
These approaches obviously increase the effort at the front-end and many programmers are not adopting (and may never adopt) them. However, the trend toward test-first does seem to be gaining momentum.

A natural extension of test-first in Agile and potentially more structured environments is the notion of live specifications. In this approach, the automated tests become the independent and executable definition of the behavior of the system. The tests define the behavior of a system by example, and can be considered to be executable specifications (of a sort). Of course, examples alone cannot define the behavior of systems completely and some level of logical specification will always be required. However, the live-specification approach holds great promise, particularly as way of reducing regressions.

The ideal seems to be that where a change is required by users, the live specification is changed, new tests added and existing tests changed or retired as required. The software changes are made in parallel. The new and changed tests are run to demonstrate the changes work as required, and the existing (unchanged) tests are, by definition, the regression test pack. The format, content and structure of such live-specifications are evolving and a small number of organisations claim some successes. It will be interesting to see examples of the approach in action.

Unified Requirements and Systems Testing

The test-first approaches discussed above are gaining popularity in Agile environments. But what can be done in structured, waterfall, larger developments?

Some years ago (in my first Eurostar paper in 1993), I proposed a ‘Unified Approach to System Functional Testing’. In that paper, I suggested that a tabular notation for capturing examples or test cases could be used to create crude prototypes, review check lists and structured walkthroughs of requirements. These ‘behaviours’ as I called them could be used to test requirements documents, but also reused as the basis of both system and acceptance testing later on. Other interests took priority and I didn’t take this proposal much further until recently.

Several developments in the industry make me believe that a practical implementation of this unified approach is now possible and attractive to practitioners. See for example the model-based papers here: www.geocities.com/modelbasedtesting/online_papers.htm or the tool described here: teststories.info. To date, these approaches have focused on high formality and embedded/industrial applications.

Our approach involves the following activities:

  1. Requirements are tabulated to allow cross-referencing.
  2. Requirements are analysed and stories, comprising feature descriptions and a covering set of scenarios and examples (acceptance criteria) are created
  3. The scenarios are mapped to paths though the business process and a data dictionary; paper and automated prototypes can be generated from the scenarios
  4. Using scenario walkthroughs, the requirements are evaluated, omissions and ambiguities identified and fixed.
  5. The process paths, scenarios and examples may be incorporated into software development contracts, if required.
  6. The process paths, scenarios and examples are re-used as the basis of the acceptance test which is conducted in the familiar way.
Essentially, the requirements are ‘exampled’, with features identified and a set of acceptance criteria defined for each – in a structured language. It is the structure of the scenarios that allows tabular definitions of tests for use in manual procedures as well as skeletal automated tests to be generated automatically. There are several benefits deriving from this approach, but the two that concern us here are:
  • The definition of tests and the ability to generate automated scripts occurs before code is written which means that the test-first approach is viable for all projects, not just Agile.
  • The database of requirements, processes, process paths, features, examples and data dictionary are cross-referenced. The database can be used to support more detailed business-oriented impact analysis.
The first benefit has been discussed in the previous section. The second has great potential.

The business knowledge captured in the process will allow some very interesting what-if questions to be asked and answered. If a business process is to change, the system features, requirements, scenarios and tests affected can be traced. If a system feature is to be changed, the scenarios, tests, requirement and business process affected can be traced. This knowledge should provide at least at a high level, a better understanding of the impact of change. Further, it promotes the notion of live specifications and Trusted Requirements.

There is a real possibility that the (typically) huge investment in requirements capture will not be wasted and the requirements may be accurately maintained in parallel with a covering set of scenarios. Further, the business knowledge captured in the requirements and the database can be retained for the lifetime of the system in question.

Improving Software Analysis Tools

The key barrier to performing better technical impact analyses is the lack (and expense) of appropriate tools to provide a range of source code analyses. Tools that provide visualisations of the architecture, relationships between components and hierarchical views of these relationships are emerging. Some obvious challenges make life somewhat difficult though:
  1. Tools are usually language dependent so mixed environments are troublesome.
  2. The source code for third-party components used in your system may not be available.
  3. Visualisation software is available, but for real-size systems, the graphical models can become huge and unworkable.
These tools are obviously focused at architects, designers and developers and are naturally technical in nature.

An example of how tools are evolving in this area is Structure101g by (headwaysoftware.com). This tool can perform detailed structural analyses of several languages (“java, C/C++ and anything”) but can, in principle provide visualisations, navigation and query facilities for any structural model. For example, with the necessary plugins, the tool can provide insights into XML/XSLT libraries and web site maps at varying levels of abstraction.

As tools like this become better established and more affordable, they will surely become ‘must-haves’ for architects and developers in large systems environments.

Anti-Regression Strategy – Making it Happen

We’ll close this article series with some guidelines summarised from this and previous articles. Numerals in brackets refer to the article number.
  1. Regressions in working software affect business users, technical support, testers, developers, software and project management and stakeholders. It is everyone’s problem (I, V).
  2. A comprehensive anti-regression strategy would include both regression prevention and detection techniques from a technical and business viewpoint. (I, II).
  3. Impact analysis can be performed from both a business and technical viewpoints. (all)
  4. Technical impact analysis really needs tool support; consider open source, proprietary (or consider building your own to meet your specific objectives).
  5. Regression testing may be your main defence against regression, but should never be the only one; impact analysis prevents regression and informs good regression testing. (I, II, IV).
  6. Regression testing can typically be performed at the component, system or business level. These test levels have different objectives, owners and may be automated to different degrees (III).
  7. Regression tests may be created in a test-driven regime, or as part of requirements or design based approaches. Reuse of tests saves time, but check that these tests actually meet your anti-regression objectives (III).
  8. Regression tests become less effective over time; review your test pack regularly, especially when you are about to add to it. (This could be daily in an Agile environment!) (III)
  9. Analyses of production data will tell you the format, volumes and patterns of data that is most common – use it as a source of test data and a model for coverage; but don’t forget to include the negative tests too though! (III)
  10. If you need to be selective in the tests you retain and execute then you’ll need an agreed process, forum, decision-maker or makers or criteria for selection (agreed with all stakeholders in 1 above) (III).
  11. Most regression testing can and should be automated. Understand your context (objectives, test levels, risk areas, developer/tester motivations and capabilities etc.) before defining your automation strategy (III, IV).
  12. Consider what test levels, system stimulation and outcome detection methods, ownership, capabilities and tool usability are required before defining an automation regime (IV).
  13. Creating an automation regime retrospectively is difficult and expensive; test-first approaches build regression testing into the DNA of project teams (V).
  14. There is a lot of thinking, activity and new approaches/tools being developed to support requirements testing, exampling, live-specs and test automation; take a look (V).
I wish you the best of luck in your anti-regression initiatives.

I’d like to express sincere thanks to the Eurostar Team for asking me to write this article series and attendees at the Test Management Summit for inspiring it.

Paul Gerrard 23 August 2010.

Tags: #regressiontesting #anti-regression

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account