Paul Gerrard

First published 21/10/2009

I'm relieved, excited and delighted to tell you that The Tester's Pocketbook has been published and is available. (It is a pocketbook, with 104 pages and c. 19k words).

The book summarises the thinking on Test Axioms and the axiom definitions are hosted (and will be maintained in future) on the Test Axioms website.

Thanks to all my reviewers and people who supported me.

Tags: #paulgerrard #testaxioms #testerspocketbook

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 12/03/2010

On Thursday's SIGIST meeting, it was great to have such a positive reaction to my workshop and closing talk.

The Fallibility axiom (p41) tells us our sources of knowledge are undependable. The tester is a human being and prone to error. The system is being tested because we are uncertain of its behaviour or reliability. As a consequence, the plan for any test worth running cannot be relied upon to be accurate before we follow it. Predictions of test status (e.g. coverage achieved or test pass- rate) at any future date or time are notional. The planning quandary is conveniently expressed in the testing uncertainty principle:

• One can predict test status, but not when it will be achieved;
• One can predict when a test will end, but not its status.

Consequently, if a plan defines completion of testing using test exit criteria to be met at a specified date (expressed in terms of tests run and the status of those tests) it is wise to regard them as planning assumptions, rather than hard targets.

• If exit criteria are met on time or earlier, our planning assumptions are sound: We are where we want to be.
• If exit criteria are not met or not met on time, our plan was optimistic: Our plan needs adjustment, or we must relax the criteria.

Whichever outcome arises, we still need to think very carefully about what they actually mean in our project.

Tags: #testaxioms #uncertainty

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 01/12/2009

The Knowledge Base has moved HERE!

This is a new website currently hosting a directory of tools in the DevOps, SDET and Testing support domains and it also provides a searchable index of tools. There are over 2208 tools registered although 693 are actually programming languages.

The site also monitors the blog pages of 277 bloggers. These again are indexed and searchable.

Numbers correct as of 25/8/2015.

Tags: #tkb #ToolsKnowledgeBase

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/11/2009

This talk setting out some thoughts on what's happening in the testing marketplace. Covers Benefits-Based Testing, Testing Frameworks, Software Success Improvement, Tester Skills and provides some recommendations for building your career.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #testingtrends

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 03/12/2009

You may or may not find this response useful. :–)

“It depends”.

The “it depends” response is an old joke. I think I was advised by David Gelperin in the early 90s that if someone says “it depends” your response should be “ahh, you must be a consultant!”

But it does depend. It always has and will do. The context-driven guys provide a little more information – “it depends on context”. But this doesn't answer the question of course – we still get asked by people who really do need an answer – i.e. project managers who need to plan and to resource teams.

As an aside, there’s an interesting discussion of “stupid questions” here. This question isn't stupid, but the blog post is interesting.

In what follows – let me assume you’ve been asked the question by a project manager.

The 'best' dev/tester ratio is possibly the most context-specific question in testing. What are the influences on the answer?

• What is the capability/competence of the developers and testers respectively and absolutely?
• What do dev and test WANT to do versus what you (as a manager) want them to do?
• To what degree are the testers involved in early testing (they just system test? Or are involved from concept thru to acceptance etc.)
• What is the risk-profile of the project?
• Do stakeholders care if the system works or not?
• What is the scale of the development?
• What is the ratio of new/custom code versus reused (and trusted) code/infrastructure?
• How trustworthy is the to-be-reused code anyway?
• How testable will the delivered system be?
• Do resources come in integer whole numbers or fractions?
• And so on, and so on…

Even if you had the answers to these questions to six significant digits – you still aren’t much wiser because some other pieces of information are missing. These are possibly known to the project manager who is asking the question:

• How much budget is available? (knowing this – he has an answer already)
• Does the project manager trust your estimates and recommendations or does he want references to industry ‘standards’? i.e. he wants a crutch, not an answer.
• Is the project manager competent and honest?

So we’re left with this awkward situation. Are you being asked the question to make the project manager feel better; to give him reassurance he has the right answer already? Does he know his budget is low and needs to articulate a case for justifying more? Does he think the budget is too high and wants a case for spending less?

Does he regard you as competent and trust what you say anyway? This final point could depend on his competence as much as yours! References to ‘higher authorities’ satisfy some people (if all they want is back-covering), but other folk want personal, direct, relevant experience and data.

I think a bit of von Neumann game theory may be required to analyse the situation!

Here’s a suggestion. Suppose the PM says he has 4 developers and needs to know how many testers are required. I’d suggest he has a choice:

• 4 dev – 1 tester: onus is on the devs to do good testing, the tester will advise, cherry pick areas to test and focus on high impact problems. PM needs to micro manage the devs, and the tester is a free-agent.
• 4 dev – 2 testers: testers partner with dev to ‘keep them honest’. Testers pair up to help with dev testing (whether TDD or not). Testers keep track of the coverage and focus on covering gaps and doing system-level testing. PM manages dev based on tester output.
• 4 dev – 3 testers: testers accountable for testing. Testers shadow developers in all dev test activities. System testing is thorough. Testers set targets for achievement and provide evidence of it to PM. PM manages on the basis of test reports.
• 4 dev – 4 testers: testers take ownership of all testing. But is this still Agile??? ;–)

Perhaps it’s worth asking the PM for dev and tester job specs and working out what proportion of their activities are actually dev and test? Don’t hire testers at all – just hire good developers (i.e. those who can test). If he has poor developers (who can’t/won’t test) then the ratio of testers goes up because someone has to do their job for them.

Tags: #estimation #testerdeveloperratio

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 03/12/2009

The V-model promotes the idea that the dynamic test stages (on the right hand side of the model) use the documentation identified on the left hand side as baselines for testing. The V-Model further promotes the notion of early test preparation.

Figure 4.1 The V-Model of testing.

Early test preparation finds faults in baselines and is an effective way of detecting faults early. This approach is fine in principle and the early test preparation approach is always effective. However, there are two problems with the V-Model as normally presented.

The V-Model with early test preparation.

Firstly, in our experience, there is rarely a perfect, one-to-one relationship between the documents on the left hand side and the test activities on the right. For example, functional specifications don’t usually provide enough information for a system test. System tests must often take account of some aspects of the business requirements as well as physical design issues for example. System testing usually draws on several sources of requirements information to be thoroughly planned.

Secondly, and more important, the V-Model has little to say about static testing at all. The V-Model treats testing as a “back-door” activity on the right hand side of the model. There is no mention of the potentially greater value and effectiveness of static tests such as reviews, inspections, static code analysis and so on. This is a major omission and the V-Model does not support the broader view of testing as a constantly prominent activity throughout the development lifecycle.

The W-Model of testing.

Paul Herzlich introduced the W-Model approach in 1993. The W-Model  attempts to address shortcomings in the V-Model. Rather than focus on specific dynamic test stages, as the V-Model does, the W-Model focuses on the development products themselves. Essentially, every development activity that produces a work product is “shadowed” by a test activity. The purpose of the test activity specifically is to determine whether the objectives of a development activity have been met and the deliverable meets its requirements. In its most generic form, the W-Model presents a standard development lifecycle with every development stage mirrored by a test activity. On the left hand side, typically, the deliverables of a development activity (for example, write requirements) is accompanied by a test activity “test the requirements” and so on. If your organization has a different set of development stages, then the W-Model is easily adjusted to your situation. The important thing is this: the W-Model of testing focuses specifically on the product risks of concern at the point where testing can be most effective.

The W-Model and static test techniques.

If we focus on the static test techniques, you can see that there is a wide range of techniques available for evaluating the products of the left hand side. Inspections, reviews, walkthroughs, static analysis, requirements animation as well as early test case preparation can all be used.

The W-Model and dynamic test techniques.

If we consider the dynamic test techniques you can see that there is also a wide range of techniques available for evaluating executable software and systems. The traditional unit, integration, system and acceptance tests can make use of the functional test design and measurement techniques as well as the non-functional test techniques that are all available for use to address specific test objectives.

The W-Model removes the rather artificial constraint of having the same number of dynamic test stages as development stages. If there are five development stages concerned with the definition, design and construction of code in your project, it might be sensible to have only three stages of dynamic testing only. Component, system and acceptance testing might fit your normal way of working. The test objectives for the whole project would be distributed across three stages, not five. There may be practical reasons for doing this and the decision is based on an evaluation of product risks and how best to address them. The W-Model does not enforce a project “symmetry” that does not (or cannot) exist in reality. The W-model does not impose any rule that later dynamic tests must be based on documents created in specific stages (although earlier documentation products are nearly always used as baselines for dynamic testing). More recently, the Unified Modeling Language (UML) described in Booch, Rumbaugh and Jacobsen’s book [5] and the methodologies based on it, namely the Unified Software Process and the Rational Unified Process™ (described in [6-7]) have emerged in importance. In projects using these methods, requirements and designs might be documented in multiple models so system testing might be based on several of these models (spread over several documents).

We  use the W-Model in test strategy as follows. Having identified the specific risks of concern, we specify the products that need to be tested; we then select test techniques (static reviews or dynamic test stages) to be used on those products to address the risks; we then schedule test activities as close as practicable to the development activity that generated the products to be tested.
 

Tags: #w-model

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 14/12/2011

When the testing versus checking debate started with Michael’s blog here http://www.developsense.com/blog/2009/08/testing-vs-checking/ I read the posts and decided it wasn’t worth getting into. It seemed to be a debate amongst the followers of the blog and the school rather than a more widespread unsettling of the status quo.

I fully recognise the difference between testing and checking (as suggested in the blogs). Renaming of what most people call testing today to checking and redefining testing in the way Michael suggests upset some folk, cheered others. Most if not all developer testing and all testing through an API using tools becomes checking – by definition. I guess developers might sniff at that. Pretty much what exploratory testers do becomes the standard for what the new testing is. So they are happy. Most testers tend not to follow blogs so they are still blissfully unaware of the debate.

Brian Marick suggested the blogs were a ‘power play’ in a Tweet and pointed to an interesting online conversation here http://tech.groups.yahoo.com/group/agile-testing/message/18116. The suggested redefinitions appear to underplay checking and promote the virtue of testing. Michael clarified his position and said it wasn’t here http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/ and said:

“The distinction between testing and checking is a power play, but it’s not a power play between (say) testers and programmers. It’s a power play between the glorification of mechanizable assertions over human intelligence. It’s a power play between sapient and non-sapient actions.”

In the last year or so, I’ve had a few run-ins with people and presenters at conferences when I asked what they meant by checking when they used the word. They tended to forget the distinction and focus on the glorification bit. They told me testing was good (“that’s what I get paid for”) and checking was bad, useless or for drones. I’m not unduly worried by that – but it’s kind of irritating.

The problem I have is that if the idea (distinguishing test v check) is to gain traction and I believe it should, then changing the definition of testing is hardly going to help. It will confuse more than clarify. I hold that the scope of testing is much broader than testing software. In our business we test systems (a system could be a web page, it could be a hospital). The word and activity is in widespread use in almost every business, scientific and engineering discipline you can imagine. People may or may not be checking, but to ask them to change the name and description of what they do seems a bit ambitious. All the textbooks, papers and blogs written by people in our business will have to be reinterpreted and possibly changed. Oh, and how many dictionaries around the world need a correction? My guess is it won’t happen.

It’s much easier to say that a component of testing is checking. Know exactly what that is and you are a wiser tester. Sapient even.

The test v check debate is significant in the common exploratory contexts of individuals making decisions on what they do right now in an exploratory session perhaps. But it isn’t significant in the context of larger projects and teams. The sapience required in an exploratory session is concentrated in the moment to moment decision making of the tester. The sapience in other projects is found elsewhere.

In a large business project, say an SAP implementation, there might be ten to twenty legacy and SAP module system test teams plus multiple integration test teams as well as one or several customer test teams all working at a legacy system, SAP module or integrated system level. SAP projects vary from maybe fifty to several thousand man-years of effort of which a large percentage (tens of percent) is testing of one form or another. Although there will be some exploration in there – most of the test execution will be scripted and it’s definitely checking as we know it.

But, the checking activity probably counts for a tiny percentage of the overall effort and much of it is automated. The sapient effort goes into the logistics of managing quite large teams of people who must test in this context. Ten to twenty legacy systems must be significantly updated, system tested, then integrated with other legacy systems and kept in step with SAP modules that are being configured with perhaps ten thousand parameter changes. All this takes place in between ten and thirty test environments over the course of one to three years. And in all this time, business as usual changes on the legacy systems and the system to be migrated and/or retired must be accommodated.

As the business and projects learn what it is about, requirements evolve and all the usual instability disturbs things. But change is an inevitable consequence of learning and large projects need very careful change management to make sure the learning is communicated. It’s an exploratory process on a very large scale. Testing includes data migration, integration with customers, suppliers, banks, counterparties; it covers regulatory requirements, cutover and rollback plans, workarounds, support and maintenance processes as well as all the common non-functional areas.

Testing in these projects has some parallels with a military campaign. It’s all about logistics. Test checking activity compares with ‘pulling the trigger’.

Soldiering isn’t just about pulling triggers. In the same way, testing isn’t just about checking. Almost all the sapient activity goes into putting the testers into exactly the right place at the right time, fully equipped with meaningful and reliable environments, systems under test, integral data and clear instructions, with dedicated development, integration, technical, data, domain expert support teams. Checking may be manual or automated, but it’s a small part of the whole.

Exploration in environments like these can’t be done ‘interactively’. It really could take months and tens/hundreds of thousands of pounds/dollars/euros to construct the environment and data to run a speculative test. Remarkably, exploratory tests are part of all these projects. They just need to be wisely chosen and carefully prepared, just like other planned tests, because you have a limited time window and might not get a second chance. These systems are huge production lines for data so they need to be checked endlessly end to end. It’s a factory process so maybe testing is factory-like. It’s just a different context.

The machines on the line (the modules/screens) are extremely reliable commercial products. They do exactly what you have configured them to do with a teutonic reliability. The exploration is really one of requirements, configuration options and the known behaviour of modules used in a unique combination. Test execution is confirmation but it seems that it can be done no other way.

It rarely goes smoothly of course. That’s logistics for you. And testing doing what it always does.

Tags: #testingvchecking #checking #factory

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 01/12/2009

The extraordinary growth in the Internet is sweeping through industry. Small companies can now compete for attention in the global shopping mall – the World Wide Web. Large corporations see ‘The Web’ not only as an inexpensive way to make company information on products and services available to anyone with a PC and browser, but increasingly as a means of doing on-line business with world wide markets. Companies are using the new paradigm in four ways:

• Web sites - to publicise services, products, culture and achievements.
• Internet products - on-line services and information to a global market on the Web.
• Intranet products - on-line services and information for internal employees.
• Extranet products - on-line services and information enabling geographically distributed organisations to collaborate.

Web-based systems can be considered to be a particular type of client/server architecture. However, the way that these systems are assembled and used means some specialist tools are required and since such tools are becoming available we will give them some consideration here.

The risks that are particular to Web based applications are especially severe where the system may be accessed by thousands or tens of thousands of customers. The very high visibility of some systems being built mean that failure in such systems could be catastrophic. Web pages usually comprise text base files in Hypertext Markup Language (HTML) and now contain executable content so that the traditional separation of ‘code and data’ is no longer possible or appropriate. Browsers, plug-ins, active objects and Java are also new concepts which are still immature.

There are four main categories of tools which support the testing of Web applications:

Application test running

Test running tools that can capture tests of user interactions with Web applications and replay them now exist. These tools are either enhancements to existing GUI test running tools or are new tools created specifically to drive applications accessed through Web browsers. The requirements for these tools are very similar to normal GUI test running tools, but there are some important considerations.

Firstly, the Web testing tool will be designed to integrate with specific browsers. Ask your vendor whether the tool supports the browsers your Web application is designed for, but also check whether old versions of these browsers are also supported. To test simple text-oriented HTML pages, the Web testing tool must be able to execute the normal web page navigation commands, recognise HTML objects such as tables, forms, frames, links to other pages and content consisting of text, images, video clips etc. HTML text pages are often supplemented by server-side programs typically in the form of Common Gateway Interface (CGI) Scripts which perform more substantial processing. These should be transparent to the test tool.

Increasingly, web applications will consist of simple text-based web pages, as before, but these will be complemented with ‘active content’ components. These components are likely to be Java applets, Netscape ‘plug-ins’, ActiveX controls. Tools are only just emerging which are capable of dealing with these components. Given the portable nature of the Java development language, tools written in Java may actually be completely cable of dealing with any legitimate Java object in your Web application, so may be an obvious choice. However, if other non-Java components are present in your application, a ‘pure-Java’ tool may prove inadequate. Another consideration is how tools cope with dynamically generated HTML pages – some tools cannot.

HTML source, link, file and image checkers

Tools have existed for some time which perform ‘static tests’ of Web pages and content. These tools open a Web page (a site Home page, for example) to verify the syntax of the HTML source and check that all the content, such as images, sounds and video clips can be accessed and played/displayed. Links to other pages on the site can be traversed, one by one. For each linked page, the content is verified until the tool runs out of unvisited links. These tools are usually configured to stop the search once they encounter a link to an off-server page or another site, but they can effectively verify every page and that every home-site based link and content is present. Most of these tools provide graphical reports on the structure of Web sites which highlight the individual pages, internal and external links, missing links and other missing content.

Component test-drivers

Advanced Web applications are likely to utilise active components which are not directly accessible, using a browser-oriented Web testing tool. Currently, developers have to write customised component drivers, for example using the main{} method in Java classes to exercise the methods in a the class without having to use other methods in other classes.

As web applications become more sophisticated, the demand for specialised component drivers to test low level functionality and integration of components will increase. Such tools may be delivered as part of a development toolkit, but unfortunately, development tool vendors are more often interested in providing the ‘coolest’ development, rather than testing tools.

Internet performance testing

Web applications are most easily viewed as a particular implementation of client/server. The performance testing tools for the web that are available are all enhanced versions of established client/server based tools. We can consider the requirements for load generation and client application response time measurement separately.

Load generation tools rely on a master or control program running on a server which drive physical workstations using a test running tool to drive the application, or test drivers which submit Web traffic across the network to the Web servers. In the first case, all that is new is that it is the Web-oriented test running tool which drives the client application through a browser. For larger tests, test drivers capable of generating Web traffic across the network are used. Here, the test scripts dispatch remote procedure calls to the web servers. Rather than a SQL-oriented database protocol such as ODBC, the test script will contain HTTP calls which use the HTTP protocol instead. All that has changed is the test driver programs.

Client application response time measurement is done using the Web test running tool. This may be a standalone tool running on a client workstation or the client application driver, controlled by the load generation tool.

Tags: #tools #automation #web #internet

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 06/11/2009

This talk will be presented at EuroSTAR 2007 in Stockholm in December 2007. I presented the talk at the EuroSTAR-Testnet mini-event at Nieuwegein, Holland on the same night as Liverpool played AC Milan in the Champions League Cup Final hence the picture on slide 2. (It's a shame they lost 2-1). The focus of the talk is that using lessons learned can help you formulate a better test strategy, or as I am calling them nowadays, 'Project Intelligence Strategies'.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #sap #erp #lessonslearned

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 06/11/2009

A discussion of the main improvement types possible and how the TOM maturity model can be used to get a process improvement initiative started.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #softwaresuccessimprovement

Paul Gerrard My linkedin profile is here My Mastodon Account