Paul Gerrard

First published 07/12/2011

Its been interesting to me to watch over the last 10 or maybe 15 years the debate over whether exploratory or scripted testing is more effective. There's no doubt that one can explore more of a product in the time it takes for someone to follow a script. But then again – how much time exploratory testers lose spent bumbling around lost, aimlessly going over the same ground many times, hitting dead ends (because they have little or no domain or product knowledge to start with). Compare that with a tester who has lived with the product requirements as they have evolved over time. They may or may not be blinkered, but they are better informed – sort of.

I'm not going to decry the value of exploration or planned tests – both have great value. But I reckon people who think exploration is better than scripted under all circumstances have lost sight of a thing or two. And that phrase 'lost sight of a thing or two' is significant.

I'm reading Joseph T. Hallinan's book, “Why We Make Mistakes”. Very early on, in the first chapter no less Hallinan suggests, “we're built to quit”. It makes sense. So we are.

When humans are looking for something, smuggled explosives, tumors in x-rays, bugs in software – humans are adept at spotting what they look for – if, and it's a big if, these things are common – in which case they are pretty effective – spotting what they look for most of the time.

But, what if what they seek is relatively rare? Humans are predisposed to just give up the search prematurely. It's evolution stupid! Looking for, and not finding, food in one place just isn't sensible after a while. you need to move on.

Hallinan quotes (among others) the cases of people who look for PA-10 rifles in luggage in airports and tumours in xrays. In these cases, people look for things that rarely exist. In the case of radiologists, mammograms reveal tumours only 0.3 percent of the time. 99.7 percent of the time the searcher will not find what they look for.

In the case of guns or explosives in luggage the occurrence is rarer. In 2004, according to the thegunsource.com, 650 million passengers travvelled in the US by air. But only 598 firearms were found – about one in a million occurences.

Occupations that seek to find things that are rare have considerable error rates. The miss rate for radiogists looking for cancers is around 30%. In one study at the world famous Mayo clinic, 90% of the tumours missed by radiologists were visible in previous x-rays.

In 2008, from the UK I travelled to the US, to Holland and Ireland. On my third trip, returning from Ireland with the same rucksdack on my back at the security check at Dublin airport (i.e. my sixth flight), when going through security, I was called to one side by a security officer. A lock-knife with a 4.5 inch blade was found in my rucksack. Horrified, when presented with the article I asked could it please be disposed of! It was mine, but in the bag by mistake – and had been there for six months unnoticed, by me and by five airport security scans. This was the sixth flight with the offending article in the bag. Five previous scans at airports terminals had failed to detect a quite heavy metal object – pointed and a potentially dangerous weapon. How could that happen? Go figure.

Back to software. Anyone can find bugs in crappy software. Its like walking in bare feet in a room full of loaded mouse traps. But if you are testing software of high quality, it's harder to find bugs. It may be you give up before you have given yourself time to find the really (or not so) subtle ones.

Would a script help? I don't know. It might help because in principle, you have to follow it. But it might make you even more bored. All testers get bored/hungry/lazy/tired and are more or less incompetent or uninformed – you might give up before you've given yourself time to find anything significant. Our methods, such as they are, don't help much with this problem. Exploratory testing can be just as draining/boring as scripted.

I want people to test well. It seems to me that the need to test well increases with the criticality and quality of software, and motivation to test aligns pretty closely. Is exploration or scripted testing of very high quality software more effective? I'm not sure we'll ever know until someone does a proper experiment (and I don't mean testing a 2000 line of code toy program in a website or nuclear missile).

I do know that if you are testing high quality code and just before release it usually is of high quality, then you have to have your eyes open and your brain switched on. Both of em.

Tags: #exploratorytesting #error #scriptedtesting

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 06/11/2009

This talk will be presented at EuroSTAR 2007 in Stockholm in December 2007. I presented the talk at the EuroSTAR-Testnet mini-event at Nieuwegein, Holland on the same night as Liverpool played AC Milan in the Champions League Cup Final hence the picture on slide 2. (It's a shame they lost 2-1). The focus of the talk is that using lessons learned can help you formulate a better test strategy, or as I am calling them nowadays, 'Project Intelligence Strategies'.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #sap #erp #lessonslearned

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 06/11/2009

As I am a member of the ISEB Testing Certificate board I should immediately say that these comments, observations etc. should not be considered the views of that august body.

The ISEB Testing Certificate Scheme, at the Foundation level, has been in operation for a couple of years now. Over two thousand testers, developers, managers and a significant number of users (often roped into testing) have taken and passed the Foundation exam and presumably display their certificate on their office wall. Like many others, I don’t have a wall I can call my own, so it takes pride of place on the wall of my study at home. As 2000 draws to a close, I’m looking back at my personal experience as an exam candidate who was thorough with the clep study guide, an accredited trainer and as one of those responsible for the syllabus (but not in that order!).

When was the last time you sat a formal exam? When I took the Foundation paper, I had not sat an exam for 19 years. Although I was confident of passing, like everyone else, I had pre-exam nerves. I guess these are not to be feared if they are under control, but it was clear from the comments of the other victims, that most folk enjoyed their hour of pressure…once it was over. After the exam, there’s a frustrating time to wait for the result and when it comes, there’s a nervous moment as you open the envelope. If you’ve passed, then the strain of a three-day intensive course and exam seem worthwhile after all.

I’ve presented our own course many times now, in public and on client sites. There are now few surprises in the questions, complaints and occasional praise I receive. What are other people’s reactions to the scheme? The most common comment relates to the sheer quantity of material to be assimilated in (usually) less than three days. I know what they mean. During the course, I speak around 20,000 words per day. Trust me – we taped and later transcribed a course I gave last year to build into our TBT version. The range of topics is extremely broad. To cover all of the foundation topics thoroughly would probably take three or four weeks worth of courses. For candidates having little experience, haven’t read a book on testing, or who left college many years ago, it’s a real challenge to take it all in.

Is it just too much? In my opinion, the quantity of material to be covered isn’t necessarily a problem. But the large range of topics covered means that inevitably, some are less useful than others. Few ISEB candidates are developers, so the white box test sections often cause grief to non-technical students (and the trainer). However, there is a strong argument for including such topics. If we teach what developers should do, testers can argue the case for better upstream testing practices when they return to work.

Is the training useful? With few exceptions, people who take the course come away with many, many ideas they can use on their return to the real world. I have received emails months after a course telling me of lessons being applied. Typical success stories are “…my boss finally understands what I’m trying to do as a tester”, “I’m now confident that our testing is thorough”, “…the developers show us more respect and we get on better”. For testers with all ranges of experience, it seems to be useful.

Is the qualification worthwhile? I think so. As an employer of testers, I’m confident that if a job applicant has the qualification they will be able to at least understand what I’m talking about. They should know how to be productive in any testing project and be able to argue a good case for doing testing to their peers. An independent consultant friend of mine told me that having the qualification actually got her an assignment recently. Someone willing to use his or her own money to fund their training and qualification is worth taking seriously, don’t you think? Yes, it’s worthwhile.

Is the scheme successful? Well, it will be when it’s finished. We all hope the Practitioner scheme gets going soon. In the meantime, there are strong signs that the scheme is penetrating other countries as diverse as Ireland, Holland, Scandinavia, Australia with others on the way. It’s going the right way.

Paul Gerrard, January 2001

Tags: #iseb #certification

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 05/11/2009

This talk setting out some thoughts on what's happening in the testing marketplace. Covers Benefits-Based Testing, Testing Frameworks, Software Success Improvement, Tester Skills and provides some recommendations for building your career.

Registered users can download the paper from the link below. If you aren't registered, you can register here.

Tags: #testingtrends

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 01/12/2009

The extraordinary growth in the Internet is sweeping through industry. Small companies can now compete for attention in the global shopping mall – the World Wide Web. Large corporations see ‘The Web’ not only as an inexpensive way to make company information on products and services available to anyone with a PC and browser, but increasingly as a means of doing on-line business with world wide markets. Companies are using the new paradigm in four ways:

• Web sites - to publicise services, products, culture and achievements.
• Internet products - on-line services and information to a global market on the Web.
• Intranet products - on-line services and information for internal employees.
• Extranet products - on-line services and information enabling geographically distributed organisations to collaborate.

Web-based systems can be considered to be a particular type of client/server architecture. However, the way that these systems are assembled and used means some specialist tools are required and since such tools are becoming available we will give them some consideration here.

The risks that are particular to Web based applications are especially severe where the system may be accessed by thousands or tens of thousands of customers. The very high visibility of some systems being built mean that failure in such systems could be catastrophic. Web pages usually comprise text base files in Hypertext Markup Language (HTML) and now contain executable content so that the traditional separation of ‘code and data’ is no longer possible or appropriate. Browsers, plug-ins, active objects and Java are also new concepts which are still immature.

There are four main categories of tools which support the testing of Web applications:

Application test running

Test running tools that can capture tests of user interactions with Web applications and replay them now exist. These tools are either enhancements to existing GUI test running tools or are new tools created specifically to drive applications accessed through Web browsers. The requirements for these tools are very similar to normal GUI test running tools, but there are some important considerations.

Firstly, the Web testing tool will be designed to integrate with specific browsers. Ask your vendor whether the tool supports the browsers your Web application is designed for, but also check whether old versions of these browsers are also supported. To test simple text-oriented HTML pages, the Web testing tool must be able to execute the normal web page navigation commands, recognise HTML objects such as tables, forms, frames, links to other pages and content consisting of text, images, video clips etc. HTML text pages are often supplemented by server-side programs typically in the form of Common Gateway Interface (CGI) Scripts which perform more substantial processing. These should be transparent to the test tool.

Increasingly, web applications will consist of simple text-based web pages, as before, but these will be complemented with ‘active content’ components. These components are likely to be Java applets, Netscape ‘plug-ins’, ActiveX controls. Tools are only just emerging which are capable of dealing with these components. Given the portable nature of the Java development language, tools written in Java may actually be completely cable of dealing with any legitimate Java object in your Web application, so may be an obvious choice. However, if other non-Java components are present in your application, a ‘pure-Java’ tool may prove inadequate. Another consideration is how tools cope with dynamically generated HTML pages – some tools cannot.

HTML source, link, file and image checkers

Tools have existed for some time which perform ‘static tests’ of Web pages and content. These tools open a Web page (a site Home page, for example) to verify the syntax of the HTML source and check that all the content, such as images, sounds and video clips can be accessed and played/displayed. Links to other pages on the site can be traversed, one by one. For each linked page, the content is verified until the tool runs out of unvisited links. These tools are usually configured to stop the search once they encounter a link to an off-server page or another site, but they can effectively verify every page and that every home-site based link and content is present. Most of these tools provide graphical reports on the structure of Web sites which highlight the individual pages, internal and external links, missing links and other missing content.

Component test-drivers

Advanced Web applications are likely to utilise active components which are not directly accessible, using a browser-oriented Web testing tool. Currently, developers have to write customised component drivers, for example using the main{} method in Java classes to exercise the methods in a the class without having to use other methods in other classes.

As web applications become more sophisticated, the demand for specialised component drivers to test low level functionality and integration of components will increase. Such tools may be delivered as part of a development toolkit, but unfortunately, development tool vendors are more often interested in providing the ‘coolest’ development, rather than testing tools.

Internet performance testing

Web applications are most easily viewed as a particular implementation of client/server. The performance testing tools for the web that are available are all enhanced versions of established client/server based tools. We can consider the requirements for load generation and client application response time measurement separately.

Load generation tools rely on a master or control program running on a server which drive physical workstations using a test running tool to drive the application, or test drivers which submit Web traffic across the network to the Web servers. In the first case, all that is new is that it is the Web-oriented test running tool which drives the client application through a browser. For larger tests, test drivers capable of generating Web traffic across the network are used. Here, the test scripts dispatch remote procedure calls to the web servers. Rather than a SQL-oriented database protocol such as ODBC, the test script will contain HTTP calls which use the HTTP protocol instead. All that has changed is the test driver programs.

Client application response time measurement is done using the Web test running tool. This may be a standalone tool running on a client workstation or the client application driver, controlled by the load generation tool.

Tags: #tools #automation #web #internet

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 22/06/2011

Many thanks to Helmut Pichler and Manfred Baumgartner of Anecon who invited me to speak at their joint seminar with Microsoft at the Microsoft office in Vienna, Austria in May. Thanks also to Andreas Pollak of Microsoft who organised the event and who acted as my able assistant when my remote control did not work.

The event agenda and sessions are described here. Andreas assembled the Powerpoint and a voice recording into a movie which is reproduced below. Apologies for the first minute of the talk, as my remote control gadget didn't work. Andreas kindly offered assistance :O)

The talk, ah yes, the talk. Essentially, it's an attempt to discuss the meaning of quality and how testers use test models. Abstract below.

I hope I don't upset too many British Royal Watchers, Apple Product devotees or McDonalds lovers with this talk. I'm not one of you, I'm afraid.

Abstract: Rain is great for farmers and their crops, but terrible for tourists. Wind is essential for sailors and windmills but bad for the rest of us. Quality, like weather, is good or bad and that depends on who you are. Just like beauty, comfort, facility, flavour, intuitiveness, excitement and risk, quality is a concept that most people understand, but few can explain. It’s worse. Quality is an all-encompassing, collective term for these and many other difficult concepts.

Quality is not an attribute of a system – it is a relationship between systems and stakeholders who take different views and the model of Quality that prevails has more to do with stakeholders than the system itself. Measurable quality attributes make techies feel good, but they don’t help stakeholders if they can’t be related to experience. If statistics don’t inform the stakeholders’ vision or model of quality, we think we do a good job. They think we waste their time and money. Whether documented or not, testers need and use models to identify what is important and what to test. A control flow graph has meaning (and value) to a programmer but not to a user. An equivalence partition has meaning to users but not the CEO. Control flow, equivalence partitions are models with value in some, but never all, contexts. If we want to help stakeholders to make better-informed decisions then we need test models that do more than identify tests. We need models that take account of the stakeholders’ perspective and have meaning in the context of their decision-making. If we measure quality using technical models (quality attributes, test techniques) we delude both our stakeholders and ourselves into thinking we are in control of Quality.

We’re not.

In this talk, Paul uses famous, funny and tragic examples of system failures to illustrate ways in which test models and (therefore testing) failed. He argues strongly that the pursuit of Quality requires that testers need better test models and how to create them, fast.

Tags: #quality #tornadoes #testmodels

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 21/10/2009

I'm relieved, excited and delighted to tell you that The Tester's Pocketbook has been published and is available. (It is a pocketbook, with 104 pages and c. 19k words).

The book summarises the thinking on Test Axioms and the axiom definitions are hosted (and will be maintained in future) on the Test Axioms website.

Thanks to all my reviewers and people who supported me.

Tags: #paulgerrard #testaxioms #testerspocketbook

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 12/03/2010

On Thursday's SIGIST meeting, it was great to have such a positive reaction to my workshop and closing talk.

The Fallibility axiom (p41) tells us our sources of knowledge are undependable. The tester is a human being and prone to error. The system is being tested because we are uncertain of its behaviour or reliability. As a consequence, the plan for any test worth running cannot be relied upon to be accurate before we follow it. Predictions of test status (e.g. coverage achieved or test pass- rate) at any future date or time are notional. The planning quandary is conveniently expressed in the testing uncertainty principle:

• One can predict test status, but not when it will be achieved;
• One can predict when a test will end, but not its status.

Consequently, if a plan defines completion of testing using test exit criteria to be met at a specified date (expressed in terms of tests run and the status of those tests) it is wise to regard them as planning assumptions, rather than hard targets.

• If exit criteria are met on time or earlier, our planning assumptions are sound: We are where we want to be.
• If exit criteria are not met or not met on time, our plan was optimistic: Our plan needs adjustment, or we must relax the criteria.

Whichever outcome arises, we still need to think very carefully about what they actually mean in our project.

Tags: #testaxioms #uncertainty

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 01/12/2009

The Knowledge Base has moved HERE!

This is a new website currently hosting a directory of tools in the DevOps, SDET and Testing support domains and it also provides a searchable index of tools. There are over 2208 tools registered although 693 are actually programming languages.

The site also monitors the blog pages of 277 bloggers. These again are indexed and searchable.

Numbers correct as of 25/8/2015.

Tags: #tkb #ToolsKnowledgeBase

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account

First published 07/04/2016

At Eurostar 2010 in Copenhagen, the organisers asked me to do a brief video blog, and I was pleased to oblige. I had presented a track talk on test axioms in the morning and I had mentioned a couple of ideas in the talk. these were the “quantum theory of testing” and “testing relativity”. The video goes into a little more detail.

The slides I presented are included in the slideshare set below. The fonts don't seem to have uploaded, I'm afraid:

Tags: #testingrelativity #quantumtesting

Paul Gerrard Please connect and contact me using my linkedin profile. My Mastodon Account