Paul Gerrard

First published 28/04/2006

I've been asked to present the closing keynote at this year's Eurostar Conference in Manchester on December 7th. Here's the abstract: When it comes to improving the capabilities of our testers, if you believe the training providers brochures, you might think that a few days training in a classroom is enough to give a tester all the skills required to succeed. But it is obvious that to achieve mastery, it can take months or years to acquire the full range of technical and inter-personal skills required. Based on my experience as a rowing coach, this keynote describes how an athletic training programme is run and compares that with the way most testers are developed. An athlete will have a different training regime for the different periods of the year and coaching, mentoring, inspiring and testing are all key activities of the coach. Training consists of technical drills, strength, speed, endurance and team work. Of course a tester must spend most of their time actually doing their job, but there are many opportunities for evaluation and training to occur even in a busy schedule. Developing tester capability requires a methodical, humane approach with realistic goals, focused training, regular evaluation, feedback and coaching as well as on-the-job experience. You can see the presentation here: multi-page HTML file | Powerpoint Slide Show

 

I originally created this presentation for the BCS SIGIST meeting on the Ides of March.

Tags: #developingtesters #athletes #Rowing #TesterDevelopment

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 29/01/2010

Last week I presented a talk called “Advancing Testing Using Axioms” at the First IIR Testing Forum in Helsinki, Finland.

Test Axioms have been formulated as a context-neutral set of rules for testing systems. Because they represent the critical thinking processes required to test any system, there are clear opportunities to advance the practice of testing using them.

The talk introduces “The First Equation of Testing” and discusses opportunities to use the Axioms to support test strategy development, test assessment and improvement and suggests that a tester skills framework could be an interesting by-product of the Axioms. Finally, “The Quantum Theory of Testing” is introduced.

Go to the web page for this talk.

Tags: #testaxioms #futures #advancingtesting

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 06/11/2009

Tools for Testing Web Based Systems

Selecting and Implementing a CAST Tool

Tools Selection and Implementation (PDF) What can test execution tools do for you? The main stages of tool selection and implementation, and a warning: Success may not mean what you think!

Registered users can downaload the paper from the link below. If you aren't registered, you can register here.

Tags: #testautomation #cast

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/10/2010

The first four articles in this series have set out the main approaches to combating regression in changing software systems. From a business and technical viewpoint, we have considered both pre-change regression prevention (impact analysis) and post-change regression detection (regression testing). In this final article of the series, we’ll consider three emerging approaches that promise to reduce the regression threat and present some considerations of an effective anti-regression strategy with a recap of the main messages of the article series.

Three Approaches: Test, Behaviour and Acceptance Test-Driven Development

There is an increasing amount of discussion on development approaches based on the test-driven model. Ten years or so ago, before lightweight (later named Agile) approaches became widely publicized, test-driven development (TDD) was rare. Some TTD happened, but mostly in high integrity environments where component development and testing was driven by the need to meet formal functional and structural test coverage targets.

Over the course of the last ten years however, the notion of developers creating automated tests typically based on stories and discussions with on-site customers is becoming more common. The leaders in the Agile community are tending to preach behaviour- (BDD) and even acceptance test-driven development (ATDD) to improve and make accessible the test assets in Agile projects. They are also an attempt to move the Agile emphasis from coding to delivery of stakeholder value.

The advocates of these approaches (see for example testdriven.net, gojko.net, behaviour-driven.org, ATDD in Practice) would say that the approaches are different and of course, in some respects they are. But from the point of view of our discussion of anti-regression approaches, the relevance is this:

1. Regression testing performed by developers is probably the most efficient way to demonstrate functional equivalence of software (given the limited scope of unit testing).
2. The test-driven paradigm ensures that regression test assets are acquired and maintained in synchrony with the code – so are accurate and constantly reusable.
3. The existence of a set of trusted regression tests means that the programmer is protected (to some degree) from introducing regressions when they change code (to enhance, fix bugs in or refactor code).
4. Programmers, once they commit to the test-first approach tend to find their design and coding activities more predictable and less stressful.

These approaches obviously increase the effort at the front-end and many programmers are not adopting (and may never adopt) them. However, the trend toward test-first does seem to be gaining momentum.

A natural extension of test-first in Agile and potentially more structured environments is the notion of live specifications. In this approach, the automated tests become the independent and executable definition of the behavior of the system. The tests define the behavior of a system by example, and can be considered to be executable specifications (of a sort). Of course, examples alone cannot define the behavior of systems completely and some level of logical specification will always be required. However, the live-specification approach holds great promise, particularly as way of reducing regressions.

The ideal seems to be that where a change is required by users, the live specification is changed, new tests added and existing tests changed or retired as required. The software changes are made in parallel. The new and changed tests are run to demonstrate the changes work as required, and the existing (unchanged) tests are, by definition, the regression test pack. The format, content and structure of such live-specifications are evolving and a small number of organisations claim some successes. It will be interesting to see examples of the approach in action.

Unified Requirements and Systems Testing

The test-first approaches discussed above are gaining popularity in Agile environments. But what can be done in structured, waterfall, larger developments?

Some years ago (in my first Eurostar paper in 1993), I proposed a ‘Unified Approach to System Functional Testing’. In that paper, I suggested that a tabular notation for capturing examples or test cases could be used to create crude prototypes, review check lists and structured walkthroughs of requirements. These ‘behaviours’ as I called them could be used to test requirements documents, but also reused as the basis of both system and acceptance testing later on. Other interests took priority and I didn’t take this proposal much further until recently.

Several developments in the industry make me believe that a practical implementation of this unified approach is now possible and attractive to practitioners. See for example the model-based papers here: www.geocities.com/modelbasedtesting/online_papers.htm or the tool described here: teststories.info. To date, these approaches have focused on high formality and embedded/industrial applications.

Our approach involves the following activities:

1. Requirements are tabulated to allow cross-referencing.
2. Requirements are analysed and stories, comprising feature descriptions and a covering set of scenarios and examples (acceptance criteria) are created
3. The scenarios are mapped to paths though the business process and a data dictionary; paper and automated prototypes can be generated from the scenarios
4. Using scenario walkthroughs, the requirements are evaluated, omissions and ambiguities identified and fixed.
5. The process paths, scenarios and examples may be incorporated into software development contracts, if required.
6. The process paths, scenarios and examples are re-used as the basis of the acceptance test which is conducted in the familiar way.

Essentially, the requirements are ‘exampled’, with features identified and a set of acceptance criteria defined for each – in a structured language. It is the structure of the scenarios that allows tabular definitions of tests for use in manual procedures as well as skeletal automated tests to be generated automatically. There are several benefits deriving from this approach, but the two that concern us here are:

• The definition of tests and the ability to generate automated scripts occurs before code is written which means that the test-first approach is viable for all projects, not just Agile.
• The database of requirements, processes, process paths, features, examples and data dictionary are cross-referenced. The database can be used to support more detailed business-oriented impact analysis.

The first benefit has been discussed in the previous section. The second has great potential.

The business knowledge captured in the process will allow some very interesting what-if questions to be asked and answered. If a business process is to change, the system features, requirements, scenarios and tests affected can be traced. If a system feature is to be changed, the scenarios, tests, requirement and business process affected can be traced. This knowledge should provide at least at a high level, a better understanding of the impact of change. Further, it promotes the notion of live specifications and Trusted Requirements.

There is a real possibility that the (typically) huge investment in requirements capture will not be wasted and the requirements may be accurately maintained in parallel with a covering set of scenarios. Further, the business knowledge captured in the requirements and the database can be retained for the lifetime of the system in question.

Improving Software Analysis Tools

The key barrier to performing better technical impact analyses is the lack (and expense) of appropriate tools to provide a range of source code analyses. Tools that provide visualisations of the architecture, relationships between components and hierarchical views of these relationships are emerging. Some obvious challenges make life somewhat difficult though:

1. Tools are usually language dependent so mixed environments are troublesome.
2. The source code for third-party components used in your system may not be available.
3. Visualisation software is available, but for real-size systems, the graphical models can become huge and unworkable.

These tools are obviously focused at architects, designers and developers and are naturally technical in nature.

An example of how tools are evolving in this area is Structure101g by (headwaysoftware.com). This tool can perform detailed structural analyses of several languages (“java, C/C++ and anything”) but can, in principle provide visualisations, navigation and query facilities for any structural model. For example, with the necessary plugins, the tool can provide insights into XML/XSLT libraries and web site maps at varying levels of abstraction.

As tools like this become better established and more affordable, they will surely become ‘must-haves’ for architects and developers in large systems environments.

Anti-Regression Strategy – Making it Happen

We’ll close this article series with some guidelines summarised from this and previous articles. Numerals in brackets refer to the article number.

1. Regressions in working software affect business users, technical support, testers, developers, software and project management and stakeholders. It is everyone’s problem (I, V).
2. A comprehensive anti-regression strategy would include both regression prevention and detection techniques from a technical and business viewpoint. (I, II).
3. Impact analysis can be performed from both a business and technical viewpoints. (all)
4. Technical impact analysis really needs tool support; consider open source, proprietary (or consider building your own to meet your specific objectives).
5. Regression testing may be your main defence against regression, but should never be the only one; impact analysis prevents regression and informs good regression testing. (I, II, IV).
6. Regression testing can typically be performed at the component, system or business level. These test levels have different objectives, owners and may be automated to different degrees (III).
7. Regression tests may be created in a test-driven regime, or as part of requirements or design based approaches. Reuse of tests saves time, but check that these tests actually meet your anti-regression objectives (III).
8. Regression tests become less effective over time; review your test pack regularly, especially when you are about to add to it. (This could be daily in an Agile environment!) (III)
9. Analyses of production data will tell you the format, volumes and patterns of data that is most common – use it as a source of test data and a model for coverage; but don’t forget to include the negative tests too though! (III)
10. If you need to be selective in the tests you retain and execute then you’ll need an agreed process, forum, decision-maker or makers or criteria for selection (agreed with all stakeholders in 1 above) (III).
11. Most regression testing can and should be automated. Understand your context (objectives, test levels, risk areas, developer/tester motivations and capabilities etc.) before defining your automation strategy (III, IV).
12. Consider what test levels, system stimulation and outcome detection methods, ownership, capabilities and tool usability are required before defining an automation regime (IV).
13. Creating an automation regime retrospectively is difficult and expensive; test-first approaches build regression testing into the DNA of project teams (V).
14. There is a lot of thinking, activity and new approaches/tools being developed to support requirements testing, exampling, live-specs and test automation; take a look (V).

I wish you the best of luck in your anti-regression initiatives.

I’d like to express sincere thanks to the Eurostar Team for asking me to write this article series and attendees at the Test Management Summit for inspiring it.

Paul Gerrard 23 August 2010.

Tags: #regressiontesting #anti-regression

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 06/11/2009

This paper, written by Paul Gerrard was presented to the EuroSTAR Conference in London, November 1995 and won the prize for 'Best Presentation'.

Client/Server (C/S) technology is being taken up at an incredible rate. Almost every development organisation has incorporated C/S as part of their IT strategy. It appears that C/S will be the dominant architecture taking IT into the next millennium. Although C/S technology is gaining acceptance rapidly and development organisations get better at building such systems, performance issues remain as an outstanding risk when a system meets its functional requirements.

This paper sets out the reasons why system performance is a risk to the success of C/S projects. A process has been outlined which the authors have used to plan, prepare and execute automated performance tests. The principles involved in organising a performance test have been set out and an overview of the tools and techniques that can be used for testing two and three-tier C/S systems presented.

In planning, preparing and executing performance tests, there are several aspects of the task which can cause difficulties. The problems that are encountered most often relate to the stability of the software and the test environment. Unfortunately, testers are often required to work with a shared environment with software that is imperfect or unfinished. These issues are discussed and some practical guidelines are proposed.

Download the paper from here.

Tags: #performancetesting #client/server

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 13/07/2011

It seems to me that, to date, perceptions of exploration in communities that don't practice it have always been that it is appropriate only for document- and planning-free contexts. It's not been helped by the emphasis that is placed on these contexts by the folk who practice and advocate exploration. Needless to say, the certification schemes have made the same assumption and promote the same misconception.

But I'm sure that things will change soon. Agile is mainstream and a generation of developers, product owners and testers who might have known no other approach are influencing at a more senior level. Story-based approaches to define requirements or to test existing requirements 'by example' and drive acceptance (as well as being a source of tests for developers) are gaining followers steadily. The whole notion of story telling/writing and exampling is to ask and answer 'what if' questions of requirements, of systems, of thinking.

There's always been an appetite for less test documentation but rarely the trust – at least in testers (and managers) brought up in process and standards-based environments. I think the structured story formats that are gaining popularity in Agile environments are attracting stakeholders, users, business analysts, developers – and testers. Stories are not new – users normally tell stories to communicate their initial views on requirements to business analysts. Business analysts have always known the value of stories in eliciting, confirming and challenging the thinking around requirements.

The 'just-enough' formality of business stories provides an ideal communication medium between users/business analysts and testers. Business analysts and users understand stories in business language. The structure of scenarios (given/when/then etc.) maps directly to the (pre-conditions/steps/post-conditions) view of a formal test case. But this format also provides a concise way of capturing exploratory tests.

The conciseness and universality of stories might provide the 'just enough' documentation that allows folk who need documentation to explore with confidence.

I'll introduce some ideas for exploratory test capture in my next post.

Tags: #exploratorytesting #businessstories #BusinessStoryManager

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 28/05/2007

As a matter of record, I wanted to post a note on my involvement with the testing certification scheme best known in the UK (and many other countries) as the ISEB Testing Certificate Scheme. I want to post some other messages commenting on the ISEB, ISTQB and perhaps other schemes too, so a bit of background might be useful.

In 1997, a small group of people in the UK started to discuss the possibility of establishing a testing certification scheme. At that time, Dorothy Graham and I were probably the most prominent. There was some interest in the US too, I recall, and I briefly set up a page on the Evolutif website promoting the idea of a joint European/US scheme, and asking for expressions of interest in starting a group to formulate a structure, a syllabus, an examination and so on. Not very much came of that, but Dot and I in particular, drafted an outline syllabus which was just a list of topics, about a page long.

The Europe/US collaboration didn't seem to be going anywhere so we decided to start it in the UK only to begin with. At the same time, we had been talking to people at ISEB who seemed interested in administering the certification scheme itself. At that time ISEB was a certifying organisation having charitable status, independent of the British Computer Society (BCS). That year, ISEB decided to merge into the BCS. ISEB still had it's own identity and brand, but was a subsidiary of BCS from then on.

ISEB, having experience of running several schemes for several years (whereas we had no experience at all) suggested we form a certification 'board' with a chair, terms of reference and constitution. The first meeting of the new board took place on 14th January 1998. I became the first Chair of the board. I still have the Terms of Reference for the board, dated 17 May 1998. Here are the objectives of the scheme and the board extracted from that document:

Objectives of the Qualification • To gain recognition for testing as an essential and professional software engineering specialisation by industry. • Through the BCS Professional Development Scheme and the Industry Structure Model, provide a standard framework for the development of testers' careers. • To enable professionally qualified testers to be recognised by employers, customers and peers, and raise the profile of testers. • To promote consistent and good testing practice within all software engineering disciplines. • To identify testing topics that are relevant and of value to industry • To enable software suppliers to hire certified testers and thereby gain commercial advantage over their competitors by advertising their tester recruitment policy. • To provide an opportunity for testers or those with an interest in testing to acquire an industry recognised qualification in the subject.

Objectives of the Certification Board The Certification Board aims to deliver a syllabus and administrative infrastructure for a qualification in software testing which is useful and commercially viable. • To be useful it must be sufficiently relevant, practical, thorough and quality-oriented so it will be recognised by IT employers (whether in-house developers or commercial software suppliers) to differentiate amongst prospective and current staff; it will then be viewed as an essential qualification to attain by those staff. • To be commercially viable it must be brought to the attention of all of its potential customers and must seem to them to represent good value for money at a price that meets ISEB's financial objectives.

The Syllabus evolved and was agreed by the summer. The first course and examination took place on 20-22 October 1998, and the successful candidates were formally awarded their certificates at the December 1998 SIGIST meeting in London. In the same month, I resigned as Chair but remained on the board. I subsequently submitted my own training materials for accreditation.

Since the scheme started, over 36,000 Foundation examinations have been taken with a pass rate of about 90%. Since 2002 more than 2,500 Practitioner exams have been taken, with a relatively modest pass rate of approximately 60%.

The International Software Testing Qualificaton Board (ISTQB) was established in 2002. This group aims to establish a truly international scheme and now has regional boards in 33 countries. ISEB have used the ISTQB Foundation syllabus since 2004, but continue to use their own Practitioner syllabus. ISTQB are developing a new Practitioner level syllabus to be launched soon, but ISEB have already publicised their intention to launch their own Practitioner syllabus too. It's not clear yet what the current ISEB accredited training providers will do with TWO schemes. It isn't obvious what the market will think of two schemes either.

Interesting times lie ahead.

Tags: #istqb #iseb #TesterCertification

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/11/2010

I attended the Unicom “Next Generation Testing Conference” this week and there was lots of good discussion on the difference between Agile and Waterfall.

Afterwards (isn’t it always afterwards!) I thought of something I wished I’d raised at the time, so I thought I’d write a blog by way of follow up. Now I don’t do blogs, although I’ve been meaning to for ages. So this is a new, and hopefully pleasant, experience for me.

I’ve always been concerned about how to contract with a supplier in an agile way. Supplier management is a specific area of interest for me. I’ve listened to many presentations and case studies on this, but frankly haven’t been convinced yet. However, I’ve had one of those light bulb moments. We spend so much time talking about how to improve the delivery and predictability of systems and yet our industry has a bunch of suppliers whose business depends upon us not getting requirements right!

This isn’t their fault though as the purchasing process most companies go through encourages and rewards this behaviour. In a competitive bidding process where price is sensitive, all bidders are seeking to give a keen price for their interpretation of the requirements, knowing that requirements are typically inconsistent and complete. This allows them to bid low, maybe at cost or even lower, and then make their profit on the re-work. I’d always thought re-work was around 30% but Tom Gilb gave figures that show it can be much higher than that, so that’s a good profit incentive isn’t it.

So, here we all are, seeking to reduce spend putting pressure on daily rates, etc. We’re looking at the wrong dimension here! There is a much bigger prize!

Can we reduce the cost of purchasing, reduce the cost of re-work and meet our goals of predictable system delivery. Now, (and some of you will be thinking finally!) I’m convinced agile can help achieve this, but is any customer out there enlightened enough to realise this? Or is it more important to them to maintaining the status quo and avoid change?

Tags: #agile #contracts #suppliers #test

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 29/06/2011

Tags: #exploratorytesting

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 03/07/2006

A couple of weeks ago, after the BCS SIGiST meeting I was chatting to Martin Jamieson (of BT) about tools that test 'beneath the GUI'. A while later, he emailed a question...

At the recent SIGIST Johanna Rothman remarked that automation should be done below the level of the GUI. You then stood up and said you're working on a tool to do this. I was explaining to Duncan Brigginshaw (www.odin.co.uk) yesterday that things are much more likely to change at the UI level than at the API level. I gave him your example of programs deliberately changing the html in order to prevent hacking – is that right? However, Duncan tells me that he recommends automating at the UI. He says that the commercial tools have ways of capturing the inputs which shield the tester from future changes e.g. as objects. I think you've had a discussion with Duncan and am just wondering what your views are. Is it necessary to understand the presentation layers for example?

Ulitimately, all GUI interfaces need testing of course. The rendering and presentation of HTML, the execution of Javascript, ActiveX and Java objects obviously need a browser involved for a user to validate their behaviour. But Java/ActiveX can be tested through drivers written by programmers (and many are).

Typically, Javascript isn't directly accessible to GUI tools anyway (as it is typically used for field validation and manipulation, screen formatting and window management). One can write whole applications in JavaScript if you wish.

But note that I'm saying that a browser is essential for a user to validate layout and presentation. If you go down the route of using a tool to automate testing of the entire application from GUI through to server based code, you need quite sophisticated tools, with difficult to use scripting (programming) languages. And lo and behold, to make these tools more usable/accessible to non-programmers, you need tools like AXE to reduce (sometimes dramatically) the complexity of the scripting language required to drive automated tests.

Now, one of the huge benefits of these kinds of testing frameworks, coupled with 'traditional' GUI test tools is they allow less technical testers to create, manage and execute automated tests. But, if you were to buy a Mercury WinRunner or QTP license plus an AXE licence, you'd be paying 6k or 7k PER SEAT, before discounts. This is hugely expensive if you think about what most automated tools are actually used for – compared with a free tool that can execute tests of server-based code directly.

Most automated tools are used to automate regression tests. Full stop. I've hardly ever met a system tester who actually set out to find bugs with tools. (I know 'top US consultants' talk about such people, but they seem to exist as a small minority. What usually happens is that the tester needs to get a regression test together. Manual tests are run, when the software is stable and tests pass, they get handed over to the automation folk. I know, I know, that AXE and tools like it allow testers to create automated test runs. However, with buggy software, you never get past the first run of a new test. So much for running the other 99 using the tool – why bother.

Until you can run the other 99, you don't know whether they'll find bugs anyway. So folk resort to running them manually because you need a human being checking results and anomalies, not a dumb tool. The other angle is that most bugs aren't what you expect – by definition. e.g. checking a calculation result might be useful, but the tab order, screen validation, navigation, window management/consistency, usability and accessibility AREN'T in your preprared test plan anyway. So much for finding bugs proactively using automation. (Although bear in mind that there are free/cheap tools exist to validate HTML, accessibility, navigation and validation).

And after all this, be reminded, the calculated field is actually generated by the server based code. The expected result is a simple number, state variable or message. The position, font, font size and 57 other attributes of the field it appears in are completely irrelevant to the test case. The automated tool is, in effect, instructed by the framework tool to ignore these things, and focus on the tester's predicted result.

It's interesting (to me anyway) that the paper that is most downloaded from the Gerrard Consulting website is my paper on GUI testing. gerrardconsulting.com/GUI/TestGui.html It was written in 1997. It gets downloaded between 150 and 250 times a month approximately. Why is that for heavens sake – it's nine years old! The web isn't even mentioned in the paper! I can only think people are obsessed with the GUI and haven't got a good understanding of how you 'divide and conquer' the complex task of testing a GUI into simpler tasks. Some that can be automated beneath the GUI, some that can be automated using tools other than GUI test running tools, some that can be automated using GUI test running tools and some that just can't be automated. I'm sure most folk with tools are struggling to meet higher than realistic expectations.

So, what we are left with is an extremely complex product (the browser), being tested by a comparably (probably more) complex product, being controlled by another complex product to make the creation, execution and evaluation of tests of (mainly) server-based software an easy task. Although it isn't of course. Frameworks work best with pretty standard websites or GUI apps with standard technologies. Once you go off the beaten track, the browser vendor, the GUI tool vendor and the framework vendor all need to work hard to make their tools compatible. But all must stick to the HTTP 2.0 protocol which is 10 years(?) old. How many projects set themselves up to use bleeding edge technology, and screw the testers as a consequence? Most, I think.

So. There we have it. If you are fool enough to spend £4-5,000 per seat on a GUI tool. You then to be smart enough to spend another £2,000 or so on a Framework (PER USER).

Consider the alternative.

Suppose you knew a little about HTML/HTTP etc. Suppose you had a tool that allowed you to get web pages, interpret the HTML, insert values to fields, submit the form, execute the server based form handler, receive the generated form, validate the form in terms of new field values, save copies of the received forms on your PC, compare those forms with previously received forms, and deal with the vagaries of secure HTTPS, and ignore the complexities of the user interface. The tool could have a simple script language, based on keywords/commands, stored in CSV files, managed by Excel.

If the tool could scan a form, not yet tested, and generate the script code to set the values for each field in the form, you'd have a basic but effective script capture facility. Cut and paste into your CSV file and you have a pretty effective tool. Capture the form map (not gui – you don't need all that complexity, of course) and use the code to drive new test transactions.

That's all pretty easy. The tool I've built does 75-80% of this now. My old Systeme Evolutif wesite (including online training) had 17,000 pages, with around 100 Active Server Pages script files. As far as I know, there's nothing the tool cannot test in those 17,000 pages. Of course most are relatively simple. But they are only simple in that they use a single technology. There's thousands of lines of server-base code. If/as/when I created a regression test pack for the site, I can (because the tool is run on the command line) run that test every hour against the live site. (Try doing that with QTP). If there is a single discrepancy in the HTML that is returned, the tool would spot it of course. I don't need to use the GUI to do that. (One has to assume the GUI/browser behaves reliably though).

Beyond that, a regression test based on the GUI appearance would never spot things in the HTML unless you wrote code specifically to do that. Programmers often place data in hidden fields. By definitition, hidden fields never appear on the GUI. GUI tools would never spot a problem – unless you wrote code to validate HTML specifically. Regression tests focus on results generated by server-based code. Requirements specify outcomes that usually do not involve the user interface. In most cases, the user interface is entirely irrelevant to the successful outcome of a functional test. So, a test tool that validates the HTML content is actually better than a GUI tool (please note). By the way, GUI tools don't usually have very good partial matching facilitites. With code-based tools, you can use regular expressions (Regexs). Much better control for the tester then GI tools.

Finally. If you use a tool to validate returned messages/HTML, you can get the programmer to write code that syncs with the test tool. A GUI with testability! For example, the programmer can work with the tester to provide the 'expected result' in hidden fields. Encrypt them if you must. The developer can 'communicate' directly with the tester. This is impossible if you focus on the GUI. It's really quite hard to pass technical messages in the GUI without the user being aware.

So. A tool that drives server-based code is more useful (to programmers in particular because you don't have the unnecessary complexities of the GUI). They work directly on the functionality to be tested – the server based code. They are simpler to use. They are faster (there's no browser/GUI and test tool in the way). They are free. AND they are more effective in many (more than 50%?) cases.

Where such a tool COULD be used effectively, who in their right mind would choose to spend £6,000-7,000 per tester on LESS EFFECTIVE products?'

Oh, and did I say, the same tool could test all the web protocols MAIL, FTP etc. and could easily be enhanced to cover web sevices (SOAP, WSGI blah blah etc.) – the next big thing – but actually services WITHOUT a user interface! Please don't get me wrong, I'm definitely not saying that GUI automation is a waste of time!'

In anything but really simple environments, you have to do GUI automation to achieve coverage (whatever that means) of an application. However, there are aspects of the underlying functionality that can be tested beneath the GUI and sometimes it can be more effective do do that but only IF there aren't complicated technical issues in the way (that would be hidden behind the GUI and the GUI tool ignores them).

What's missing in all this is a general method that guides testers to using manual, automation above or below the GUI. Have you ever seen anything like that? One of the main reasons people get into trouble with automation is because they have too high expectations and are overambitious. It's the old 80/20 rule. 20% of functionality dominates the testing (but could be automated). Too often, people try and automate everything. Then 80% of the automation effort goes on fixing the tool to run tests of the least important 20% of tests. Or something like that. You know what I mean. The beauty of frameworks is they hide the automation implementation details from the tester. Wouldn't it be nice if the framework SELECTED the optimum automation method as well? I guess this should depend on the objective of a test. If the test objective doesn't require use of the GUI – don't use the GUI tool! Current frameworks have 'modes' based on the interfaces to the tools. Either they do GUI stuff, or they do Webservices stuff or... But a framework ought to be able to deal with gui, under the gui, web services, command-line stuff etc. etc. Just a thought.

I feel a paper coming on. Maybe I should update the 1997 article I wrote!

Thanks for your patience and trigging some thoughts. Writing the email was an interesting way to spend a couple hours, sat in a dreary hotel room/pub.

Posted by Paul Gerrard on July 4, 2006 03:08 PM

Comments

Good points. In my experience the GUI does change more often the the underlying API. But often, using the ability of LR to record transactions is still quicker than hoping I've reverse-engineered the API correctly. More than once I've had to do it without any help fom developers or architects. ;–)

Chris http://amateureconblog.blogspot.com/

Paul responds:

Thanks for that. I'm interested to hear you mention LR (I assume you mean Load Runner). Load Runner can obviously be used as an under the bonnet test tool. And quite effective it is too. But one of the reasons for going under the bonnet is to make life simpler, and as a consequence, a LOT cheaper.

There are plenty of free tools (and scripting languages with neat features) that can be perhaps just as effective as LR in executing basic transactions – and that's the point. Why pay for incredibly sophisticated tools that compensate for each other, when a free simple tool can give you 60, 70, 80% of what you need as a functional tester?

Now LR provides the facilitites, but I wouldn't recommend LR as a cheap tool! What's the going rate for an LR license nowadays? $20k, $30k?

Thanks. Paul.

Tags: #testautomation #gui #regressiontesting

Paul Gerrard My linkedin profile is here My Mastodon Account