Paul Gerrard

First published 08/12/2009

The Project Euler site presents a collection of 'maths-related' problems to be solved by computer – 250+ of them and the site allows you to check your answers etc. You don't need to be a mathematician for all of them really, but you do need to be a good algorithm designer/programmer.

But it also reminded me of a recurring thought about something else. Could the problems be used as 'testing' problems too? The neat thing about some of them is that testing them isn't easy. Some problems have only one answer – they aren't very useful for testers – there is only one test case (or you need simply to write/reuse a parallel program to act as oracle). But others like problem 22 for example provide input files to process http://projecteuler.net/index.php?section=problems&id=22 The input file could be edited to generate variations – i.e. test cases to demonstrate the code works in general, not just a specific situation. Because some problems must work for infinite cases, simple test techniques probably aren't enough (are they ever?)

The Euler problem statements aren't designed for a specific technique – although they define requirements precisely, they are much closer to a real and challenging problem. The algorithms used to solve the problems are a mystery – and there may be many many ways of solving the same problem. (cf screens that simply update records in a database – pretty routine by comparison). The implementation doesn't influence our tests – its a true black box problem.

Teaching testing “the certified way” starts from the wrong end. We teach technique X, give out a prepared requirement (that happens to fit technique X – sort of) and say, “prepare tests using technique X”. But real life and requirements (or lack of them) aren't like that. Requirements don't usually tell you which technique to use! The process of test model selection (and associated test design and coverage approaches) is rarely taught – even though this is perhaps the most critical aspect of testing.

All of which makes me think that maybe we could identify a set of problem statements (not necessarily 'mathematical') that don't just decompose to partitions and boundaries, states or decisions and we should use these to teach and train. We teach students using small applications and ask them to practice their exploratory testing skills. Why don't we do the same with requirements?

Should training be driven by the need to solve problems rather than trot out memorised rote test design techniques? Why not create training exercises (and certification(?)) from written instructions, a specification, a pc and some software to test?

Wouldn't this be a better way to train people? To evaluate their ability as testers? This is old hat really – but still few people do it.

What stops us doing it? Is it because really – we aren't as advanced as we think we are? Test techniques will never prove correctness (we know that well) – they are just heuristic, but perhaps more systematic ways of selecting tests. Are the techniques really just clerical aids for bureaucratic processes rather than effective methods for defect detection and evidence collection? Where's the proof that says they are more effective? More effective – than what?

Who is looking at how one selects a test model? Is it just gut feel, IQ, luck, happens to be on my desk? Is there a method of model selection that could be defined and taught? Examined? Why don't we teach people to invent and choose test models? It seems to me that this needs much more attention that anyone gives it today.

What do you think?

Tags: #teaching #hands-ontraining #Euler #certification #model

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 05/11/2009

we're stood in a boat ankle deep in water. the cannibals are coming to kill and eat us. The testers are looking for the holes in the boat saying – we can't push off yet the river is full of hungry crocodiles.

The testers are saying – if we push off now, we'll all die.

The skipper is saying – if we don't push off soon, we'll all die.

I'ts the same with software.

Tags: #ALF

Paul Gerrard My linkedin profile is here My Mastodon Account

Background

Michael Bolton recently posted a message on LinkedIn as follows (https://www.linkedin.com/posts/michael-bolton-08847_low-code-testing-tools-are-really-low-testing-activity-6921751556463685633-pcQS:

“Low-code testing tools” are really “low-testing code tools”

In what follows I infer no criticism of Michael or the people who responded to the post, whether positive or negative.

I want to use Michael’s proposition to explain why we need to be much more careful about:

• How we interpret what other people say or write
• How we assess its clarity, credibility, logical consistency or truthfulness
• Whether we understand the assumptions, experience or agenda of the author, or of ourselves as the audience
• And so on…

This is a critical thinking exploration of a phrase of a ten-word, five-term sentence.

I’ve been reading a useful book recently, “The Socratic Way of Questioning” – written by Michael Britton and published by Thinknetic. Thinknetic publish quite a few books in the critical thinking domain and you can see these here: (https://www.amazon.co.uk/Thinknetic/e/B091TWBHXN/)

I’ll use a few ideas I gleaned from the book, describe their relevance in a semi-serious analysis of Michaels’ proposition and hopefully, illustrate some aspects of Critical Thinking – which is the real topic of this article. I don’t claim to be an expert in the topic. But I’m a decent reviewer and can posit a reasonable argument. I’ll try to share my thought process concerning the proposition and take a few detours on the overall journey to expand on a few principles.

I’ve no doubt that some of you could do a better job than I have.

The importance of agreed definitions

The first consideration is that of definition. How can you have a meaningful discussion without having agreed meanings of the words used in that discussion? Socrates himself is said to spend at least half of his time in argument trying to get consensus on the meanings of words. Plato’s Republic, the most famous Socratic dialogue spends most of its time defining just one term – justice

Turning towards the proposition, what do the terms used actually mean? Do they mean the same thing to you as they mean to Michael? Is there an agreed, perhaps universally agreed definition of these terms? So, what do the following terms mean?

• Code
• Testing
• Tools
• Low-Code
• Low-Testing

Now, in exploring the potential or actual meanings of these terms, I’ll have to use some, perhaps many terms that I won’t define. Defining them precisely in a short article is inevitably going to get very cumbersome – so I won’t do that. Of course, we need an agreed glossary of terms and their definition as the basis of the definitions at hand. But it’s almost a never-ending circle of definitions. The Oxford English Dictionary, when I last looked, has compiled over 600,000 definitions of words and phrases covering “1000 years of English” (https://public.oed.com/history/">https://public.oed.com/history/).

Recently, I have been doing some research with ‘merged’ regular English dictionaries and some of the testing-related glossaries available on the internet. I now have some Python utilities using open-source libraries to scrape web pages and PDF documents and scan them for terms defined in these sources as well as detection of phrases deemed ‘meaningful’. (What meaningful means is not necessarily obvious). I'll let you know what I'm up to in a future post.

We have a real problem in the language used in our product and service marketing, and sadly our exchanges on testing topics in discussions/debates

Definitions

Here goes with my definitions and sources, where I could identify and/or select a preferred source. I have tried to use what I think might be Michael’s intended definition. (I am surely wrong, but here goes anyway).

Code: generally defined as a programming or script language used to create functional software to: control devices; capture, store, manipulate and present data; provide services to other systems or to humans. In this context, we’re thinking of code that controls the execution of some software under test.

Testing: …is the process of evaluating a product by learning about it through experiencing, exploring, and experimenting, which includes to some degree: questioning, study, modelling, observation, inference, etc. (Source: (https://www.satisfice.com/blog/archives/856) – there’s a security vulnerability on the page by the way – so use an incognito window if your browser blocks it).

Tools: Oxford English Dictionary): a device or implement, especially one held in the hand, used to carry out a particular function. Merriam Webster: something (such as an instrument or apparatus) used in performing an operation or necessary in the practice of a vocation or profession. In our context – software test execution automation – we refer to software programs that can: stimulate a system under test (SUT); send inputs to and receive outputs from the SUT; record specific behaviours of the SUT; compare received outputs or recorded behaviours with prepared results. And so on.

Low-Code: There is no agreed definition. Here is an example relating to software-building which is reasonable (https://www.mendix.com/low-code-guide/">https://www.mendix.com/low-code-guide/): “Low-code is a visual approach to software development that optimizes the entire development process to accelerate delivery. With low-code, you can abstract and automate every step of the application lifecycle to streamline the deployment of a variety of solutions.” I’d substitute software test development for software development. In the context of testing, the terms ‘codeless’ is more often used. This definition appears on a tools review website ((https://briananderson2209.medium.com/top-10-codeless-testing-tools-in-2020-every-tester-should-know-2cb4bd119313)): “Simply put, codeless test automation means creating automated tests without the need to physically write codes. It provides testers lacking sophisticated programming skills with project templates for workflow, element libraries, and interface customization.”

Low-testing: I’m pretty certain Michael has invented this term without defining it. But we can guess, I think. There are two alternatives – either (a) the quantity of testing is lower (than what?). We don’t know. But let’s assume it refers to ‘enough’... or (b) the quality of testing is lower than it should be (by Michael’s standard, or someone else’s – we don’t know). I suspect (b) is the intended interpretation.

Interpreting the Proposition

Since we now have some working definitions of the individual terms, we can expand the proposition into a sentence that might be more explicit. We’ll have to document our assumptions. This carries a risk: have we ‘guessed’ and misunderstood Michael’s purpose, thinking, assumptions and intended meaning?

Anyway. Here goes:

“Using test execution tools that require less (or no) coding means the quality (or quantity, possibly) of your testing will be reduced.”

Here are our assumptions in performing this expansion:

• The ‘low-testing’ phrase implies the lower quality or quantity testing is achieved. (Does it matter which? It would probably help to know, of course).
• Only using such tools causes this loss of quality or quantity. Does having such tools, but not using them, not?
• Having knowledge of some of Michael’s previous writings about tools and testing justifies our (still ambiguous) expanded phrase, I believe.

Ambiguities

We have several ambiguities in our reformulation of the original proposition:

• Do low-code tools have the effect of lowering the quality/quantity of testing or do ‘high-code’ – that is, all tools – do this?
• Do tools lower the quantity, or do they lower the quality of testing? (Or both?)
• Lowered compared to what? Testing without tools? (Or see below… tools with ‘higher-code’)
• What is meant by ‘the quality of testing’? How is that defined? How could it be measured? Evaluated?
• The quantity of testing might be reduced – but might this be a good thing, if the quality is the same or improved?
• There’s no mention of the benefits of using (low-code) tools – but is the overall effect of using tools (low or high code) detrimental or beneficial?
• There is no mention of context here. Is the proposition true for all contexts or just some? Could using such tools possibly be a universally best practice? Or a universally bad one?

Why This Proposition?

Is this proposition making a serious point? Does it inform a wider audience? I don’t think so. It’s clickbait. Some people will (perhaps angrily) agree and some (angrily) disagree. These people will probably be driven by preconceptions and existing beliefs. Do ‘thinking people’ respond to such invitations?

It’s a common tactic on internet forums and I’ve used the tactic myself from time to time. Although I think it’s a lazy way of starting a debate because the ‘thoughts from abroad/musings/worldview/the toilet’ (strike out what doesn’t apply) – tend not to trigger debate, they just cause some folk to extol/confirm their existing beliefs.

So, don’t ask me, ask Michael.

Summary

I’ve written over a thousand words so far, analysing a ten-word, five-term proposition. I’m sure there is much more I could have written if I thought more deeply about the topic. A word-ratio of 100 to 1 isn’t surprising. Many books have been written discussing less complicated but more significant propositions. Love, justice, quality, democracy… you know what I mean.

The lack of agreed definitions of the terms we use can be a real stumbling block, leading to misinterpretations and misconceptions hardly likely to improve our communications, consensus and understanding. Even with documented definitions there are problems if people dispute them.

Before you engage in a troublesome conversation of a proposition, ask for definitions of the terms used and try, with patience, to appreciate what the other person is actually saying. You may find gold or you may demolish their proposition. Or both.

Finally

Am I suggesting you apply critical thinking to all conversations, all communications? No of course not. Life is too short. And surely, if you analyse all your partner’s words you are bound to land in scalding hot water. So, like testing and all criticism – it has its place. A weapon of choice, in its place.

Critical thinking, like testing, should be reserved for special occasions. When you really want to get to the bottom of what an author, speaker or presenter is saying or a supplier is supplying. How can you agree or disagree without understanding exactly what is being proffered? Critical thinking works.

What’s my Response to the Proposition?

“After all this circumlocution, what do you think of the proposition, Paul?”

I think there are more interesting propositions to debate.

Tags: #thinkingtools #testing #low-codetesttool #codelesstesttool #criticalthinking #clickbait #testingterminology #CriticalThinking

Paul Gerrard My linkedin profile is here My Mastodon Account