Paul Gerrard

My experiences in the Test Engineering business; opinions, definitions and occasional polemics. Many have been rewritten to restore their original content.

First published 14/04/2015

Why Write this Blog?

Last week there was a flurry of tweets related to the perceived lack of female keynote speakers at a 'well-known software testing conference'. Various views were expressed:

  • No, there weren't enough women keynotes on the conference circuit.
  • Perhaps we should have all-women conferences?
  • Perhaps we have a women keynote quota system for conferences?
  • The http://speaking-easy.com/ website aims to connect women wanting coaching in conference speaking to get into the apparently closed circuit of male conference and keynote speakers.
  • And so on.

It was suggested that perhaps a quota system, mandating a minimum number of women speakers on any programme be applied. Personally, I'm reluctant to back that approach. It would undermine the efforts of women who get onto that same programme through their effort and on merit. I believe that for a conference to have credibility to its delegates and the speakers themselves, it has to be a level playing field. So that's that then. Won't women always be in the minority when it comes to conference appearances? I don't think so. I see no reason why there can't be more women speakers and keynote speakers at conferences. I see no reason why some less-than-inspiring male speakers can't be displaced by more inspiring women speakers. Is success in becoming an in-demand keynote speaker based on luck, or favouritism or is it one of those black arts where the secrets are protected by a small guild of people with funny handshakes? No, I have to tell you that the secret to success is mostly common sense – with some talent and hard work thrown into the mix. I want to share what I know about being an in-demand keynote. It works for me. Perhaps it will work for you (whatever your gender).

Most Keynote Speakers are Uninspiring

By and large, the performances of keynote speakers – at least half in my opinion – are poor. 'Surely not!', you might say. 'Aren't these men (usually) the cream of the speaking fraternity? Are they not thought leaders, gurus and inspirational speakers?' Perhaps in the mind of some programme chairs they are, but not usually in mine. At the outset, I must say that I am difficult to please. Even so, I think there are definite reasons for less than stellar performances from a lot of keynotes. Some keynotes are there simply because they work for 'a large company'. Joe Soap from Amafacemicrogooglyblahblah Corporation. They are there because they have an impressive sounding title and work for a company who are making a move in a particular market (or are nowhere in a market and want to be). Perhaps they have something to say, but usually, it's nothing that can't already be found on their corporate website. Maybe their company is a sponsor. Maybe they work for a bank. Maybe the programme chair wants a job? Then there are the speakers who are the self-promoters. Those who spend half an hour talking up the risks of not doing such and such a thing. Then they spend the rest of their slot espousing approaches, methods or tools that just happen to be part of their market offering. Tool vendors tend not to get keynote slots as they find it 'too hard' not to give sales pitches. But it seems that service and training companies aren't so prejudiced against by programme chairs. Perhaps they should be. Trainers and coaches, being the most practiced and well-positioned to promote themselves, are prominent on the conference keynote circuit. Call me cynical. Maybe I am. But that's the last of the negativity.

Conference Realities

Every programme chair wants to have the very best talent on show in keynotes and their conference programme as a whole. It makes for an excellent conference and an enjoyable and informative experience for all of the delegates and speakers alike. But conferences are, in the main, businesses too. The organisers invest a lot of time and money through the year. Venues often need booking and demand substantial deposits, years in advance. So organisers tend to be quite risk-averse. In the mind of the organisers, there is always a need to balance the need for the best, newest, most interesting speakers with the need to market and sell the conference to the world outside. People want to see at least some familiar faces on conference programmes and some 'big names' too. So there's a natural conservatism at work. The exception to this conservatism is where a conference operates in a market or domain that is exploding. I went to the Mobile World Congress show in Barcelona last month. With 2,500 exhibitors and 92,000 visitors and (I am guessing) EUR 50-100m income or more. But they go for the big names and don't take many chances. Compared with the multitudes in the exhibition, I guess there were only a thousand or so people in the conference part of the show. The conference isn't their priority I suppose. TED talks, either the larger international shows or the smaller provincial affairs are remarkably difficult shows to get into. You really do need to be a world-renowned speaker, politician, scientist, doer of good works, best-selling author or philanthropist. And yes, there are also some speaking opportunities at events that pay $100,000 for an appearance, but these are out of reach unless you are an ex-president, prime minister or major-winning golfer. These are exceptional platforms for speakers. I want to concentrate on more mundane, more technical conference opportunities that pay modest fees to make it worthwhile for the speaker. That's enough background – let's get down to business.

What you Need to be an In-Demand Keynote Speaker

I will share with you what I believe you need to do (and be) to be a keynote speaker in demand. Not the also ran, averagely dull keynote. No. You should aim to be one of the keynotes that people in the circles you move in know, respect and want on their conference programmes because you are both a good speaker and a 'draw'. Here's how.

Motivation

The first and most important attribute you need to be a keynote speaker is your desire to be one. That's a simple statement, and it's a bit more complex than that, but your motivation is the key. This motivation to be a keynote is not the same as wanting to make money, promote your business or yourself. I'm afraid, if these are your motivations, most people will see through you. Not all will. You might have business partners, friends or acolytes who adore you, but these will still be a small minority, unless you frequent cliquey, isolated events or vendor shows. No, the motivation has to be that you think you can actually do a better job than other people at speaking, or I could say, communicating or even (get this!) performing. When you sit back and listen to someone else's keynote talk, you get angry enough to both think and say, "I can do better than that. And I'll tell you why". In your "Keynote Capability Self-Assessment", these are the boxes you need to tick.

You Have to be Driven

The first attribute you need is perhaps a certain type of arrogance. You must sincerely think that you can stand up in front of a thousand or more people and tell them what to do. You must honestly believe that you know better than they do (for why else would you be lecturing them in the first place?) I have said arrogance but that doesn't quite cut it. It's a combination of hutzpah, confidence, courage, belief and sheer bloody-mindedness. I can't quite put my finger on it, but you have to be willing and able to put yourself 'out there'. When you're in the spotlight and someone in the faceless crowd challenges you with a piercing or even a stupid question, you have to want to reinforce your message, not just defend it. You must want to be, like in the children's' game, the 'King of the Castle'. There are few things more toe-curlingly (or deliciously) embarrassing than a keynote who is caught out. It's a bit like those situations when a member of the public asks a politician a simple, direct question and the hapless politician has no credible response. This keynoting game - it's not for everyone.

You Must Have Something to Say

Of course I'm sure you have, but it is worth emphasising that if you are simply trotting out clichés, mantras, received wisdom, or low-risk restatements of common sense or even other well-regarded speakers' ideas, you might get away with this in a track talk, but not a keynote. Having said that, there is a place in all conferences for people who can communicate difficult concepts or can convince people to adopt what might appear to be simple things that are hard to achieve. But this puts you in the 'excellent communicator/motivator' bracket.

You must be a Good Communicator

Are you a good communicator? How do you know? Having a large vocabulary, using long words (when short ones will do) is not necessarily a good thing. Great communicators are not (usually) loudmouths, rabble rousers or soapbox orators. Look up a definition of communication skills and use that for reference. Suffice to say, I have seen fantastic talks from people who are absolutely not natural speakers. TED talks are often good examples where the speakers are not slick, but their material and their natural, no frills delivery gets their message over beautifully. It's a really hard one to judge oneself, so I recommend you ask advice of friends or people who have heard you speak in the recent past. But, do not think you can wing it, or think that your poor skills are the fault of your stupid audience.

Know Your Audience

Having something interesting or useful to say is not your decision. It is a matter of other peoples' opinion. You might think you have pearls of wisdom to share, but your target audience might not. Fads, trends and revolutions come and go. Being on the bleeding edge of a new approach might suit your target audience. But it might also be too big a step for them to appreciate or even, most embarrassingly, they think of your talk as old hat or just inappropriate. You have to know your market, what they know and don't know, what their problems and aspirations are and what will go down well or like a lead balloon. Knowing your audience is at least important as motivation. So, these are the personal attributes of you as a speaker that you must pay attention to.

Getting Your First Keynote – Catch 22

But how do you get selected to speak at conferences? The best product on the planet - you, in this case, will never sell itself without some marketing. This is where most people fall down. Probably the most important aspect here, is experience. Most programme chairs want experienced keynotes, not first timers. First timers pose a huge risk in the eyes of organisers and programme chairs. That's another stopper then. How to get that first keynote without previous experience? How do I get experience if I can't get my first keynote? There are some simple (note, I don't mean easy) ways of getting your keynoting career off the ground.

Create Your Own Conference; Hire Yourself

First, how about creating your own conference and putting yourself on the bill? Now this is not as arrogant or as crazy as it seems. If you want to talk about a new topic that you think is important, perhaps there is no conference in existence that focuses on the themes you are interested in. (If there is, you are already playing catch-up, aren't you?) Perhaps there is no regional conference in your part of the world. Perhaps there's a local group that needs livening up. Offer to help organise and suggest that having a keynote speaker might sell the event more. Volunteer to be the first.

Organise a Conference and Network

In any circumstance, get yourself involved in a conference as an organiser, programme secretary or part of the selection or management team. Put this on your CV. Being part of a programme team means you'll have to find keynotes from time to time. Get in touch with other programme chairs, ask them for advice and names of other keynotes who have performed well for them. Make them aware that you are choosy and that you know a good keynote from a bad one. (You happen to be a keynote too). Become part of the 'keynote employers' circle – make people aware you know their business.

Never Create a Track Talk Again

Only pitch keynote talks. From now on, you only ever write keynote talks. If you have to propose using an online form, call your talk a keynote regardless of whether you are pitching for a keynote or track. Make the reviewers believe that your talk is a cut above the average proposal. You never know, if the chair needs a keynote they might pick yours.

Pitch Multiple Talks

I can hear conference committees groan right now... Give the programme chair or committee a choice. If they want you on the programme at all, when they see a gap, your excellent proposal might just slot in. But very often, conference programmes get shuffled around quite dramatically before they are settled. Think of it like a game of cards. The more cards you have in the deck, the more likely your card will come off the top. Of course, you need to have the content for two or three excellent talks. Your number 1 talk might be great but suppose it clashes with someone else's talk? Give the chair the opportunity to select your number 2 talk. Most importantly, never take an existing abstract, change a few buzzwords and pitch that as a second or third offering. If you did have a good talk to start with, it will be obvious what you've done and all your proposals will be tarred with the same brush as hacks – and will be rejected. You must put the same effort into all of your proposals. They must have variety. They must all be 'top-drawer'.

How Many Talks Should I Have?

Some speakers get away with creating one new talk every few years and bang it to death. Maybe they concentrate on writing terrific talks. I try and create two or three talks on new topics every year. For each of these topics, if someone chooses them, I create a tutorial from them. I usually pitch topics as both keynotes and tutorials of course. More cards in the game. As some of your talks peak and seem to become less popular, you need other talks that are leading-edge, novel or topical that are advancing to replace them. Maybe you have three talks:
  • Last year's talk – reliable – some people will take it again
  • This year's talk – evolving – popular and seems to be this year's favourite
  • Your new talk – experimental – maybe run once or twice as a track so far.

Pay Close Attention to Calls for Papers

Target the conferences you want to speak at and diarise the key dates for each. Look out for where the call for papers will be announced. If you know who next year's chair is ask them what kind of keynotes might they look for – they might take note and get back to you later.

Before, or just as the call is published get in touch with the programme chair or someone on the programme committee. Ask them what would be a perfect pitch for a keynote that aligns with the theme or, what kind of topic would be a good 'off piste' talk they'd consider for the programme. They'll probably trot out the conference theme of course. But the key question to ask is, 'when will the keynotes be decided?' Make sure you send a well-written abstract for a killer keynote a week or two before this date. This might not be part of the standard call. But, having offered a talk, I would expect the chair to keep it – just in case.

Submit to Keynotes, Offer them as Tracks

Submit to keynotes, knowing that you'll probably only get a track talk. Make sure you add the phrase 'this is a keynote talk' in the abstract. When a programme team are discussing what's in and what's out, it's unusual for them to discount a talk until they have to. Programme teams have a hard job and in some ways are quite indecisive because they are somewhat democratic. So your talk is quite likely to have one advocate in a programme team. But it has to be a good abstract of course to catch the eye of your advocate.

Offer to be a Reserve

It seems obvious, but medium-sized conferences that won't get hundreds or thousands of offers of talks sometimes struggle to get enough good proposals. Offer to be a reserve track presenter, tutorial giver or keynote. If there's a gap in any programme, especially in the shows that you help with – volunteer. Make it known that you are happy to step in and save the day in a crisis.

At conferences you attend, remind the organiser or chair before the opening talk that, 'I always carry a spare talk on a USB stick with me'. It will be noticed. If there's a disaster and your talk fits, you might just get the call.

Also, because programme chairs often network to learn of new keynotes, if another programme chair enquires and wants a keynote who can step up in a hurry – you might be remembered.

In-House Conferences

Quite often, people who hear you speak and appreciate your talk ask you to come and present at one of their internal conferences or get-togethers. Never say no, unless the hassle is too great or they ask you to speak on a subject you are not expert in. As often as not, internally they will call you a keynote speaker – it makes you (and the organiser) sound more important – as you are of course. If this is your first keynote – no problem – do the talk, get it on your CV. You're on the move.

Never Say No

This might sound like a controversial suggestion. What I mean is never say no to an invite to speak at what you regard as a 'good' conference or one that won't cost you a lot in money or time. Always be open to go to new shows. The reason I say this, is if you are just getting started, get as much experience as you can.

Some years ago, I was invited to speak at the 'Northern {insert obscure Midwest US state here} Software Testing Association'. I'd never heard of it, and had to use an atlas to find the location. But I met some very smart and lovely people, they treated me very kindly and I got several fantastic stories that I've been using for years and years – in keynotes.

Rewards

Ah yes – the fantastic fees you get as a keynote speaker. Or not. The fees cover your expenses and a bit more besides – if you're lucky. To be paid to pontificate to a group of friendly peers is also very flattering. But I have to say, unless you are an ex-president, business or golfing icon you can't make enough money to live on.

Actually, to create a good keynote talk takes a considerable amount of time, and you find that you have to re-use experience acquired over many years of being a practitioner, researcher and dreamer. The fee rate isn't quite so attractive when you take all that time into account. In fact – the rate sucks.

If you aspire to be an 'in demand keynote', you have to settle for two main rewards. The adrenalin and energy boost of being on stage in front of your peers is one. The other is the occasional person saying something very complimentary about your talk.

It sometimes ends in tears.

Tags: #WomeninTech #BeingaKeynote

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 27/06/2014

The nice folk at Testing Cup in Poland have posted a video of my keynote on You tube.

 



Tags: #ALF

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 08/08/2014

The nice folk at Testing Cup in Poland have posted a video of my keynote on You tube.

I originally gave a EuroSTAR keynote in 2002 titled 'What is the Value of Testing and how can we improve it?'. This talk brings the topic up to date but also introduces the New Model Testing that I'm working on at the moment. 

I've written a  paper that describes New Model Testing that can be downloaded here.



Tags: #valueoftesting #NewModelTesting #TestingCup

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 27/03/2013

Did you know? We’re staging some webinars

Last night, we announced dates for two webinars that I will present on the subject, “Story-Based Test Automation Using Free Tools”. Nothing very exciting in that, except that it’s the first time we have used a paid-for service to host our own webinar and marketed that webinar ourselves. (In the past we have always pitched our talks through other people who marketed them).

Anyway, right now (8.40 PM GMT and less than 24 hours since we started the announcements) we have 96 people booked on the webinar. Our GoToWebinar account allows us to accept no more than 100. Looks like a sell-out. Great.

Coincidentally, James Bach and Michael Bolton have revisited and restated their positions on the “testing versus checking” and “manual versus automated testing” dichotomies (if you believe they are dichotomies, that is). You can see their position here: http://www.satisfice.com/blog/archives/856.

I don’t think these two events are related, but it seemed to me that it would be a good time to make some statements that set the scene for what I am currently working on in general and the webinar specifically.

Business stories and testing

You might know that we (Gerrard Consulting) have written and promoted a software development method (http://businessstorymethod.com) that uses the concept of business stories and have created a software as a service product (http://businessstorymanager.com) to support the method. The method is not a test method, but it obviously involves a lot of testing. Testing that takes place throughout the development process – during the requirements phase, development phase, test phase and ongoing post-production phases.

Business stories are somewhat more to us than ‘a trigger for a conversation’, but we’ll use the term ‘stories’ to refer to them from now on.

In the context of these phases, the testing in scope might be called by other names and/or be part of processes other than ‘test’. Requirements prototyping, validation, (Specification by Example/Behaviour-Driven Development/Acceptance Test Driven Development/ Test-Driven Development – take your pick), feature-acceptance testing, system testing, user-testing and regression testing during and after implementation and go-live.

There’s quite a lot of this testing stuff going on. Right now, the Bach-Bolton dialogue isn’t addressing all of this in a general way, so I’m keeping a watching brief on events in that space. I look forward to a useful, informant outcome.

How we use (business) stories

In this blog, I want to talk specifically about the use of stories in a structured domain-specific language (using, for example Gherkin format (see https://github.com/cucumber/gherkin) to example (and that is a KEY word) requirements. I’m not interested in the Cucumber-specific extensions to the Gherkin syntax. I’m only interested in the feature heading (As a…/I want…/So that…) and the scenario structure (given…/when…/then…) etc. and how they are used to test in a broader sense:

  • Stories provide accessible examples in business language of features in use. They might be the starting point of a requirement, but usually not a full definition of a requirement. Without debating whether requirements can ever be complete, we argue that Specification by Example is not (in general) possible or desirable. See here: http://gerrardconsulting.com/index.php?q=node/596
  • If requirements provide definitions of behaviour in a general way, stories can be used to create examples of features described in requirements that are specific and, if carefully chosen, can be used to clarify understanding, to prototype behaviours and validate requirements in the eyes of stakeholders, authors and recipients of requirements. We describe this process here: http://gerrardconsulting.com/index.php?q=node/604
  • Depending on who creates these stories and scenarios and for what purpose, these scenarios can be used to feed a BDD, ATDD or Specification by Example approach. The terminology used in these approaches varies, but a tester would recognise them as a keyword-driven approach to test automation. Are these automated scenarios checks or tests? Probably checks. But these automated checks have multiple goals beyond ‘defect-detection’.

Story-based testing and automation

You see, the goals of an automated test (and let me persist in calling them tests for the time being) varies and there are several distinct goals of story-based scenarios as test definitions.

In the context of a programmer writing code, the rote automation of scenarios as tests gives the programmer a head start in their test-driven development approach. (And crafting scenarios in the language of users segues into BDD of course). The initial tests a programmer would have needed to write already exist so they have a clearer initial goal. Whether the scenarios exist at a sufficiently detailed level for programmers to use them as unit-tests is a moot point and not relevant right now. The real value of writing tests and running them first derives from:

  1. Early clarification of the goal of a feature when defined
  2. Immediate feedback of the behaviour of a feature when run
  3. When the goal is understood and the tests pass, then the programmer can more safely refactor their code
Is this testing? 2 is clearly an automated test. 3 is the reusable regression test that might find its way into a continuous integration and test regime. These tests typically exercise objects or features through a technical API. The user interface probably won’t be exercised.

There is another benefit of using scenarios as the basis of automated tests. The language of the scenario (which is derived from the businesses’ language in a requirement) can be expected to be reused in the test code. We can expect (or indeed mandate) the programmer to reuse that language in the naming of their variables and objects in code. The goals of Ubiquitous Language in systems (defined by Eric Evans and nicely summarised by Martin Fowler here http://martinfowler.com/bliki/UbiquitousLanguage.html) are supported.

Teams needing to demonstrate acceptance of a feature (identified and defined by a story), often rely on manual tests executed by the user or tester. The tester might choose to automate these and/or other behaviour or user-oriented tests as acceptance regression tests.

Is that it? Automated story tests are ‘just’ regression tests? Well maybe so.

The world is going 'software as a service' and the development world moves closer to continuous delivery approaches every day. The time available to do manual testing is shrinking rapidly. In extremis, to avoid bottlenecks in the deployment pipeline (http://continuousdelivery.com/2010/02/continuous-delivery/) there may be time only to perform cursory manual testing. Manual, functional testing of new features might take place in parallel with development and automation of functional tests must also happen ahead of deployment because automated testing becomes part of the deployment process itself. Perhaps manual testing becomes a test-as-we-develop activity?

But there are two key considerations for this high-automation approach to work:

  1. I’ve said elsewhere that Continuous Delivery is a beast that eats requirements (http://gerrardconsulting.com/index.php?q=node/608) and for CD to work, then the quality of requirements must be much higher than we are accustomed to. We use the term trusted requirements. You could say, tested and trusted. We, and I mean testers mostly, need to validate requirements using stories so the developers receive both trusted requirements and examples of features in use. Without trusted requirements, CD will just hit a brick wall faster.
  2. Secondly, it seems to me that for the testers not to be a bottleneck, then the manual checking that they do must be eliminated. Whichever tests can be automated should be. The responsibility for automation of checking must move from being a retrospective activity to possibly a developer activity. This will free the manual testers to conduct and optimise their activity in the short time they have available.
  3. There are several spin-off benefits of basing tests on stories and scenarios. Here’s two: if test automation is built early, then all checks can take advantage of it; if automation is built in parallel with the software under test, then the developers are much more likely to consider the test automation and build the hooks to allow it to operate effectively. The continuous automated testing provides the early warning system of continuous delivery regimes. These don't 'find bugs', rather they signal functional equivalence. Or not.

    I wrote a series of four articles on 'Anti-Regression Approaches' here: http://gerrardconsulting.com/index.php?q=node/479. What are the skills of setting up regression test regimes? Not necessarily the same as those required to design functional tests. Primarily, you need automation skills and a knowledge of the internals of the system under test. Are these testing skills? Not really. They are more likely to be found in developers. This might be a good thing. Would it not be best to place responsibility for regression detection on those people responsible for introducing regressions? Maybe developers can do it better?

    One final point. If testers are allowed (and I use that word deliberately) to test or validate requirements using stories in the way we suggest, then the quality of requirements provided to developers will improve. And so will the software they write. And the volume of testing we are currently expected to resource will reduce. So we need fewer testers. Or should I say checkers?

    This is the essence of the “redistributed testing” offer that we, as testers, can make to our businesses.

    The webinar is focused on our technical solution and is driven by the thinking above.

    Last time I looked we had 97 registrants on the 4th April Webinar. If you are interested, the 12th April webinar takes place at 10 AM GMT – you can register for it here: https://attendee.gotowebinar.com/register/4910624887588157952

    Tags: #testautomation #businessstorymethod #businessstories #BusinessStoryManager #BDD #tdd #ATDD

    Paul Gerrard My linkedin profile is here My Mastodon Account

First published 20/09/2017

Revolution

If you are not working on a “Digital” project, the hype that surrounds the whole concept of Digital and that is bombarding business and IT professions appears off-putting to say the least. But it would be wrong to ignore it. The Digital Transformation programmes that many organisations are embarking on are affecting business across all industry and government sectors. There is no doubt that it also affects people in their daily lives.

That sounds like yet another hype-fuelled statement intended to get the attention. It is attention grabbing, but it’s also true. The scope of Digital[1] is growing to encompass the entirety of IT related disciplines and business that depends on it: that is – all business.

It is becoming clear that the scope and scale of Digital will include all the traditional IT of the past, but when fully realised it will include the following too:

  • The IoT– every device of interest or value in the world will become connected; sensors of all types and purpose will be connected – by the billion – to the internet.
  • Autonomous vehicles – cars, planes, ships, drones, buses will become commonplace in the next ten years or so. Each will be a “place on the move”, fully connected and communicating with its environment.
  • Our home, workplace, public and private spaces will be connected. Our mobile, portable or wearable devices will interact with their environment and each other – without human intervention.
  • Robots will take over more and more physical tasks and make some careers obsolete and humans redundant. Robots will clean the city, fight our wars and care for the elderly.
  • Software in the form of ‘bots’ will be our guardian angel and a constant irritant – notifying us of the latest offers and opportunities as we traverse our Smart Cities[2].
  • The systems we use will be increasingly intelligent, but AI won’t be limited to corporates. Voice control may well be the preferred user-interface on many devices in the home and our car.
  • The operations or ‘Digital Storm’ of commerce, government, medicine, the law and warfare will be transformed in the next few years. The lives of mid-21st century citizens could be very different from ours.

Motivation

Still not convinced that Digital will change the world we live in? The suggested scale of change is overwhelming. Why is this happening? Is it hype or is it truly the way the world is going?

The changes that are taking place really are significant because it appears that this decade – the 2010’s – are the point at which several technological and social milestones are being reached. This decade is witness to some tremendous human and technological achievements.

  1. One third of the world is connected; there are plans to connect the remaining two-thirds[3]
  2. The range of small devices that can be assembled into useful things has exploded. Their costs are plummeting.
  3. Local and low power networking technologies can connect these devices.
  4. Artificial Intelligencewhich has promised so much for so many years is finally delivering in the form of Machine Learning.
  5. Virtual and Augmented Reality-based systems are coming. Sony VR launched (13/10/2016) to over 1.8million people and Samsung VR starts at under $100.
  6. Robotics, drone technology and 3D printing are now viable and workable whilst falling in cost.
Almost all businesses have committed to transform themselves using these technological advances – at speed – and they are calling it Digital Transformation.

Ambition

If you talk to people working in leading/bleeding edge Digital projects, it is obvious that the ambition of these projects is unprecedented. The origin of these projects can be traced to some critical, but dateless assumptions being blown away. It’s easy to imagine some Digital expert convincing their client to do some blue-sky thinking for their latest and greatest project. “The rules of the game are changed” they might advise:
  • There need be no human intervention in the interactions of your prospects and customers and your systems[4].
  • Your sales and marketing messages can be created, sent to customers, followed up and changed almost instantly.
  • You have the full range of data from the smallest locale to global in all media formats at your disposal.
  • Autonomous drones, trucks and cars can transport products, materials and people.
  • Physical products need not be ordered, held in stock and delivered at all – 3D printing might remove those constraints.
  • And so on.

Systems of Systems and Ecosystems

According to NASA the Space Shuttle[5] – with 2.5 million parts and 230 miles of wire – is (or was) the most complex machine ever built by man. With about a billion parts, a Nimitz class supercarrier[6] is somewhat more complex. Of course, it comprises many, many machines that together comprise the super-complex system of systems – the modern aircraft carrier.

A supercarrier has hundreds of thousands of interconnected systems and with its crew of 5-6,000 people could be compared to an average town afloat. Once at sea, the floating town is completely isolated except for its radio communications with base and other ships.

The supercarrier is comparable to what people are now calling Smart Cities. Wikipedia suggests this definition[7]:

“A smart city is an urban development vision to integrate multiple information and communication technology (ICT) and IoT  solutions in a secure fashion to manage a city’s assets – the city’s assets include, but are not limited to, local departments' information systems, schools, libraries, transportation systems, hospitals, power plants, water supply networks, waste management, law enforcement, and other community services.”

The systems of a Smart City might not be as complex as those of an aircraft carrier, but in terms of scale, the number of nodes and endpoints within the system might be anything from a million to billions.

A smart city is not just bigger than an aircraft carrier – it also has the potential to be far more complex. The inhabitants and many of the systems move in the realm of the city and beyond. They move and interact with each other in unpredictable ways. On top of that, the inhabitants are not hand-picked like the military; crooks, spies and terrorists can usually come and go as they please.

Unlike a ship – isolated at sea, the smart city is extremely vulnerable to attack from individuals and unfriendly governments and is comparatively unprepared for attack.

But it’s even more complicated than that.

Nowadays, every individual carries their own mobile system – a phone at least – with them. Every car, bus and truck might be connected. Some will be driverless. Every trash can, streetlight, office building, power point, network access point is a Machine to Machine (M2M) component of a Digital Ecosystem which has been defined thus:

“A Digital Ecosystem is a distributed, adaptive, open socio-technical system with properties of self-organisation, scalability and sustainability inspired from natural ecosystems”[8].

Systems of Every Scale

The picture I’ve been painting has probably given you the impression that the Digital systems being now architected and built are all of terrifying scale. But my real point is this: The scale of Digital ranges from the trivial to the largest systems mankind has ever attempted to build.

The simplest system might be, for example, a home automation product – where you can control the heating, lighting, TV and other devices using a console, your mobile phone or office PC. The number of components or nodes might be ten to thirty. A medium complexity system might be a factory automation, monitoring and management system where the number of components could be several thousand. The number of nodes in a Smart City will run into the millions.

The range of systems we now deal with spans a few dozen to millions of nodes. In the past, a super-complex system might have hundreds of interconnected servers. Today, systems are now connected using services or microservices – provided by servers. In the future, every node on a network – even simple sensors – is a server of some kind and there could be millions of them.

Systems with Social Impact

It might seem obvious to you now, but there is no avoiding the fact that Digital systems almost certainly have a social impact on a few, many or all citizens who encounter them. There are potentially huge consequences for us all as systems become more integrated with each other and with the fabric of society.

The scary notion of Big Brother[9] is set to become a reality – systems that monitor our every move, our buying, browsing and social activities – already exist. Deep or Machine Learning algorithms generate suggestions of what to buy, where to shop, who to meet, when to pay bills. They are designed to push notifications to us minute by minute.

Law enforcement will be a key user of CCTV, traffic, people and asset movement and our behaviours. Their goal might be to prevent crime by identifying suspicious behaviour and controlling the movement of law enforcement agents to places of high risk. But these systems have the potential to infringe our civil liberties too.

The legal frameworks of all nations embarking on Digital futures are some way behind the technology and the vision of a Digital Future that some governments are now forming.

In the democratic states, civil liberties and the rules of law are very closely monitored and protected. In non-democratic or rogue states, there may be no limit to what might be done.

Ecosystems of Ecosystems

The span of Digital covers commerce, agriculture, health, government, the media in its various forms and the military; it will affect the care, travel, logistics, and manufacturing industries. There isn’t much that Digital won’t affect in one way or another.

A systems view does not do it justice – it seems more appropriate to consider Digital systems as ecosystems within ecosystems.

This text is derived from the first chapter of Paul's book, “Digital Assurance”. If you want a free copy of the book, you can request one here.

[1] From now on I’ll use the word Digital to represent Digital Transformation, Projects and the wide range of disciplines required in the ‘Digital World’.

[2] See for example, http://learn.hitachiconsulting.com/Engineering-the-New-Reality

[3] Internet.org is a Facebook-led organisation intending to bring the Internet to all humans on the planet.

[4] Referred to as ‘Autonomous Business Models’.

[5] http://spaceflight.nasa.gov/shuttle/upgrades/upgrades5.html

[6] http://science.howstuffworks.com/aircraft-carrier1.htm

[7] https://en.wikipedia.org/wiki/Smart_city

[8] https://en.wikipedia.org/wiki/Digital_ecosystem

[9] No, not the reality TV show. I mean the despotic leader of the totalitarian state, Oceania in George Orwell’s terrifying vision, “1984”.

Tags: #assurance #Digital #ALF #DigitalAssurance

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 29/06/2015



Tags: #BDD #SP.QA #RobotFramework #ATDD

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 13/10/2020

Nicholas Snogren posted on LinkedIn a reference to an “Axioms of Testing” presentation from 2009 and asked me to comment on his “Tenets of Software Testing”. There are some similarities but not many I think, some parallel too, but his question prompted me to give a longer response than I guess was expected. I said...

“Hi, thanks for asking for my opinion. Your tenets look interesting – and although I don't think they map directly to what I've written, they raise points in my mind that need a little airing – my mind grows cobwebby over time, and it's good to brush off old ideas. A bit like exercising muscles that haven't been used for a while haha.”

I give my response as a comparison with my Tester's Pocketbook, and Test Axioms website and presentations. (I notice that some of these posts are around 12 years old and some links don't work (anymore). Some are out of my control, others I'll have to track down and correct others – let me know if you want that.)

Tenets v Axioms

Firstly, let's get our definitions right.

According to dictionary.com, a Tenet is “any opinion, principle, doctrine, dogma, etc., especially one held as true by members of a profession, group, or movement.” Tenets might be regarded as beliefs that don't require proof and don't provide a strong foundation.

From the same source, an Axiom is, “i) a self-evident truth that requires no proof. ii) a universally accepted principle or rule. iii) Logic, Mathematics. a proposition that is assumed without proof for the sake of studying the consequences that follow from it”

I favoured the use of Axioms as a foundation for thinking in the testing domain. Axioms, if they are defensible, would provide a stronger foundational set of ideas. When I proposed a set of Testing Axioms, there was some resistance – here's a Prezi talk that introduces the idea.

James Bach in particular challenged the idea of Axioms and suggested I was creating a school of testing (when schools of testing were getting some attention) here.

By and large, by defining the Axioms in terms that are context-neutral, challenges have tended to be easy to acknowledge, disarm and set aside. Critics, almost entirely from the context-driven school, jumped the gun so to speak – they clearly hadn't read what I had written at the time before critiquing. Only one or two people responded to James' call to arms to criticise the Axioms and challenged them.

The Axioms are fully described in The Tester's Pocketbook – http://testers-pocketbook.com/.

The Axioms of Testing website – https://testaxioms.com/ – sets out the Axioms with some explanation and provides around 50% of the pocketbook content for free.

Axioms caught the attention (and criticism) of people because I pitched them as universal principles or laws of testing. Tenets, being less strident in their definition might not attract attention (or criticism) in the same way.

Immediate Comments on the Tenets

The Tenets are numbered and italicised. My comments in plain text.

  1. A software product’s behavior is exhibited by interactions.
  2. There is potentially an infinite number of possible behaviors in software.

These are properties of software. I'm not sure what 1 says other than behavior is triggered by interactions and presumably observed through interactions. Although a lot of software behaving autonomously might respond to internal events such as the passing of time and might not exhibit any behaviour through interactions e.g. a change of internal state. I'm not sure 1 says much.

Tenet 2 is reasonable for reasonably-sized software artefacts.

In the Tester's Pocketbook, I hardly use the term software. I prefer that we test systems. Software is usually an important part of every system. Humans do not interact with software (except by reading or writing it). Software exists in the context of program compilation, hosted on operating systems, running on devices which have peripherals and other interconnected systems which may or may not have user interfaces.

Basing Axioms on Systems means that the Axioms are open to interpretation as Axioms of testing ANY system (i.e. anything. I don't press that idea – but it's an attractive one). Another 'benefit' is that all of the Systems Thinking principles can also be brought to bear on our arguments. Outside its context, Software is not a System.

3. Some of those behaviors are potentially negative, that is, would detract from the objectives of the software company or users.

I use the term Stakeholders to refer to parties interested in the valuable, reliable behavior of systems and the outcome and value of testing those systems.

4. The potentiality for that negative behavior is risk.

OK, but it could be better worded. I would simply say 'potential modes of failure' rather than negative behaviour.

5. It’s impossible to guarantee a lack of risk as it’s impossible to experience an infinite number of behaviors.

Not really. You can guarantee a no-risk situation if no one cares or no one cares enough to voice their concerns before testing (or after testing). There is always the potential for failure because systems are complex and we are not skilled enough to create perfect systems.

6. Therefore a subset of behaviors must be sampled to represent the risk.

Rather than represent, I would say trigger the failure(s) of concern to explore the risk and better inform a risk-assessment.

7. The ability to take an accurate sample, representative of the true risk, is a testing skill.

Not sure what you mean by sample – tests or test cases, I presume? Representative is a subjective notion, surely; 'true' I don't understand; and a testing skill would need more definition than this, wouldn't it?

8. A code change to an existing product may also affect the product in an infinite number of ways.

I'd use 'ANY' change, to a 'SYSTEM'. Why 'also'? What would you say fits into a 'not only.... but also...' clause? But I'm not sure I agree with this assertion anyway. A code change changes some software artefact. The infinite effects (faulty behaviors?) derive from infinite tests (or uses in production) – which you say in 5 is impossible to achieve. I'm not sure what you're trying to say here.

9. It is possible to infer that some behaviors are more likely to be affected by that change than others.

You can infer anything you like by calling upon the great Unicorn in the sky. How will you do this? Either you use tools which are limited in capability or you might use change and defect history or you might guess based on partial knowledge and experience.

10. The risk -of that change- is higher within the set of behaviors that are more likely to be affected by that change.

Do you mean probability of failure or the consequence of failure? I assume probability. At any rate, this is redundant. You have already asserted this in 9. But it's also more complicated than this – a cosmetic defect on an app can be catastropic and a system failure negligible at times.

11. The ability to accurately estimate a scope of affected behavior is another testing skill.

I would call this the skills of impact analysis rather than testing. Developers are relatively poor at this, even having a far deeper technical knowledge (either they aren't able or lack the time to impact-analyse to any reliable degree). So we rely on testing to catch regressions which is less than ideal. Testers depend on their experience rather than system internals knowledge. But, since buggy systems behave in essentially unpredictable ways, we must admit our experience is limited and fallible. It's not a 'skill' that I would dwell on.

12. The scope and sampling ideas alone are meaningless without empirical evidence.

The scope and sampling ideas have meaning regardless of whether you implement them. I suppose you might say they are useless ideas if you don't gather evidence.

13. Empirical evidence is gathered through interactions with the product, observation of resultant behavior, and assessment of those observations.

The word empirical is redundant. I would use the word 'some' here. We also get evidence from operation in production, for example. (Unless you include that already?)

14. The accuracy and speed of scope estimation, behavior sampling, and gathering of evidence are key performance indicators for the tester.

If you are implying 13 are tester skills, I suppose you could make this assertion. But you haven't said what the value of evidence is yet. Is the purpose of testing only to evaluate the performance of testers? Hope not ;O)

15. Heuristics for the gathering of such evidence, the estimation of scope, and the sampling of behavior are defined in the Heuristic Test Strategy Model.

Heuristics are available in a wide range of sources including software, systems and engineering standards. Why choose such a limited source?

Inspiration for the Tenets

These tenets were inspired by James Bach’s “Risk Gap” and Doug Hubbard’s book “How to Measure Anything.” Both Bach and Hubbard discuss a very similar idea from different spaces. Hubbard suggests that by defining our uncertainty, we can communicate the value of reducing the uncertainty. Bach describes the “knowledge we need to know” as the “Risk Gap.” This Risk Gap is our uncertainty, and in defining it, we can compute the value of closing it. In testing, I realized we have three primary areas of uncertainty: 1) what is the “risk gap,” or knowledge we need to find out, 2) how can we know when we’ve acquired enough of that unknown knowledge, and 3) how can we design interactions with the program to efficiently reveal this knowledge.

There are several interesting anomalies to unpick here:

  • I recall James telling a story about Tom Gilb asserting anything could be measured. James suggested Love and Tom obliged. I don't think James was impressed.
  • 'Defining uncertainty' – how do you do that reliably? Numerically? Objectively? We can put any numbers we like against probability and consequence. Being certain, with or without evidence, is always subjective. People can say they are more certain, but based on ... what? How do we correlate data with a human emotion and use that to make engineering decisions? People can be easily deceived – by themselves, not just by others. Consider this, for example, and this.
  • Risk Gap – how is a quantity of knowledge measured? What units? With what certainty? These are aspects that James has argued against since the early 1990s.
  • Your three challenges 1), 2)and 3) are reasonable as goals. How do Bach and Hubbard argue you achieve them, if not by calling on the subjective opinions of other people?

Some More General Comments

You seem to be trying to 'make a case' for testing as a tool to address the risk of failure in systems. I (like and) use that same approach in a rounder sense in my conference talks and writings, when practicable. My observations on this are:

  1. The logic doesn't flow as it should because of flaws in the individual statements
  2. You have no pre-definition of test, testing or its purpose at the outset, so it's not clear what your destination is
  3. There's no defined goal of testing, other than to gather evidence to (my words) reassess risks and thereby reduce uncertainty (but you don't say why that's a 'good thing')
  4. Testing enables a reassessment of risk, but that reassessment may increase risk if, for example, you find a bug in something that was previously deemed reliable. (there's a bigger conversation to be had, but risk is not a BAD thing, it's the barrier(s) you need to navigate or break through to gain your REWARD).
  5. Extant, significant risks are a BARRIER to accepting or releasing systems. As such, the goal of testing is to provide evidence that to the people who make the decision, those risks are acceptable or negligible (reducing uncertainty, sure but never eliminating it). But the ultimate goal of testing is to show that the system WORKS. Encountering failures is a detour from that goal. The testing goal is broader than exploring risks.
  6. You don't mention stakeholders at all. Why do we test? To provide evidence to testing stakeholders – our customers – so they can make better-informed decisions.

Summary

I don't want to give the impression that I'm criticising Nicholas or am arguing against the concept of Tenets or Principles or Axioms of testing. All I have tried to do is offer reasonable criticism of the Tenets to show that is a) extremely difficult to postulate bullet-proof Tenets, Principles or Axioms and b) it is extremely easy to criticise such efforts by:

  • Exposing flaws in the language used and the logic in an argument that C follows B follows A etc.
  • Identifying implicit assumptions of meaning, scope, dependency and authority and
  • Offering examples of context that contradict, or expose flaws in, the statements made.

I do this because I have been there many times since 2008 and occasionally have to defend the Test Axioms from such criticisms. I have to say, Critical Thinking is STILL a rare skill – I wish criticism were more often proffered as a result of it.

References

  1. The Tester's Pocketbook – http://testers-pocketbook.com/
  2. Axioms of Testing website – https://testaxioms.com/


Tags: #ALF #TestAxioms #Tenets #CriticalThinking

Paul Gerrard My linkedin profile is here My Mastodon Account

First published 10/04/2014

I'm working with Lalitkumar who edits the Tea Time With Testers online magazine. It has a large circulation and I've agreed to write an article series for him on 'Testing the Internet of Everything'. I'll also be presenting webinars to go with the articles, the first of which is here: https://attendee.gotowebinar.com/register/1854587302076137473 It takes place on Saturday 19 April at 15.30pm. An unusual time – but there you go.

You can download the magazine from the home page here: teatimewithtesters.com/

Lalit has asked for questions on the article and I'll respond to these during the webinar. But questions on a more broad range of testing-related subjects, I'll write a response for the magazine. But I'll also blog these questions and answers here.

Questions that result an interesting blog will receive a free Tester's Pocketbook - if you go through the TTWT website and contact Lalit - anything goes. I look forward to soem challenging questions :O)

The first Q&A Will appear shortly...



Tags: #TTWT #TeaTimewithTesters #IOE #IOT

Paul Gerrard My linkedin profile is here My Mastodon Account