There is a statement that has been discussed throughout the testing community that stirs a great deal of controversy: “Test Automation is not Testing, It’s Checking!”.
It is a really powerful statement when describing the benefits of manual testing, specifically exploratory testing. However, the terminology can provide alternative interpretations. The words checking and testing are essential when writing test plans. Checks can be applied to a single validation step within a test case. Meanwhile, a test will consist of several steps including checking steps. As a result, the statement has the unintended consequence of implying that test automation is inferior to manual testing.
The subject of test automation and the way it is used has been debated a great deal and usually yields a variety of opinions. Some are strongly opposed to automated testing and see it as a risky and costly software testing strategy. Others see it as a necessity when testing applications that are forever evolving and increasing in complexity.
Out of all the varying ideas surrounding test automation, there is one thing that almost everyone appears to agree on: Manual testing cannot and must not be completely replaced by test automation. It is unlikely that 100% test automation will ever be possible, at least it won’t be in our lifetime. Even if it were possible, software applications are designed to be used by humans. This makes 100% automation ill-advised as computers do not have the correct mindset required to recognize potential usability issues. If test automation is to be included in a test plan, then it must be backed up with manual testing – scripted and exploratory.
Automated Testing
An automated test will verify that a feature is working exactly the same way it was before. It is really useful when testing areas of the software that have not been changed. If a change has been made that doesn’t directly affect a certain feature, it is reasonable to expect that feature to still work the same way as it did before.
An automated test will run precisely the way it’s been designed to. However, this can be both a gift and a curse. The inability for the test to deviate from the plan means that it is unable to cope when things go wrong unexpectedly. I am going to quote Bas Dijkstra in an earlier blog post because he said it so brilliantly: “[A test with a failed step] simply says ‘f*ck you’ and exits with a failure or exception, leaving you with no feedback at all about the behaviour to be verified in steps 11 through 50”.
It is unlikely that a test used to verify an unchanged feature will fail. Unchanged features can be confirmed as still working and not get affected by changes made in other areas of the software. Automated tests should be used to verify that unchanged features are still working as expected. This allows dedicating more time to manually testing new or edited functionality.
Scripted Manual Testing
Scripted manual tests verify that a feature meets the expected requirements. If a change has been made to a feature, an automated test is likely to fail because of that change. With manual tests, there is scope to implement a workaround when something unexpected happens.
An option does exist for scripted manual tests to cover unchanged features as well. Creating manual test cases is a lot quicker and less costly than developing automated tests. However, tight deadlines usually mean that there is no time to run these tests manually. Thus, these unchanged features do not get tested. The risk that a change to the software has affected a seemingly unrelated feature will always be present.
Exploratory Testing
Exploratory testing brings a completely different benefit. Both automated and manual scripted tests cover features that the tester already knows about. Alternative workflows and features are explored through exploratory testing. New defects are often found. These are defects which were never thought of when the test scripts were written. However, just like test automation, this must never be the sole method for a testing strategy. Without a script or even a record of what has been done, it is unclear what has been tested. Often, there isn’t even any evidence that testing actually took place.
Returning to the original statement, both forms of scripted testing (manual and automated) can be easily defined as testing. But I do not see exploratory testing as testing. Testing and checking steps involve analyzing the quality of the software against known criteria. Scripts will contain expected outcomes which are compared against actual outcomes. Exploratory tests will have no such thing. Exploratory testing is exploring the unknown, going off the beaten track and discovering those new and unidentified bugs. These are unlikely to be found with scripted testing. They will also better identify usability issues which automated testing will never be able to cover.
Summary
Running automated tests that cover unchanged areas of the software, frees up time and resource for running manual tests that cover new or changed features in the software. This is not essential but it can drastically help to improve the overall test coverage. A variety of scripted manual tests and unscripted exploratory tests should be used to test new and edited features. This demonstrates that all features have been tested. It also allows for additional exploration of unknown areas that will find any unexpected defects. All 3 methods bring a great deal of value to a testing project and are instrumental in ensuring the quality of any developed piece of software.
Do you agree or do you believe that there is a method that is superior to another? Let us know your thoughts in the comments below ➡
If someone is telling you that “TEST AUTOMATION IS NOT TESTING, IT’S CHECKING”, they are, in my opinion, confused. If they think or claim they’re quoting me or James Bach, trust me: they’re not. This is what we say on the matter: http://www.satisfice.com/blog/archives/856
The confusion is highlighted by a couple of problems that we would highlight in their statement.
The first problem is in saying that “Y is not X”. That’s technically true; a tree is not a forest; skating is not playing hockey; and stirring is not cooking. But it’s much clearer to point out that Y is a part of X. Trees are parts of forests; skating is part of hockey, and stirring happens in cooking. Test automation is not testing in just the same way that an automatic transmission is not driving; in the same way that biting is not eating; in the same way that compiling is not programming.
This is not to say that an automatic transmission is not valuable. An automatic transmission aids us in driving, but it isn’t driving. Gear shifting is part of driving… but it’s a part that can done by very sophisticated and invisible (to the driver) machinery. It’s the part of driving that the human driver can delegate doesn’t have to worry much about.
This is not to say that biting is not valuable, either. Biting affords us the opportunity to eat without choking; yet biting is not eating. Biting. a mechanical activity, happens inside eating, as a part of eating, which tends to be a highly social activity. Biting can even be done with tools, like knives or blenders.
This is not to say that compilers are not valuable. Indeed, source code has to be compiled into machine code for the machine to handle it. People used to perform that step, but they don’t any more; compiling is a part of programming that can be delegated to the machinery. But nobody confuses compiling and programming; compiling happens as a part of programming.
And this is not to say that checking is not valuable either. Checking is a *tactic* of testing. It’s useful and powerful in some aspects of testing, and not so useful and less powerful in others.
When people say “TEST AUTOMATION IS NOT TESTING, IT’S CHECKING”, what they might be trying to do (somewhat ineptly) is to remind you how important the tester is. The check, is the part of testing that is skill-free; that can be done by the machinery. Testing is the larger set of activities: identification of risk, design of the check, programming of the check, (and then after the check has been run) the interpretation of the result and framing that for your testing client. That’s a big deal! Those are the parts of testing that the machinery can’t do; that people need YOU to do.
Automated checking is one (and only one) application of automation *within* testing… which brings us to the second problem in the statement “TEST AUTOMATION IS NOT TESTING, IT’S CHECKING”.
Automation (I prefer to say tools) can be used in lots of ways inside of testing; not only and not necessarily for checking. Tools can be used for all manner of things: data generation, configuration and setup, visualization, recording, computation of results comparable to the product being tested, and for, yes, virtual mouse-clicking and button-pushing. So the claim that “test automation is checking” is like saying “all use of programmer tools is compiling”, which is ridiculous. More on that: http://www.satisfice.com/articles/cdt-automation.pdf
A couple of things to be careful of: “An automated test will verify that a feature is working exactly the same way it was before. ” That’s not quite so. An automated check can verify that a feature has produced the same output as before, given the same input as before. That can allow us to infer that the feature is working the same way as before. However, there are lots of inputs and outputs that we don’t check; there are lots of conditions that we’re not examining; and lots of internal behaviour that we don’t observe. It’s important, I think, to remain vigilant and to review our automated checks periodically.
As you rightly point out, “The risk that a change to the software has affected a seemingly unrelated feature will always be present.” I agree, and I’ll also suggest the risk that test have not detected problems even when the product hasn’t changed.
“Scripted manual tests verify that a feature meets the expected requirements.” I worry about that too. No test can verify that a feature meets the expected requirements; as a different Dijkstra (Edsger, not our mutual friend Bas) put it “testing can prove the presence of bugs, but not their absence”. The product can appear to meet its explicit requirements, but can have all kinds of problems not noticed during the test, or problems in conditions other than those in the test.
A few other notes on “manual” and “automated” testing: http://www.developsense.com/blog/2013/02/manual-and-automated-testing/, http://www.developsense.com/blog/2017/11/the-end-of-manual-testing/; and on scripted and exploratory testing http://www.satisfice.com/blog/archives/1509
Thank you for raising the issues. I hope this helps you when someone says “TEST AUTOMATION IS NOT TESTING, IT’S CHECKING”.
Hi Louise.
As someone who has blogged about this very subject (Automation is checking, not testing, it’s interesting to read an alternative view on this.
I’d like to start with your statement ‘…the statement has the unintended consequence of implying that test automation is inferior to manual testing.’
This may be a way of interpreting the statement, however in my blog post on this, I was careful to point out that testing is important, and the means by which you do this – human using a keyboard (aka manual), or using a tool (automation) are equal partners. The problem I want to see addressed is the over reliance on Test Automation for everything. Every job posts as its number 1 priority that a tester has to be able to code, or they wont be able to do the job. This is completely false, and as an industry we have to stand up and say that a tester first and foremost has to be able to examine, question, determine what needs to be tested, before we start worrying about how we will do the testing. If the key analytical skills are ignored, you could be the best coder in the world, but you wouldn’t know what tests you need to write code for. So the unintended consequence may (to some people) seem as though we are saying that automation is inferior, but that is because it has been given too much prominence, so any shift away from promoting automation will look like that.
Secondly I do believe that automation is checking. and not testing. In order for automated code to work, you have to know what the expected result is. Each automated scenario is only going to run a series of steps and check for a known end result, reporting success or failure. The benefits are the ability to repeat these checks as many times as needed without a human tester executing the steps manually. The downside is of course the maintenance aspect (I have seen so many flaky automation packs), and the fact that the test will not deviate from the path set for it, as you mention. Testing is different, in that you are looking for known outcomes, but able to deviate and try other things to test what will happen. That feeds in to comments on exploratory testing below.
I do agree with you that ‘Manual testing cannot and must not be completely replaced by test automation’. Again, there is a misconception by senior leaders in tech that once a set of tests have been automated, all the work is done and the number of testers can magically be reduced, as can the test execution time. Something else we all must address, as the message doesn’t seem to be getting across.
Finally, I’m interested in your view that you do not see exploratory testing as testing. The best example I could think of is going in to a stationers shop to buy a new pen. You see scrap paper nearby so that you can take off the lid and scribble to see if it works, you like the colour, is the width of the ink correct for you, does the pen feel comfortable to hold. These are tests, but you wouldn’t write them first, so they are essentially exploratory tests. The point you make is that to perform testing, you need to know expected outcomes. That is true if that is the only definition of what testing is, but testing to me also covers the learning and questioning aspect. As a tester, I want to learn about the product, what it does, and with user stories/functional documents, I should have a good idea as to what should happen, but I should also be looking for oddities. Actions that result in events that were not documented, and possibly not foreseen. These are valuable as they provide feedback to the product owner/business stakeholder that there are other scenarios that need to be catered for.
It is definitely a statement that is open to interpretation (or misinterpretation, depending on peoples views). Whatever its meaning, it has definitely led to some interesting discussions.
I came up with my statement that exploratory testing is not testing because of my own personal definition of testing, something I am regretting slightly as I do not want to make the mistake of implying that exploratory testing is not important. If going by my definition, then it could be referred to as exploratory learning (although I don’t want to change what is a well used term).
The pen analogy is an interesting one, I would like to expand on it a little. When testing the pen, there will usually be some information which you can base your testing on. This would be the expected outcome. The test script for this would include checks that the pen worked, the width matched what was defined, and the colour also matched what was defined. However, there are other elements which cannot be defined and this would require exploratory testing – like is the pen comfortable to hold, how does the colour and width look when actually writing with the pen (my handwriting seems to change with different pens). This is equally important, if not more so. Of course, sometimes there is no information on the pen at all and there might just be a jar of random pens. This would be ad hoc testing.