Test Automation Is Still Testing, But Don’t Go At It Alone

There is a statement that has been discussed throughout the testing community that stirs a great deal of controversy:  “Test Automation is not Testing, It’s Checking!”.

It is a really powerful statement when describing the benefits of manual testing, specifically exploratory testing. However, the terminology can provide alternative interpretations. The words checking and testing are essential when writing test plans. Checks can be applied to a single validation step within a test case. Meanwhile, a test will consist of several steps including checking steps. As a result, the statement has the unintended consequence of implying that test automation is inferior to manual testing.

The subject of test automation and the way it is used has been debated a great deal and usually yields a variety of opinions. Some are strongly opposed to automated testing and see it as a risky and costly software testing strategy. Others see it as a necessity when testing applications that are forever evolving and increasing in complexity.

Out of all the varying ideas surrounding test automation, there is one thing that almost everyone appears to agree on: Manual testing cannot and must not be completely replaced by test automation. It is unlikely that 100% test automation will ever be possible, at least it won’t be in our lifetime. Even if it were possible, software applications are designed to be used by humans. This makes 100% automation ill-advised as computers do not have the correct mindset required to recognize potential usability issues. If test automation is to be included in a test plan, then it must be backed up with manual testing – scripted and exploratory.

Automated Testing

An automated test will verify that a feature is working exactly the same way it was before. It is really useful when testing areas of the software that have not been changed. If a change has been made that doesn’t directly affect a certain feature, it is reasonable to expect that feature to still work the same way as it did before.

An automated test will run precisely the way it’s been designed to. However, this can be both a gift and a curse. The inability for the test to deviate from the plan means that it is unable to cope when things go wrong unexpectedly. I am going to quote Bas Dijkstra in an earlier blog post because he said it so brilliantly: “[A test with a failed step] simply says ‘f*ck you’ and exits with a failure or exception, leaving you with no feedback at all about the behaviour to be verified in steps 11 through 50”.

It is unlikely that a test used to verify an unchanged feature will fail. Unchanged features can be confirmed as still working and not get affected by changes made in other areas of the software. Automated tests should be used to verify that unchanged features are still working as expected. This allows dedicating more time to manually testing new or edited functionality. 

Scripted Manual Testing

Scripted manual tests verify that a feature meets the expected requirements. If a change has been made to a feature, an automated test is likely to fail because of that change. With manual tests, there is scope to implement a workaround when something unexpected happens.

An option does exist for scripted manual tests to cover unchanged features as well. Creating manual test cases is a lot quicker and less costly than developing automated tests. However, tight deadlines usually mean that there is no time to run these tests manually. Thus, these unchanged features do not get tested. The risk that a change to the software has affected a seemingly unrelated feature will always be present.

Exploratory Testing

Exploratory testing brings a completely different benefit. Both automated and manual scripted tests cover features that the tester already knows about. Alternative workflows and features are explored through exploratory testing. New defects are often found. These are defects which were never thought of when the test scripts were written. However, just like test automation, this must never be the sole method for a testing strategy. Without a script or even a record of what has been done, it is unclear what has been tested. Often, there isn’t even any evidence that testing actually took place.

Returning to the original statement, both forms of scripted testing (manual and automated) can be easily defined as testing. But I do not see exploratory testing as testing. Testing and checking steps involve analyzing the quality of the software against known criteria. Scripts will contain expected outcomes which are compared against actual outcomes. Exploratory tests will have no such thing. Exploratory testing is exploring the unknown, going off the beaten track and discovering those new and unidentified bugs. These are unlikely to be found with scripted testing. They will also better identify usability issues which automated testing will never be able to cover.

Summary

Running automated tests that cover unchanged areas of the software, frees up time and resource for running manual tests that cover new or changed features in the software. This is not essential but it can drastically help to improve the overall test coverage. A variety of scripted manual tests and unscripted exploratory tests should be used to test new and edited features. This demonstrates that all features have been tested. It also allows for additional exploration of unknown areas that will find any unexpected defects. All 3 methods bring a great deal of value to a testing project and are instrumental in ensuring the quality of any developed piece of software.

Do you agree or do you believe that there is a method that is superior to another? Let us know your thoughts in the comments below  ➡