Risks in Test Automation

“The biggest risk is not taking any risk… In a world that’s changing really quickly, the only strategy that is guaranteed to fail is not taking risks.” — Mark Zuckerberg

While this is more of a philosophical statement, one needs to look at it objectively and more pragmatically. Let’s first understand Risks and what they are. Risk is: The probability of occurrence of an unexpected outcome causing a negative impact.

If a risk is not identified and mitigated on time, then there can be catastrophic effects such as loss of revenue and perhaps even loss of life. It is therefore very important to identify, analyze and mitigate risks on time. Risks can be broadly classified into 3 categories:

Project risks – These are the risks that pose a negative impact on the result of a project. For example, unavailability of resources such as infrastructure.
Product risks – When a risk threatens the functioning of a product and increases the possibility of not solving the customer’s problem.
Business Risk – When the risk threatens the financial value of the product and impacts revenue.

What is Test Automation?

With this background information on risks, let’s now look at Test Automation. Businesses worldwide are focusing on automation with the goal of making them faster, efficient, error-free and accurate. Test automation is the automation of certain test procedures using code and/or tool.

While Test automation brings to the fore a number of benefits such as accuracy, efficiency, speed and agility, there are a number of risks that test automation is prone to. Let’s look at some of the risks in Test Automation and begin talking about the obvious ones first.

Less or zero ROI: There is a huge amount of investment made to get automation going in any project. However, the cost of setting up automation and making it work is high. The return is slow and takes time.
Constantly evolving technology and testing playing catch up: With constantly evolving technology it is seen that testers and testing need to be innovative in terms of testing solutions, architectures, strategies and tools. This is an obvious risk as it tends to slow down the progress of the project and there is a chance of the product or parts of it remaining untested. The testers and testing need to constantly evolve alongside.
Heavy maintenance – Higher costs: Test automation is heavy on maintenance. And that’s why it is important to understand and cover for the investments.
Steeper learning curve for testers: Technology is not only evolving but also getting more and more complex, which means that the learning curve for testers is getting steeper by the day. There is a need for testers to keep abreast with modern technology not just for developers but also for testers. Also, it helps testers if they learn similar to the developers so that they can collaborate better.
Inadequate test tools: With the pressing need for testing and growing demand to constantly deliver better products, more and more businesses are going the Agile methodologies of working, which means there is a need to choose effective strategies to carry out not only in sprint testing but in sprint automation. Also, the automation strategies would have to be targeted to help test across the right layer and meet the demands of end to end test automation.

A lot of these risks are handled very well by having a tool that is cost-effective and capable of mitigating such risks and one such tool that helps you address some of these obvious risks is TestProject, that helps in scaling up to businesses and flattening the learning curve for testers. It helps testers collaborate and work seamlessly in complex environments. It helps foster collaboration through features like the Share Center allowing testers to share their test case with anyone. Now that is a great feature to have 🤩

Now, let’s look at some of the not so obvious risks in Test Automation:

Limited purview of automated tests: The scope of automation is often perceived to be gigantic. There is a prevailing notion across the industry that Automation can solve all the problems and can get you across the line faster. While the latter is true, there is very little truth in the former. Automated test procedures can solve some of the problems but certainly not all of them. Automated tests are only a part of the larger set of tests.

On the other side, it is often seen that Automated Testing is used primarily for regression testing and the scope is just limited to converting pre-existing test cases to automation code so that the tool can be used to execute the tests. This is a limited view of automation. In order for automation to be used effectively in testing, one could use it for:

Generating Testing Data Automatically
Writing Test code
Test Analysis – Data Input, Data Filtering, Data Calculations
Test Report Visualization
System Monitoring
And more…

It is therefore important to understand the purpose and scope of automation.

Untested test code: While we are trying to dig deeper into the product using test automation, it is also important to test the test code itself. It is a risk that is often unseen upfront and sometimes ignored later. This results in a lot of untested test code thereby making it very difficult to decipher if the failures noticed are due to the errors in the product code or the test code.
Unrealistic expectations – 100% Coverage: When things happen to go north and you taste success with automation, you run into the risk of building up unrealistic expectations in terms of the value that automation can yeild and managers are often seen expecting 100% automation. It is rather impossible to have the testing of a product automated 100%.
Over-dependence on automation: With the success of test automation, there is a risk that almost everything, every tiny detail in the test and various test procedures are automated. This means that that for the very simple tests also there are automated test procedures which make things bulkier and unnecessary. Automate only what is required.
Inattentional Blindness – Missing the obvious: With too much focus on automation, there is a tendency to miss the obvious in the product. Testing is not just about conformance to requirements but about finding the unknown. However, if there is too much focus on automation, you may just walk past the unknown without noticing. There is every likelihood of the obvious risks coming true.

The Missing Feasibility…

While taking up any assignment, it is always advised to see if it is possible to actually take up the project. If the project can be taken up then what may be some of the challenges or what are the aspects that one needs to be aware of before getting on with it. This in simple terms is deemed as feasibility. For test automation projects, feasibility analysis could entail tasks such as evaluating what can be automated and what cannot be? From what can be automated, we need to look at what are the layers in which automation can be done? Is it possible to do end to end automation? What tool or tools can help in automation?

A poll taken among the members of The Test Chat community on telegram

Only 53% of respondents of a recent poll revealed that they do a feasibility analysis for automation, while 16% of the respondents mentioned that they do a feasibility analysis. However, the most important part of the survey was where 12% of the respondents said that their management did not care about a feasibility analysis, which is indicative of the cognitive biases in testing and is a strong case of confirmation bias and bandwagon bias. These are the situations where most risks are probable. A thorough feasibility analysis helps mitigate several risks at the outset and is highly recommended.

Test Automation for Risk Mitigation

Test automation is also a solution for a lot of risks in testing. We can consider test automation as a mitigation for the following risks:

Human Error Reduction: Tests that are often repeated are prone to human errors and that is an ideal case for automation that allows the machine to handle redundancy, thereby reducing human error.
Increased compliance and data integrity: Tests that are automated always help in meeting compliance standards and have high levels of integrity.
Focused Effort: Automation of tests lets the machines focus on the tests, thereby allowing you to explore other important areas and look for the unknown.
Increased transparency: Automated tests provided necessary transparency as the tests and the results can be easily shared and everyone in the team can be on the same page.
Greater Agility: Automation plays a big part to help meet the goals of high quality delivery at speed.

The Downside of Risk Analysis

The whole idea of testing is based on risk analysis. Every test is a risk being analyzed or a mitigation being verified. However, risk analysis does not come easy. Here a few downsides:

Time-consuming exercise: Risk analysis is definitely not a one-man job. Developers, testers, architects, product owners and stakeholders are all parties to successful risk analysis. It means that everyone in the team needs to invest their time.
Subjective Biases: With so many people involved, risks are often subjective and fall prey to biases. What may seem to be a risk to a tester may not be a risk to the developer or vice versa.
Extremely Difficult to implement: Since it requires the investment of time and effort, it is one of the hardest things to implement.

Conclusion

Risks are a part of the testing journey of a product and it is not different for test automation. It is therefore important to understand and analyze the risks involved in test automation and seek mitigation via the tests!

This blog is also available as a webinar you can watch here:

Happy Testing 😎

Risks in Test Automation – Going Beyond the Obvious

What is Test Automation?