Organizations aspire to accomplish the trifecta of speed, quality, and cost to remain competitive. Organizations are embracing trends like Agile, DevOps, test automation, continuous testing. Test Automation is the biggest enabler of agility. Test automation increases test effectiveness, efficiency, and coverage, saving time, efforts and cost. However many of us struggle to justify and measure the value of test automation. This is simply because we don’t measure it! What gets measured, gets done.
Measuring the value derived from automated testing using an ROI formula does not make sense. We should rather measure and justify investment in automation in terms of VALUE.
Test automation is sexy but can often prove to be expensive.
But is it worth it? How do you measure the value of your test automation?
Developing the right set of metrics becomes a critical step when transitioning to automation testing. Without establishing and tracking metrics, you would be left in the dark. You need to rely on well-chosen metrics to assess if your organization is getting an acceptable value by investing in test automation. There is no perfect metric, for every environment and situation one metric might be better than the other. You need to take into account several aspects of your organization. It all depends on the context and what you want to achieve. Do not use metrics for the sake of doing it. Find a measure to gauge the value of your automation efforts and their significance to the business.
DOES TEST AUTOMATION SAVE YOUR TIME?
IF YES,
MEASURE HOW MUCH TIME?
Time saved
With the popularity of Agile and DevOps, there is a need to execute regression test cases on a weekly/daily/nightly basis. This has pushed the testing teams to invest in test automation to cope with the speed of development. Investing in automation has reduced the testing time significantly, enabling the teams to release at a more accelerated rate with confidence.
Time saved is a simple metric for justifying automation efforts because “Time saved is money saved”. Time Saved is the difference between time required for Manual Execution and time required for Automated Execution. Fragility is also taken into consideration in Time Saved as maintaining and updating scripts also requires a significant amount of time.
| T.S = ME-(AE+F)
T.S = Time Saved |
ME (time required for Manual Execution) can be easily calculated by calculating the estimates given for regression test cases.
You can calculate AE (time required for Automated Execution) using the metric “Total test duration”.
Analytics by TestProject
Agile is all about shortening feedback loops. So our automation tests must run fast and provide quick feedback. You need to keep an eye on this metric and optimize your automated tests when required so that the automated tests do not become a bottleneck in agile/DevOps settings.
Fragility (F) is the measure of time spent on fixing and updating test scripts. If an automated test sometimes passes and sometimes fails or has different run times then this is a good indicator of test flake. Flaky tests do not provide any useful feedback. They not only waste time but also reduce overall confidence in test automation. Flaky tests represent poor test quality or test environment failures. So keep an eye on this metric to identify and eliminate flakiness in your tests.
Analytics by TestProject
Example
Assume that regression test cases take 15 man-hours to execute manually and the same tests are run in 5 hours using automation. However, it took 2 hours to identify, debug and fix the fragile tests.
We can calculate ‘Time saved’ as follows –
T.S = 15 – (5+2) = 8 hours
In this case, we saved 8 man-hours using test automation.
DOES TEST AUTOMATION SAVE YOUR EFFORT?
IF YES,
MEASURE HOW MUCH EFFORT?
Equivalent Manual Test Effort (EMTE)
How much effort did we save using test automation?
How much effort would it take to execute the same tests manually without automation?
‘Equivalent Manual Test Effort’ shows the benefit of running automated tests against the time it’d take to execute these tests manually.
How to calculate EMTE? (Dorothy Graham,2010)
For example:
If an automated test(1.1) takes 3 hours to be run manually then the EMTE is 3 hours.
If an automated test(1.2) takes 3.5 hours to be run manually then the EMTE is 3.5 hours
In a sprint, if we run automated test(1.1) twice and automated test(1.2) thrice, then EMTE for that sprint is –
3*2+3.5*3= 6+10.5 = 16.5 hours of EMTE.
We can say that Automation saved 16.5 hours of manual test efforts.
We can convert this into cost by multiplying the average hourly rate of a tester if needed. Assuming that the average hourly rate of a tester is 35$, you save 16.5*35 = 577.5 $ using test automation for one sprint (considering there are only 2 test cases to execute).
DOES YOUR AUTOMATION SUITE FIND NEW ISSUES
OR
REDUCE RISK?
IF YES,
THEN QUANTIFY IT!
Test Case Effectiveness (Automation)
The objective of this metric is to know the efficiency of test cases that are executed by the automated test suite. It’s one of the few metrics that help you to determine the quality of the test cases and the value the test cases are adding.
| Test Case Effectiveness = (Number of defects detected / Number of automated test cases run) x 100 |
Example – If I run 100 automated test cases and was able to find 5 defects that means my test case effectiveness is 5/100*100= 5% percentage.
Most of the times automated tests do not have a high bug detection percentage. If the same automated tests are repeated over and over again, eventually they will no longer find new bugs. The metric “Test case effectiveness” measured for a period of can avoid pesticide paradox. So it’s important to update your automation test suite frequently considering ‘Test Case Effectiveness’.
We cannot measure the effectiveness of automation only by considering the number of defects found. The biggest benefit of automated testing is the ability to reduce and mitigate risks. We should use other metrics like “automation test coverage” and “requirement coverage” along with “Test case effectiveness” to assess risk. More the coverage lesser the risk. You can pinpoint risk by looking at the areas there are a lot of new changes and low coverage.
We need to understand that automation is a long term investment. You may not always see the benefits of it instantly. I have discussed some simple metrics to determine the value you get from test automation. You should select the right metrics based on the context of your project. There is no universal way that you can deploy to measure the value of test automation. Measure the value of your test automation considering the amount of time/money/efforts saved and most importantly the ability to reduce and mitigate risks.
Image Credits
- Featured image – www.vecteezy.com/free-vector/robot
- https://icons8.com
- https://undraw.co/