logo logo

10 Quality Metrics For Test Automation

main post image

Your organization has invested time and money into automating a portion of its testing. You’ve built your framework, scripted your tests, and now they’re running and providing you results. Great work! You could simply leave them running and hope they’re doing what you intended. However, you’ve already put so much time into getting where you are, and will indeed have to spend time maintaining them well into the future. Instead of hoping they’re successfully giving you the right results, let’s talk about some quality metrics you could put into place to monitor and ensure your test automation is delivering value!

Why Measure?

Before deciding on what to measure, it is important to consider why you want to measure automation quality and effectiveness, to begin with. Each organization is unique in its quest to understand automation quality and effectiveness, however, here are some common reasons you might want to consider:

  • Automation Effectiveness: You’ve invested quite a bit of resources into getting an automation suite built. While the goal for automating your regression suite may have been unique to your organization, a fairly common objective is to deliver code to the customers quickly without putting quality at risk. KPIs can be used to track and trend effectiveness over time and help you meet that goal.
  • Speed to Market: You wanted to speed your release timelines up and measuring that speed to market will help ensure you’re trending towards that goal.
  • You want to create a culture of accountability: To hold someone accountable, you must first communicate their success measurements. Ensuring you have a standard set of KPIs, communicated consistently, will allow you to hold people accountable, and to do so fairly.
  • Continuous Improvement: Getting better over time should be a goal for any organization. Measuring trends will help you adjust and overcome challenges as you improve your automation scripting and processes.

Pitfalls of KPIs

You’ve outlined your organizational goals for measuring quality and effectiveness.  Those should help you stay on track when selecting which KPIs to measure.  However, there are still some pitfalls to avoid when putting together you’re KPI suite:

  • Don’t forget to communicate:  Whatever metrics you’re choosing to track, make sure they are well communicated.  Especially if they will be used in any performance measurement initiatives.  Ensuring your team understands what you’re measuring and WHY will help drive trust.
  • Don’t use metrics that only measure productivity:  Avoid using metrics that solely provide feedback on productivity.
  • Don’t forget to make your metrics well-rounded:  It’s important that you ensure your metrics span the entire SDLC.  Focusing on only specific areas to measure will not provide you a well-rounded view of efficiencies or problems.

What Could You Measure?

Automatable Test Cases

What Does It Tell Me?

This metric can help you understand where your teams are prioritizing automation or what areas might still require manual validations. Breaking this metric up to cover specific components of your application can provide even greater value.

How Do I Calculate It?

% Automatable = (# of automatable tests / # of total tests) * 100

Automation Script Effectiveness

What Does It Tell Me?

This metric provides clarity on how your defects are being found. If you are investing large dollars in automation, but your automation scripts aren’t finding defects, you may want to dig in and understand the effectiveness of those scripts. Also, if you use different testing environments, like integration and staging, you may expect to see lower effectiveness in your integration environment because scripting hasn’t been completed for new features. Likewise, in your staging environment, you would expect to see automation finding the majority of the defects as you’re typically focused on regression testing.

How Do I Calculate It?

Automation Script Effectiveness = (# of defects found by automation / # of defects opened) * 100

Automation Pass Rate

What Does It Tell Me?

This metric will give you a standard percentage of how many automation tests passed. It is helpful in understanding the stability of your automation suite as well as its effectiveness. Having a low pass rate requires more time spent validating failures. If your failures turn out to be false failures, that’s an early indicator that your automation suite isn’t reliable. In addition, if you see this number dip after an automation execution, this might be a quick indicator that your release contains a higher than normal number of defects.

How Do I Calculate It?

Pass Rate % = (# of cases that passed / # of test cases executed) * 100

Automation Execution Time

What Does It Tell Me?

The metric will provide a value of how long it takes your automation suite to run from beginning to end. A long-running automation suite isn’t valuable when it comes to providing feedback to developers or delivering code to production quickly.

How Do I Calculate It?

Execution Time = End Time – Start Time

Automation Test Coverage

What Does It Tell Me?

Measuring test coverage will help a team understand how much coverage their automation suite is providing vs how much manual testing is being done. You can track this on a component basis and then summarize the data to visualize it across an entire regression suite.

How Do I Calculate It?

Automation Test Coverage % = (# of automation tests / # of total tests) * 100

Automation Stability 

What Does It Tell Me?

This metric will help you trend how well your automation performs over time. It’s not uncommon for your scripts to fail, especially flaky UI scripts if something in the system changes and automation isn’t updated prior to execution. However, if over time, your tests are continuously failing, that is a good indicator that your tests may not be stable.

How Do I Calculate It?

Automation Stability % = per test (# of failures / # of executions) * 100

Build Stability

What Does It Tell Me?

If you’ve taken the step of getting your automation into a CI/CD pipeline, you can use this metric to understand the level of code quality coming from development. If your builds are constantly breaking, there may be an opportunity for better unit testing, for example.

How Do I Calculate It?

Build Stability % = (# of build failures / # of builds) * 100

In-Sprint Automation

What Does It Tell Me?

The metric will help you understand how close you are to in-sprint automation. To be most valuable, your team should strive to automate new work in the iteration the work is completed in, and not in a subsequent sprint.

How Do I Calculate It?

In-Sprint Automation % = (# of scripts created in-sprint / # of scripts created post-sprint) * 100

Automation Progress

What Does It Tell Me?

Regardless if you’re measuring this against the goal of automating a regression suite, or a new feature, this metric will assist you in trending how you’re progressing towards that goal over time.

How Do I Calculate It?

Automation Progress % = (# of automated tests / # of tests that are automatable) * 100

Automation Pyramid

What Does It Tell Me?

While there are several theories around how the automation pyramid should look, the outcome is a general understanding that a certain amount of tests should live at the UI/E2E, Contract and Unit test levels. Having metrics to continuously monitor that your test automation coverage is following your pyramid strategy is important.

How Do I Calculate It?

Automation Pyramid = (# of tests at each level / # of total tests) * 100

Conclusion

There are a lot of options for automation metrics and as you read through each one, keep in mind why you’re wanting to implement testing KPIs in the first place. Once you identify what KPIs help meet your goals, make sure you’re communicating appropriately so people understand what KPIs may be used as a success measurement and why. Last, but certainly not least, use these consistently so you develop a trend to help identify anomalies over time. When implemented correctly, automation KPIs can help drive automation effectiveness, speed of delivery, and assist in delivering quality features to your customers.

Avatar

About the author

Johanna South

Johanna South has been actively testing or evangelizing software testing concepts for a little over a decade. She has worked in various roles throughout the Software Development Lifecycle, including manual and automation testing, systems analyst, and leading full-stack software development teams. Her broad understanding of delivering quality software gives her a unique perspective in which to pull ideas.  Johanna is currently a Sr. Director of QA for TwinSpires.com.  Also, she leads her organization’s Testing Center of Excellence and is responsible for ensuring application performance across multiple platforms. Her passion for both encouraging everyone to understand the value of quality, and learning new techniques to be a successful tester, have led her to author a blog: theQAconnection.wordpress.com.  You can find Johanna on LinkedIn and Twitter (@theQAConnection).

Join TestProject Community

Get full access to the world's first cloud-based, open source friendly testing community. Enjoy TestProject's end-to-end test automation Platform, Forum, Blog and Docs - All for FREE.

Join Us Now  

Leave a Reply

popup image

Enter a whole new world of test automation

Join 100,000 testing & dev teams taking their web & mobile testing to the next level, using #1 FREE test automation platform, designed to help anyone deliver quality at speed, with built-in reports, automated CI flows and a top-notch expert team supporting you at all times.
Get Started
FacebookLinkedInTwitterEmail