logo logo

Automation Test Strategy: What Should We Automate

Designing and Implementing an Automation Test Strategy pt 3 Determine What to Automate

Designing and implementing an automation test strategy can be overwhelming. There are multiple points to consider and if you’ve ever researched the topic online, you know there are many different points of view to think about. How you develop and implement an automation strategy should be unique to your organization’s needs and abilities. In this multi-part blog series, I will provide some in-depth thoughts about how I’ve designed and implemented test strategies, things I’ve learned, and other industry gurus’ thoughts on the topic. At the end of these installments, you’ll feel comfortable enough to design your own automation testing strategy and determine what to automate! 👨‍💻

After reading my previous posts you should now have a solid understanding of what your automation goals are and what you should look for in a test automation tool. An extremely important part of determining your automation strategy is answering the age-old question around automation: What should we automate? This post will explore why you shouldn’t automate everything and some questions to help you determine what you should automate.

Table of Contents –Automation Test Strategy: What Should We Automate

  1. Chapter 1 – Identify Your Goals and Ownership
  2. Chapter 2What Should I look For In a Tool?
  3. You’re here → Chapter 3 – What Should We Automate

Automate Everything! 

Regardless if you’re beginning automation for a new project or an existing project, you should never consider automating everything, just for the sake of automating. It’s tempting, I know. But let’s look at why that isn’t a good idea:

  • Scripting takes time; time is money– Every script you create takes someone to do the work. If you’re focused on automating everything you aren’t getting a good return on your investment.
  • Maintenance is required– It doesn’t matter how well your automation is written, it will always require maintenance. In addition to script maintenance, you’ll have maintenance of things like tooling, job executions, libraries, etc.. If you’re just automating everything, you will certainly find yourself drowning in maintenance and unable to take on new features.
  • Flaky scripts– Tying back to maintenance, if you aren’t able to maintain your scripts they will become flaky. If your automation is failing due to a lack of maintenance, people will lose faith in the results over time.
  • Increase in execution times– Every script you write will need to be executed to add any value. If you have a large set of scripts you’re executing, your execution times will naturally increase. While you can do things like running your tests in parallel, you’ll still see an increase in execution time as you add more and more scripts.
  • Validating failures– If you run a ton of unnecessary tests every time, you’re increasing the need to validate any failures you may have.

It’s worth mentioning some specific things that might make sense to automate from a scenario standpoint, but don’t always work well when implemented or can’t be automated:

  • Colors
  • Captcha
  • Accessibility
  • Documentation
  • Really long flows (consider deep linking to the part you need or testing on the backend)
  • Following user flows to the tee (consider automating the end to end flow once, and then breaking out other scenarios to just test smaller pieces at the integration layer)

Automate the Valuable Things!

We’ve talked through why you shouldn’t automate everything, but that really doesn’t help you understand what you should automate. When thinking about a specific scenario, here are some questions you can ask to determine if its an ideal candidate for automation 🔍

  • Is this a scenario that customers experience often? Think through the main user flows for your customers. For example, making sure that customers can register an account and buy things would be really important if you’re an online store.
  • If this scenario results in a defect, would it hold up a release? How quickly would a defect for the scenario be fixed? Would it block you from deploying your code, or even worse, bring down your testing or production environments?
  • Is the end result of this scenario only obtained through this path? Perhaps there is more than one way a customer can get through the flow. If this one is broken, would they still be able to accomplish the important tasks?
  • What’s the complexity to script this scenario? How quickly can you script it? Are there new tools or technologies that would be required to complete it?
  • How stable is the area this scenario resides in? Do you constantly have defects pop up in this area?
  • Is this a test that would run consistently? It might not make sense to spend time automating something that only runs once a year.

Unit Test vs. Contract Test vs. UI/End to End Test

A part of analyzing what and how to automate is determining what layer to automate your scenario at. A lot of testing gurus have published their own take on the Automation Pyramid, but here is the original pyramid for reference:

Test Automation Pyramid-determine what to automate

Essentially, the pyramid visualizes your total automation suite and defines how many tests should reside at each layer 🔺 There is some guidance on what percentage should fall in each bucket: 80% Unit tests, 15% Integration tests, and 5% E2E tests. I would caution against the automatic structure of this guidance as every organization is unique. Consider the application you’re testing and the skills of your automation team as well 🧐

Unit Test

Unit tests make up the largest layer of the pyramid. It’s a simple test focusing on a sole unit (a piece of isolated code). This test shouldn’t rely on any external systems such as a database, configurations, or even other tests.

  • Pro1: Quick to write
  • Pro2: Gives developer immediate feedback as they execute quickly
  • Pro3: Give specific details to the developer on the issue since they are contained to single units
  • Con1: Since they are specific to a unit, bugs in the integration and UI layers are missed

Integration Test

The integration testing layer is meant to increase the scope of testing and includes service or API level tests. This builds on the unit layer by ensuring units of code can integrate together without issues. This layer should also consider any contract testing you might have. For example, any 3rd party integrations with your application.

  • Pro1: Add an additional layer of testing past just unit tests
  • Pro2: More stable than UI/E2E tests; fewer changes at this layer
  • Pro3: Quicker execution; much quicker than UI tests
  • Con1: While integration tests increase the scope of testing, they do not focus on the user experience layer (UI)

End 2 End (E2E) / UI Test

The UI testing layer is important, but shouldn’t make up the majority of your testing. This layer focuses on the user experience by ensuring the UI flows are functioning as expected 💪

  • Pro1: Ensures the customer experience is intact
  • Pro2: Record and execute tools can make this layer easier to automate
  • Con1: While easier to automate, they do take much more time to design and automate
  • Con2: Flaker tests as the UI typically changes frequently; more false-negative test results
  • Con3: Execution times are longer

Test Automation Pyramid-determine what to automate


Determining which scenario to automate and what layer to automate it takes practice and diligence to ensure you’re automating to add value. Be mindful about how much you’re automating at the UI level. It’s easy to push everything, outside of unit testing, to that layer. In the end, you’ll spend more time on maintenance and executing than you will scripting new features.

Unit tests are important and can be a good way to involve developers in the automation process. Unit tests are also a good way to get quick feedback on your code commits 👩‍💻

And finally, do not automate everything. Make sure you’re asking yourself if the scenario should be automated. If you’re just starting out with automation on an existing project, a good place to start is automating your regression suite. This was the third and final part of my Automation Test Strategy series 📖 Happy testing!

Share in the comments what are your automation goals, and if you have other tips about what to look for in a test automation tool 💡


About the author

Johanna South

Johanna South has been actively testing or evangelizing software testing concepts for a little over a decade. She has worked in various roles throughout the Software Development Lifecycle, including manual and automation testing, systems analyst, and leading full-stack software development teams. Her broad understanding of delivering quality software gives her a unique perspective in which to pull ideas.  Johanna is currently a Sr. Director of QA for TwinSpires.com.  Also, she leads her organization’s Testing Center of Excellence and is responsible for ensuring application performance across multiple platforms. Her passion for both encouraging everyone to understand the value of quality, and learning new techniques to be a successful tester, have led her to author a blog: theQAconnection.wordpress.com.  You can find Johanna on LinkedIn and Twitter (@theQAConnection).

Leave a Reply