logo logo

5 Reasons Why Test Automation Lets Us Down

5 reasons why Test Automation lets us down

Test Automation – The two magic words which have revolutionized the way we Test. When machines started to execute the tests in a fraction of the time, it seemed like the perfect solution. And before we knew it, more and more heads turned. More and more teams started adopting automation for its perceived benefits. However, is it really what it seems like or is there more than what meets the eye? 👀

Let’s first begin by looking at what is Test Automation.

Test Automation

Test Automation is the automation of certain aspects of testing using code and/or a tool. 

In essence Test Automation is about the automation of any part of the test effort that can be made easy by the use of tools and/or code right from writing and execution of test procedures, generation of test data, collation of test data, analysis of test data, visualization and presentation of test results. It is very shallow to keep the idea of test automation to be restricted to regression testing.

💪 Benefits of Test Automation

Having understood that, let’s look at the benefits that Test Automation can bring:

Reduces redundancy

Test Automation can help in reducing redundancy by letting the machine take care of all the monotonous, mundane, repetitive tasks. This makes life easy for testers to let them focus their testing effort on the other features of the product/application. This allows room for deeper and wider Exploratory Testing. There is also an increased chance of identifying risks and discover unknowns early.

Faster time to market

Automation can hasten the testing effort by helping speeding test execution. What would take hours otherwise would take only a fraction of the time when automated. This means faster feedback, faster turnaround, and consequently a faster time to market.

Greater accuracy

When tasks and procedures are repeated over and over, there are chances that a step or a part of the step goes amiss. Or in cases dealing with millions of data records, there are possibilities of inaccuracies. With automation, this is largely reduced.

Less prone to errors

With every activity involving human intervention, there is always a probability of errors. When the same activities are automated, the chance of those human errors reduces significantly. Machines are good at doing the same things without errors.

Greater Agility

With all the above benefits, automation becomes a handy tool to provide us greater agility. With more and more projects choosing the Agile and/or DevOps way of working, Test Automation is the need of the hour. When there is a need to respond to changes faster or there is a demand for early and continuous testing, it is Test Automation that saves the day.

Now with so many benefits, there is still a lot of qualms about automation not delivering the results as expected. One of the biggest discussions that happen around test Automation comes from ROI or Return On Investment. There is often a question mark on whether the Automation effort would yield the right ROI. There are several automation projects which fail.

❌ Why Test Automation Fails?

Let’s look at the 5 big reasons for the failure of Test Automation projects.

1) Unrealistic Expectations – 100% coverage myth

The biggest point of failure in automation comes from unrealistic expectations around the concept of automation. First of all is the myth of 100% Test Coverage being achieved by automating everything. Some companies have achieved a great deal of success with test automation and have deployed 100% automated DevOps pipelines that help their product reach the customer faster.

While this may be true and the product may reach the customer faster, it does not mean that there are no defects or all the customer’s problems have been solved. It is impossible to achieve 100% test coverage. However, we see quite a few cases of people trying to automate everything.

2) The 80-20 Problem – No or inaccurate feasibility analysis

For many projects, there is an investment made on the basis of the perceived benefits of test automation. People are often ignorant about the hidden test automation costs which constitute 80% of costs. Costs of Maintenance, Training, Test Data, Test Environments, etc are often not considered. This leads to an overall increase in the overhead. Another aspect that is not given due importance is Feasibility Analysis. A survey done recently on The Test Chat community of testers gave astounding results:

Feasibility Analysis Survey
Feasibility Analysis Survey done among the members of The Test Chat Telegram group.

43% of the respondents did not do or were not aware of what feasibility analysis is or did it only sometimes. What was even more surprising was that 12% of the respondents said they did not do a feasibility analysis because their management did not care about any. There is very little clarity on how to brace oneself for test automation in the right layer and automation in the same sprint. That leads to a big question mark on end to end automation.

3) Lack of collaboration

One of the inherent risks in Test Automation is the fact that individuals or teams working on it are treated differently, a class apart. This creates silos in the team and leads to not only automation failures but also social disconnects within the team.

4) Not knowing when and how to use Test Automation – Cowboy Coding

One of the greatest reasons for failure is that a lot of teams and individuals do not know what is test automation and or how to use Automation. A lot of times Automation is targeted only at regression tests and the rest of the areas of automation such as test data creation or test report preparation are not considered. Not knowing what to automate in which layer is a common problem. If as a result of automation there is a defect, fixes are applied on the go. The goal seems to be defect fixing rather than defect prevention.

5) Stumble in the first steps

Very often testers begin their testing journey with UI automation. UI is one of the easiest and most favored cases for automation. However, when they encounter cases such as elements locators not being found and they see a lot of errors on their screen, that tends to put them off. They even go to the extent of getting discouraged by Test Automation. These are definitely not good signs in a project with young and upcoming testers.

✅ Possible Ways to Avoid Test Automation Failures

Having seen the most common reasons why automation may fail and be counterproductive, let us now look at ways in which we can prevent these failures:

1) Be Self Aware – 100% coverage is not possible and not everything can be automated

It is important to be aware of the benefits of automation and its implementation and implications. If business users and/or stakeholders need help in understanding these benefits, then the testers should help them understand this.

2) Do a feasibility analysis and understand what can actually be automated

It is important to understand what can be automated and what cannot be. It needs to be done right up early before starting the effort on testing. One must understand the hidden costs of automation before making the investment and not after the project is underway. Do a feasibility study at the beginning of the project to understand how feasible automation is and what are the risks involved. There are risks in Test Automation, however, a feasibility study done early can help you mitigate some of those risks.

3) Collaborate! Collaborate! Collaborate!

Collaboration is the key to success in any project. It becomes even more vital to collaborate for automation projects. Apart from using the standard collaboration tools such as Microsoft Teams or Slack, Testers can also use the features such as Share Center available in TestProject that allows us to share Codeless Test Automation with anyone. How cool is that! ✨

4) Understand what aspects of products can and should be automated. Get the basics right.

Stick to the basics of automation. If there is anything where you see the opportunity to save on human effort and you think automation will add value, then automate. Collaborate with fellow testers, developers, architects, business users, stakeholders, and customers to understand the intricacies and implications. Identify the right layers for automation. Identify the right features for automation. Identify the right data for automation. Again tools such as TestProject come with extremely powerful data generate addons. You also have the flexibility to create and add your own addons too!

5) Educate self and others – Learn about the latest tools and technology

Technology is changing fast. The testers need to keep up the pace by being up to date with the technology. Understanding the needs of the customer and complementing that with the right set of tools is extremely vital. TestProject’s AI-powered self-healing smart test recorder is a powerful tool to help new testers get on board with their testing. This is great support for new testers. Testers need to get educated about these new tools and features and help others do the same.

Food for thought… 🍉

Seek value and not ROI. You’ll then see test automation giving you more benefits than what you perceived!

About the author

Brijesh Deb

Brijesh Deb is a  Test Consultant with Beround and has over 2 decades of experience in leading, managing and coaching teams on Software Testing and embracing the Agile mindset. 

Being a hands-on tester helps him stay close to technology and understand the intricacies. He is involved in testing of products from lightweight mobile apps to large scale industrial IoT solutions. He works with some of the top most high tech companies providing everyday solutions in sectors such as Automotive, Semiconductor, Material science, etc.

Brijesh is an Agile transformation agent with the objective of helping organizations in achieving customer excellence with a focus on delivering maximum value to customers, stakeholders and teams with emphasis on cultural change driven by collaboration at all levels.

He lives in the Netherlands with his wife and 2 years old son. He loves singing rhymes with his son. He’s a wordsmith and writes on LinkedIn. He has a passion for Anthropology and loves to study human behavior. 

Leave a Reply