When automated testing retrieves inconsistent results for no apparent reason, then we know we are dealing with false results in test automation. In case of a failed test, further analysis will be required causing a great waste of time working on a solution. In this article, I will share my struggles with test automaton reliability and the magic move which restored our trust in test automation results once and for all.
The project I am currently working on, has its own internal test automation framework which has been developed by a test automation team and few developers who contributed to the automated testing aspect. Our product is an enterprise data protection solution designed for a multi-platform usage. At some point, our team was only seeing red testing reports, which weren’t showing much either to developers or managers. In that period, I have been told many times that the team had entirely lost confidence in the testing results.
Over time, we came to the conclusion that parallel releases and shorter release intervals required adoption of continuous integration and continuous deployment, which managed to bring back test automation in focus and increased its reliability. The magic move was End 2 End automated testing, integrated into faster software development cycles which enabled a continuous testing process.
Our test suite has around 850 test cases which are running on a daily basis and testing different modules in our multi-platform environment. Test execution is completely automated and triggered as soon as the builds are available at the repository. Failed test cases are rerun and reports are generated by Jenkins (read more about Continuous Integration with Jenkins CI Server). When analysis is complete, the defects are reported according to the severity level or predefined scripts.
Sounds perfect? Well, it definitely didn’t happen overnight. There were prerequisites steps we needed to take in order to increase our test automation reliability and avoid false results in test automation projects:
1. Use a Dedicated Test Environment
- You can use virtual machines created by Vagrant or test against physical hosts.
- An application should be tested in the environment that can be re-created on demand. By running tests against the environment that is in the identical state each and every time before execution, you are getting closer to making potential issues reproducible.
- Databases need to be empty and accessible for your test scripts to create and modify data.
- Ensure that the test environment is monitored and collecting debug files option is enabled.
2. Reduce Daily Test Execution Time
- Configure parallel execution of tests in separate threads.
- Depending on your solution and business, define your own “Tier 1 applications and OS” model. This model enables you to identify the applications and OS that have the highest relevance for testing, since they represent significant importance to the business.
- Make sure to not to end up with thousands of irrelevant tests – in every release, perform regular exercise to review participating tests, to identify obsolete ones and to approve new tests.
3. Team Collaboration
- You have to be close to the new code and features in order to provide information to stakeholders.
- Collaborate with developers, business analysts and product owners to reduce potential failures.
-Hopefully our experience can make a difference in your test automation project! I invite you to share your personal stories, how did you manage to overcome false results in test automation?