How important is test data while performing automated tests? How can you make sure you’re always using the right strategy while incorporating test data within your automated tests?
These questions were always very important to me, but since I’ve never come across problems or challenges in these areas – I’ve never known the impact of not having a test data strategy of how we are going to use, update and maintain test data used in our tests.
I was working on a product that was a multi-tenancy product, which means the infrastructure was shared with multiple tenants. This was being tested by an external consultancy who used to work remotely. I joined the team as a first tester in-house. I was happy to see that there was use of an end to end automated test suite. The consultants gave me a handover and took me through the tests before leaving the project.
During the whole 2 weeks of handover, I tried to understand the risks and challenges while testing this product as the domain was completely new to me. The tests were scheduled to run once as a nightly build and once at 1 pm afternoon. So the testers would look at the test results in the morning and afternoon to check if any test failed. During the first week, I realized that the testers were spending all their time in fixing the failed tests, and there were no new tests being added as there were no new features being developed (at least until the new team was settled in 🤩). Not to mention that if it was too hard to fix the failed tests, they would mark them as ignored! 😲
Now, it was my turn to start working on it. I wanted to change how we used to test this product. So as my first step, I wanted to run these end to end tests on a QA environment and UAT (User Acceptance Testing) environment for each client. When I say ‘client’, I mean we had different clients using this product in various ways and we had QA, UAT and Production environments for each of these clients.
Since every client was using the system differently, I wanted to add those tests to the test suite and run on all of these environments. So the first step I took was to check the end to end test’s code and see where and what changes I have to make to run this test on one client QA and UAT environment. Then I realized that the URL and user credentials were hardcoded in the config file. In fact, ALL the test data used in these tests was hardcoded! This posed a big challenge since every client we had obviously had different customers and different customer details. If for example I’m running a test for:
- Go to search
- Search by company name and account number
- Verify the search results
If I were to use the same test for these different clients by only changing the URL and user credentials, the tests would fail, as it was not just that those details that were different, but also the account code format for each client’s customer was different.
The more I looked into the tests, the more challenges I found which were all related to test data. Since these tests were created by an external consultancy who had the responsibility of only their QA environment – They built their tests according to their needs which worked perfectly for them.
After analyzing all of this, I thought I could refactor the code by picking one test at a time and removing the test data which was hardcoded and add it to an excel file and use that file to pass the test data. It was definitely time-consuming, but we as a team could not think of any other better option. We either could refactor the code to make this change or we had to write the framework and tests from scratch. It’s so important to develop a test automation framework which is scalable, adaptable and maintainable!
While searching for resources to find out a better solution to my problem, I came across this blog post that helped me understand more about data-driven test automation and the necessity of test data I always knew was of great importance 💪
Go ahead and share your use case stories in the comments below – How do you overcome your challenges? 🤔