It’s a fact that everyone wants to do automation, a lot of companies try, a lot of companies fail, and very few succeed. A successful company is not one that never fails, rather one that learns from its failures. Here are some of the failures that I’ve witnessed over the past few years, and the lessons I’ve learned from them that led me to write these recipes for test automation success.
Recipes for Test Automation Success
“Let’s automate everything, and stop doing manual testing entirely!”
1. If you automate all your test cases (Data-sources/Units/APIs/CLI/GUI) then you’ll ensure that it’s much less likely for regression issues to happen, but what about new issues and corner cases? You will always need exploratory, experience-based, and fault attack tests.
Plan to automate as much as you can as long as there will be an acceptable added-value to effort ratio. And know that you will always need expert functional testers to think of all the new and innovative ways to find bugs! Sure, the most noticed added value for automation is reducing test execution time, but I believe that the most important one is saving time for functional testers to innovate new approaches to their testing and increase their coverage.
“Let’s put a solid plan for when test automation is 100% done, and achieving 100% coverage.”
2. In modern-day applications, it’s difficult to say that something is *Done*. Is Facebook done? Is Gmail done? Most new applications will always undergo UI/UX/Functional changes to stay relevant. New features will be added, corner cases that show up after the billionth user will be discovered. And so, testing will never be done, and hence test automation will never be done. You can plan to complete a static set of regression tests that you have. You can monitor and increase automation efficiency and velocity. But it’s unrealistic to expect it to be 100% done within any time frame. The same goes for coverage. You can plan for 100% coverage only when you have a permanent code freeze.
Plan for the highest coverage you can reach, but be flexible to the ever-changing demands of your software. And think of automation as a long term engagement that will grow with your product and will only stop when your product stops developing.
“Let’s automate all the end to end tests, this will certainly ensure that the product has much less bugs.”
3. Though the assumption is true, but the devil is always in the details. Doing e2e test automation over a GUI layer is cumbersome, because there will always be UI/UX/Functional changes that will require refactoring (even if you’re following all the best practices). So, the rule of thumb is “Early detection is better”.
Implement automated static code analysis, automated unit tests for your code and database layers, and automated API tests for your integration layers, before you consider automating the end to end tests through the GUI. And remember that no software is ever “Bug Free”.
“If we buy the best commercial tool, all our problems will be solved!”
“If we use the best free frameworks, all our problems will be solved!”
4. Buying a commercial tool without a solid ROI analysis is like buying a Ferrari when you live on an island and don’t know how to drive; utterly useless. It’s not a matter of having money that you can throw at a tool to make automation go away, because that seldom works. And going for open source frameworks without investing in your resources to learn is like giving a Swiss army knife to a 3-year old child; extremely dangerous and will probably cause a lot of damage. It’s not a matter of cost reduction so you must use a free framework/tool, because you may waste much more money on the wrong free tool than what you would have paid for the right commercial one.
Thoroughly understanding your product, what the different test automation solutions will offer you, and what they will require in return is essential before you take any decisions. Always do your ROI analysis!
“Let’s just outsource test automation to another company and get rid of all the hassle.”
5. Outsourcing testing without a mature SDLC and communication/reporting models seldom works. You’ll just be introducing more hassle; communication hassle, reporting hassle, delayed deliveries, non-maintainable automated scripts, and a ton of other issues…
Work on maturing your processes, building quality and automation into your product, streamlining the delivery, and minimizing waste. And then if a thorough ROI analysis tells you to outsource test automation, you should definitely go for it!
“Let’s just hire a few part-time resources to do test automation, and then we can save costs by letting them go once they’re done.”
6. Hiring part-time employees to build test automation into a large scale product that’s expected to last will probably not pay off as you think it may. Quality needs to be baked into the product, and you cannot do that without the ownership of full-time employees who invest their lives in your product. Not just testers, because quality should start at the bottom of the SDLC and work its way up to the customer. Plus, test automation will probably never be *done*.
Bake quality and test automation into the product, ensure that all your team members adopt a quality mindset, maintain good control over productivity, perform data-analysis to identify slots in time where you may need external help, hire part-time engineers to fill the temporary gap, hire them permanently if the gap turns into a long-term engagement, and don’t think of people as resources, think of them as fixed assets that you need to invest in.
In the end, I believe that in order to achieve test automation success, then:
- Doing the right thing (effectiveness) and doing it the right way (efficiency) both depend greatly on collecting and analyzing data.
- Establishing a solid process and performing a thorough ROI analysis are essential before taking any decision.
- In our fast-paced world you will be tempted to do workarounds, take short-cuts or jump to decisions. You should fight that temptation, and instead use it as a drive to complete your thorough analysis faster.
- Always own, document and learn from your mistakes, that’s the only way to keep improving.
- Always invest in people, they are your biggest assets.
I hope your enjoyed this recipe to test automation success! Please share in the comments below your recipes for success 😎