Most first-time automation endeavors lack planning around maintenance cost. We typically start an automation endeavor like this:
- Decide to automate.
- Choose a tool and start automating.
- Demonstrate our automation success (hopefully).
- T-shirts for everyone!
Soon, we begin development for the next release of the product. Depending on how we decided to automate, there might be major changes required in our tool, our scripts, or both. The scripts may need to be rerecorded or rewritten. Is that effort a big deal? Maybe, maybe not; It depends on our chosen tool, our implementation, and the quantity of automation that has to be reworked.
Sadly, this chain of events causes many automation endeavors to be labeled as failures due to maintenance costs. When planning an automation endeavor, several things are often forgotten:
- Licenses and support agreements for tools.
- Script maintenance.
- Tool and framework maintenance.
- Documentation updates.
License and support agreements should be very straightforward, but often they are not; the details are in the license agreements or contracts we sign. The total cost of our licenses can increase or decrease depending on how many users we have. Some products provide perpetual licenses, meaning that even if we decide not to renew our support agreement, we can still use the products at no additional charge; Some products require the support agreement to continue their use. Neither of these support approaches is inherently bad, but one may be more preferable to our particular organizations. In any case, if we need to pay annual fees for our automation tools, we must account for those fees during our initial planning; We don’t want any surprise bills at the end of the year.
We have to remember that our automation is code. In the same way we plan for upkeep on our product code, we need to maintain our automation code, including any scripts that we create. If we change our product, we may have to change or extend our existing automation. We will also have bugs; We’ll find issues in the automated scripts which need to be debugged and fixed. Finally, taking upgrades of our automation tools may require changes to our scripts as well: Sometimes releases are not 100% backward compatible.
“But we use a tool that is code-less (or script-less)!”
So? It’s still code. Somewhere, under the hood, in that stack, there is some software. Additionally, using a non-code interface for a tool still requires us to sequence a set of commands that will perform an activity on our behalf. I submit, that’s code and therefore software, which will require some kind of maintenance. Users of this kind of tool may not have the same considerations as users of a script-based tool, but they will likely have some that are unique to their tool and implementation choices.
As much as our scripts are code, any tools or frameworks that we create for ourselves are equally so. In addition to the maintenance items related to scripts, there are other items that can cause maintenance to our tools or frameworks, including operating system or programming language upgrades; New features, new fixes, and new issues can necessitate changes in our code.
As the automation we create gets used, the documentation we write for it must evolve. We will add features and fix issues; These activities often result in needing to update our automation documentation. We should also consider updating the documentation to reflect how our tools and scripts are being used; Things that one team is doing may benefit other teams. FAQs and troubleshooting are the most likely candidates for updates from this emergent learning.
As usual, not all of us will have the same maintenance needs, however, neglecting to consider and account for them can wreck our budgeting or planning processes. We need to think about and attempt to account for as many of these items as possible. If we don’t, we run the risk of our automation being labeled as a failure, when in truth it was our planning that was at fault.
You are welcome to share your thoughts on test automation maintenance in the comments below! 🙂