When we make the decision to include test automation in a software development project, or when we decide that we are going to start developing a product with test automation since the beginning, most professionals start by installing an IDE and then start automating tests.
However, in order for test automation to be successful, it is necessary that the testing strategy will be well defined, thought-out, and planned before starting 🔍
In this article, you will be presented with 20 questions that you and your team must answer before you start automating tests. In the end, a cheat sheet that contains all the questions discussed in this article will be available, which can be used as a quick guide whenever you need to define a strategy in a new project.
Why should you not start coding?
When you start by automating tests, you miss a very important step: planning. As much as the agile world encourages agility, we must not omit steps, but rather do them effectively and assertively.
Defining a test automation strategy will bring several benefits, for example:
- What process to follow
- What tools to use
- What will be automated
- Who is responsible for what
- When will it run
- Which standards will be followed
- What to do in case of a failure
There are several other benefits that you will notice as you read the next section of the article.
The time you dedicate before starting will prevent you from having to stop to define things along the way, as well as prevent things from being done incorrectly and without standards. Even though we can always make improvements in the code, having a baseline will lower the test automation cost.
Therefore, whenever you need to include test automation in an existing software development project or start the development of a new system, take some time with the members of the quality team. Make sure to answer the 20 questions that will be presented in the next session, and document the definitions and why the choices were done ✅
Documenting helps you to keep all definitions accessible, and it’s good for the future to know what was defined and why that decision was made. And of course, let’s not forget that it will help when you need to train new people in your team.
The 20 questions
There are a lot of things that you should think about before coding. But in this article, we focused on the main 20 questions that you can think, discuss, answer and document without spending too much time in this stage.
Question 1: What will be automated?
This is perhaps the most difficult question to answer, but we start with it because it’s the most important one.
Defining what to automate is the most important decision in a test automation project. Are you going to automate frontend, backend, and/or mobile? Will you automate unit, integration, end-to-end, and/or regression testing? Are you going to do only happy path tests or alternative tests as well?
There are several questions that must be thought of in order to define what the scope of test automation will be. It is very important that this definition is done carefully, with wisdom, and with the participation of everyone on the team!
Question 2: When will it run?
The second question to be answered refers to when will the automated tests be run ⌚ Will you run the tests every day? During the day? At night? Only when a new feature is integrated into the project? Just at the end of the sprint? Every time a developer adds new code? Every time a new version is generated?
Defining when the automated tests will be executed is very important because the tests will only work for the project if they are executed. Tests that are created but not executed mean waste of time and money. So it is very important to plan when and how often you and your team want them to run.
Question 3: How will it run?
You need to think about how your tests will run. Will it be run manually when someone on your team opens the IDE and clicks the play button, or will it be in a pipeline? Will this pipeline run automatically after some action or automatically on a specific date and time? Will your tests be set up in a job (on Jenkins, for example) that should be run manually? Define a plan for how your tests will run.
Question 4: Where will it run?
Still talking about test execution, it is necessary to define where the tests will be executed. Do you have a specific environment for them to run? Is this environment a server that will run locally on your machine? Or will it be in the cloud? What environment settings will your tests need?
Question 5: Who will create?
Who are the people that will create the tests? Will it be just QAs? Will it be developers only? Will it be QAs and devs? Defining who is responsible for building the tests is very important, as this will influence deadlines and costs, as well as of course sharing responsibilities.
Question 6: Who will maintain?
As it is important to define who will create the automated tests, it is also important to define who will be in charge of the test maintenance when something changes. Sharing responsibilities helps with the team’s commitment, as well as in maintaining the automation strategy, without the risk of stopping it.
Question 7: How will it be tracked?
As with application code, automated tests also need to be tracked 👀 Where will this code be tracked? Within the same application code repository? In a separate repository? Will frontend, backend, and mobile tests be in the same repository? Which code versioning tool will be used?
Question 8: How are improvements defined?
The test automation code should also be improved. As automation is carried out, we realize that we can implement it in a better way, or that we can refactor something that is already done. It is necessary to define how the improvements will be implemented.
When someone on the team realizes that something can be improved, how will that person proceed? Where will this need for improvement be reported? Who will implement the improvements? When can it be implemented?
Question 9: When a test breaks, who will fix it?
Sometimes we discover that a test started to fail due to an original bad implementation or because of some requirement detail that was not noticed yet. It is necessary to define how the correction will take place, and who will make this kind of fix.
Question 10: What are the processes that will be used?
Perhaps the worst thing within a team that does test automation is not having a well-defined process. And when I say well defined, I do not mean complex and closed, nor that you need to spend a lot of time to define it.
Having a process defining what will be automated, how the automation will be performed, how it will be revised, who will take care of each part, and who will be responsible for the framework maintenance is very important.
It is better to have everyone going in the same direction and make adjustments if needed, rather than having everyone going in different directions.
Question 11: How will the benefits be reported?
It is always important to demonstrate the benefits that the project has obtained with the implementation of test automation. Whether to show to the team or to the company’s managers, having data to prove the efficiency of the work being executed is crucial.
How many tests are automated? What is the average implementation time? What is the average execution time? How many executions per sprint? What is the percentage of time saved compared to running these tests manually? What is the number of failures found by the automation?
This data is important to be collected on a daily basis, which will be essential every time.
Question 12: How will the defects be reported?
Another important item to be defined is how the defects found by the test automation will be reported.
There are two ways that this could be done: automatically or manually. A defect can be automatically opened as soon as a test runs and fails, without any human intervention, or it can be manually registered by someone after the failure has been analyzed 📊
Question 13: How will the origin of a failure be identified?
When a test fails, how will the failure origin be identified? It is very important to analyze a defect to see if it is an application defect or a test code defect. What kind of analysis will be done? What information is needed? Who will be involved?
Question 14: What is the automated testing coverage?
What is test coverage considering automated testing? Will it be analyzed based on the application code? Based on the requirements? Based on manual tests vs. automated tests?
It is very important to know the coverage of automated tests. But for that, you need to know the definition of coverage for this specific project, as it can change from company to company, from project to project.
A project that has a legacy of manual testing may account for coverage in a different way than a project that is just starting with automated testing.
Question 15: How will the test coverage be tracked?
After defining how the coverage will be accounted for, it is necessary to know how it will be measured. Will it be done manually? Will it be done automatically with some implementation done by the team itself? Will it be done by a tool?
Question 16: How will the decisions be documented?
In addition to the initial definitions of the test automation strategy, some decisions will need to be made throughout the project. For this, it is very important to document these decisions. Not only what was the taken decision, but also justifying why was it made.
Where will these decisions be documented? How will they be documented? Who is responsible for keeping this documentation up to date? It may seem like this is not important, but when you need it, you will be grateful for having everything well documented 📝
Question 17: How will current team members be trained?
When we decide to implement test automation, we choose the technology that will be used. But sometimes there are some team members that need to improve their test automation skills. For this reason it’s essential to:
- Analyze which team members will need training
- Decide how the training will be carried out
- Decide how much time is needed for training
- Decide what other activities are needed to distribute the knowledge within the team
If the team does not have a well-defined training plan, nothing will go ahead.
Question 18: How will new team members be trained?
The team has to be prepared to train new members who will join them. How will these members be trained? Will they do external training? How much training time is needed?
Question 19: Which test automation best practices will be used?
Everyone has a different work experience, so everyone has different standards and best practices. To build a code standard and to start automating tests, it is necessary to define what are the best practices that will be used. This will save you from a lot of trouble in the future when you need to maintain someone else’s code 👩💻
Question 20: What frameworks will be used?
Finally, one of the most difficult definitions is which test automation framework will be used. This will depend on many of the definitions that were answered in the previous 19 questions and will vary from project to project, from team to team.
The project type, the product technology needs, the team maturity, the programming language that will be used, the team skills, everything could influence this decision.
Cheat Sheet
In this section, you will get access to the 20 questions cheat sheet- a simple and well-defined summary with all the questions explored in this article. It is a quick-access resource that you can use before starting to automate tests in a project 💡
Conclusion
It is very important to know what should be done, how it should be done, and who is going to do it before you start automating tests. This will save the company’s time and money by enabling a well-defined test automation process and standards. The time invested in this step will be easily recovered with the reduction of future rework.
In this article, 20 crucial questions were presented, which must be answered before starting to automate tests of a particular product. It is a very important guide for defining the test automation strategy of any software development project.
Let me know in the comments if you have any other important question to add 💠