Technology is progressing fast, things that five years ago were a pipe dream, now are standard. But with new technology come new challenges. No matter who we are – Developers, Testers, DevOps, we need to pay attention to those changes. There is a lot of tools out there and more and more each day. That is why we need to be both efficient and strategic in how we learn about them. In this article, I will present to you a simple guide that can help you with the research on selecting the best tool for your organization. I am only covering the basics here.
Reason for tools research
Outside of affirmation that we need to stay on top, there are other reasons why you or your organization may need to research new tools. For example, to find the solution/tool that can help with a problem you are dealing right now.
In this article, we will look at research with resolving the problem in mind. Also, to avoid a conflict of interest – I won’t be using any real tools as an example.
Before we can do that, let’s take a closer look at the problem we are trying to solve.
Understanding the problem
Lets say team X in your company uses a new technology called “Future API”. Developers have only to build the user interface and define business logic. Then an AI will be configuring all layers of communication – making it much easier to use, and requiring much less design work. The issue is that since AI is responsible for it with slight changes in logic, it can completely rewrite APIs and change the contact, making the integration test impossible. Team asked you to help figure out how to test it.
Before any research starts, we need to understand what the problem is. Because you don’t want to look for a tool that will resolve the wrong issue. Remember your team doesn’t want a tool – they want their problem solved. I like the 5 whys approach for understanding the problem.
In our case, for example, is it worth considering why they want to do integration testing? Since they have no control over that layer, maybe it is unnecessary?
After asking around and observing what the team is doing, you realize the problem is deeper. There is no unit test framework for this technology, so developers have no fast way of validating their code. The UI is stable, and there are some E2E UI tests, but doing all Testing on UI will be very expensive. Plus thanks to this fast technology development pace – Developers can do in just one day what he had to spend the whole week before, but testing time hasn’t changed making testing even a bigger bottleneck.
The real issue is slow testing, and we need to make it faster.
Constraints
No matter if you are trying to solve a problem, or just building the toolbox, there are some constraints – note that they are not always solid wall. Some are more flexible than others. But they can cause friction 😵
Below are a few examples of such limitations:
- Price – what is the budget for the tool and its support.
- Legal – a lot of new products are collecting telemetry or are entirely cloud-based.
- TechStack – Technology never should be a hard constraint, but if to solve problems in let’s say Python you will have to add Java code… You need to be sure this is the best option.
Take some time to think about your constraints and write them down – the list will change as you learn more.
So after talking with the team, you found out that the biggest constraint is technology stack – the team has no experience in any technology outside of “Future API”.
Looking for candidates
The first place you should look into is your own or company toolbox – maybe you have something there. Unfortunately, probably you will have to start looking into it yourself.
As always search engines are a great help here – but the issue is quite often, you need to know what to ask to get the right answer 🤔
For example, the vendor may have recommendations on its page. Maybe they have some companies they work with that are providing a solution to your problem. Forums and communities are also an excellent place to check; it is a rare case to be the only one looking for a solution.
Of course, Technology radars are also an excellent place to look. Note at this point, you are not only looking for a specific tool but also for terminology that you can use to look for more devices.
Remember, for now; you are building a list of tools so don’t be too discriminating we will narrow it down in next stage.
Triage
We never have enough time to check all the tools we find, so we need to choose only the most promising ones.
Note – Even if you reject a tool, it is good to write down what you found about it and why are you rejecting it. It will be helpful for you in the future.
Remember our constraints? It is time to start looking into them. Put on top of the list Tools that fulfill all criteria.
Next, it is time to take a look at other aspects that are easy to check. Usually, I am trying to answer the following questions:
- Is this tool still getting updated? – Legacy tools may have a big problem with compatibility.
- Is it free or has a free trial/demo?
- What is the idea behind the tool?
- Are there any tutorials, presentations?
- Do you find this tool interesting?
- Do you think this tool will help you solve the problem?
- Take a look at its UI, is it user-friendly?
You will find most of this information by reading documentation, articles and posts made by the community.
If everything went well, you should remove quite a lot of tools from your list.
Experimentation
It is tempting to take the tool for a ride to see how it deals with the real situation. The problem is real-world issues are complex, and taking tool that you don’t know yet is an easy way to make a false opinion on the tool.
At first, do something simple like the getting started tutorial. Then use it on your own but don’t overdo it with the complexity of the task.
It’s good to have an environment to experiment – so all tools will face similar conditions. Again we are looking for a lot of information here:
First and foremost – can this tool help with our problem?
Yes, I just told you not to start with it, but you need to keep this goal in mind during the whole process.
Next, we have some secondary questions that will help us make a choice, for example, in the case of UI test automation, I am usually asking those questions:
- How long it takes to set up it?
- How hard is it to write the test?
- How hard is it to change the test?
- Are they easy to maintain?
- What do I need to set up CI?
- What is the recommended approach?
- What is the unique trait of this tool?
- Is this tool stable?
- Is it enjoyable to use, or is it frustrating?
- Are the tutorials useful or they are not helpful at all?
How much time should you spend per tool? I really can’t tell you – there is no silver bullet. For some, you may need a few hours while others will require more. Question is how much time you have? That is the main limiter. But in general, I would say that if you spend less then 8 hours with the tool, it is too little (unless you can’t make the tool work at all then I would say more then 4 hours is too much). For upper limit is you spend more than a workweek – it is too much for research. Of course, remember to record your findings.
The research went swiftly. The first tool you grabbed was easy to use. Unfortunately, the support for FutureAPI is limited and works only in simple cases for more complex apps like yours; it doesn’t yet work.
The second tool was so unstable that you couldn’t use it. After contacting customer support, you learned that this is common, but for now, they have no solution.
The third tool was a unit testing tool for FutureAPI, and it is excellent! Works well and is easy to use… the only issue is that it requires a specific way of designing applications with FutureAPI. For the current project, it would require too many rewrites, but it will be beneficial for new ones.
You take also look at the 4th tool that requires Java. The tool was challenging to use cause it needed an in-depth understanding of FutureAPI and Java. But after that, writing in it is a little slow, but it works like a charm.
Tool comparison
Ok, at this stage you probably should have all your information, assuming you were writing it down.
It is worth to take some time and sort your notes for each tool and standardize them so it will be easier to compare.
Recommendation
It is rare to find the perfect fit; usually, it is a trade of one tool that may resolve all your issues; others may be addressing only part of the problem. That’s why making recommendations is hard. What works best for me is to present a few options and explain their pros and cons.
So after comparing tools you know there is no perfect solution for them, the best option is to use the Java Library, but it would require changes to team composition. As a backup plan, you suggested the other tool, because even if it will cover only the most basic cases.
Summary
So we have taken a quick look at the tool research process. There is much more to say here, but for a start, this list will help you. So to shortly remind you:
1. Start with figuring out what you want to achieve. Solve the problem? Find something new for your toolbox?
2. Search for tools that can fulfill your goal.
3. When you have a list, take some time to read more about them.
4. Narrow list down to most promising candidates.
5. Experiment with them.
6. Write down your recommendation.
7. Share it with others!