The world of software testing has progressed significantly during the last decade and beyond. Every few months, there comes some new tool or process to make it better. But the challenge remains regarding how to effectively and adequately test the applications when it spans in a distributed system, like a microservice architecture 💡
This article discusses the design principles and best practices of software testing for microservices architecture.
Table of Contents
Understanding microservices
Architecture matters because of how it affects the quality of the product. Most notably, it impacts the software delivery velocity – maintainability, extensibility, and testability.
Chris Richardson speaks of three aspects in defining the microservices architecture in his book “Microservices Patterns“, inspired by Martin Abbott and Michael Fisher’s three-dimensional scalability model – the scale cube 🎲 It proposes three ways to scale an application:
- X-axis scalability or horizontal scaling – running multiple identical instances of an application behind a load balancer.
- Y-axis scalability or functional decomposition – decomposing monolith applications to granular, loosely coupled elements having bounded context.
- Z-axis scalability or data partitioning – scaling application, based on data partitioning, where each instance is responsible for a subset of data.
The key factors that drive the microservice architecture are modularity and scalability. The complexity of the architecture grows with the increased number of microservices, the introduction of async messaging, caching, etc.
The initial focus of testing an application is predominantly delivery-focused (doing it right to move to production), however, once it goes live, the distinction blurs between development and production-related activities.
It’s challenging!
The increased focus on faster software delivery has evolved monoliths to microservices. There has also been increased attention on software testing and observability, which advocates more test automation. However, designing an effective test strategy with high-automation test coverage is profoundly challenging.
While Mike Cohn’s test pyramid is widely used as the best practice in constructing the microservices testing strategy, the implementation remains far from the ideal. The most common challenge remains at the test design phase – implementing the proper set of tests at the right place ✅
Here are some of the complaints that we get to hear often from the development teams:
All good in isolation, but inconclusive unless tested integrated
One of the biggest problems in microservices is integration testing with many granular bounded services. The teams need to have thorough knowledge regarding all the services in context. It demands increased coordination between the teams and becomes a bottleneck to software delivery velocity.
Stubs are excellent, but we can’t trust those
To mitigate the troubles with integration tests, the teams often start using more test doubles. While it eliminates the dependency on other teams, often the use of mocks and stubs becomes overwhelming and hard to manage.
On top of it, in many cases, a feedback loop is missing on the validity of the test stubs. The integrating service implementation works differently from the stubs, eventually walking in the path of end to end tests.
The debate – Building the right thing vs. building the things right
Another challenge commonly faced by the teams is implementing the layered test suites properly. Many development teams have the desired skills and capabilities to implement testing strategies, but often the tests are not implemented in the right place. As a result, the rightward test phases become heavier and create the anti-pattern. The more we shift towards the right, the more the tests become costly, brittle, and indeterministic.
It is often a common topic of debate among engineers whether it is necessary to focus on test pyramid adherence or the effectiveness of the tests. This debate is more about the short-term vs. long-term benefits.
Although implementing more lower-level tests are costly in terms of development time, it gives immense rewards in the longer run – by reducing the tech debt, good knowledge of the application components, and high maintainability.
I know what went wrong but have no idea where it happened
The tests are often incapable of tracing back to the root cause of the problem. In many cases, it is challenging to aggregate application logs and test reports as evidence of a test failure in a single visualization, making the feedback incomplete. It needs extra effort to trace back to find the root cause of a problem.
On top of it, many communication in the reactive microservice architecture happens in asynchronous message-driven ways, which adds more trouble in following the after-effects of an event.
Our test environment is not a replica of the production
The microservices by the design principles are easier to test in isolation than the monolith systems – they are smaller and more focused. But it is a nightmare when we need to spin up a new test environment.
It needs to have many service deployments, spinning up databases, starting a new messaging system, load balancer, a service gateway, and many more. When the teams lack the capability of spinning up on-demand environments for testing purposes, it is highly possible to miss out on certain aspects till the code moves to pre-prod or prod.
Where there’s a will, there’s a way
Yes, it is difficult, but not impossible, to implement an effective microservice testing strategy. The modern tech influencers have been relying extensively on microservice architecture. There are hundreds of case studies available over the internet on this topic. Here is my take 👇
While we should embrace new tools and technologies to remain relevant with software delivery velocity, we should stick to the fundamentals. Lisa Crispin and Janet Gregory’s book “Agile Testing” describes a modification of Brian Marick’s Testing Quadrants that helps to classify different types of tests.
This classification is one of the best ways to start segregating tests and put those into proper testing phases. At the bottom of the quadrant, the tests are technology facing – these are small in size, and the scope is limited to units/components. These are the best candidates for test automation.
The top half of the quadrants are more focused on business functionalities – the scope is much larger. Typically these tests are used to determine the feature readiness.
Understanding the “Big Picture”
The scale-cube model is not only effective in understanding the microservice principles, but it also gives an insight into designing the testing strategy. On a high level, the testing objectives need to align with the three axes of the scale cube:
- Y-axis testing – Testing the microservices as per the functional decomposition. These tests are typically unit, component, contract, and integration tests. The lower-level tests depend on the test doubles (mocks and stubs), while the higher-level tests are more end to end and typically work under an integrated environment.
- X-axis testing – Testing the horizontal scaling capability. Typically performance, load, stress testing, and chaos engineering help to assess the application’s auto-scaling, self-healing properties.
- Z-axis testing – Testing the application behavior consistency across data partitions. These tests generally assess the application’s capability of routing requests to the appropriate server that contains the data. These tests provide insights on data consistency and SLA adherence by routing requests to a different set of servers.
Towards building the right things
A good understanding of the fundamentals and the big picture assists in creating the right testing strategy. Then comes the solid test design principles 📃 It needs discipline, constant communication among the team members and beyond. Better test designs require both technical and cultural changes.
Lower level tests are cheaper
This idea needs to get into the DNA of the teams – lower-level tests are cheaper and have faster delivery feedback. These test results can show the application problems in the function/unit/component level – hence, it is much easier to fix.
Adherence to the test pyramid helps to have low-cost tests and a much lower cost to fix defects than any higher level. So, it is critical to understand and stick to the pattern.
Code coverage is not enough
It is often a familiar argument to consider code coverage as the quality of unit testing in the application. But code coverage alone is inconclusive. We can achieve 100% code coverage through unit tests, with missing code for functional behavior. So, while code coverage still needs to be accounted as one of the key metrics in measuring quality, the scope of unit tests needs to go beyond that.
Unit tests need to be more functional
It needs to be designed beyond thinking in terms of permutations, and taking a more realistic business-oriented testing approach. This process is tricky, and it requires close coordination between developers and the quality advocates in a team.
Use stubs – make the stubs reliable
The lack of a feedback loop for test stubs is a common problem in automation testing. Teams make use of stubs enthusiastically when they start, eventually losing confidence in these. Two main things make the stubs unreliable:
- The number of variations in business scenarios becomes overwhelming,
- The lack of process in optimizing and validating stubs against the real systems.
The teams should focus on implementing an innovative stub solution that relies on a mix of static and dynamic stubbing strategies. It also needs to have an automated periodic validation system that can change (or at least highlight!) when the stubs are outdated. This scenario is one of the candidates for contract testing – a process that gives you more confidence in seamless integration and helps in reliable stubbing 💪
Design automation – do not let it grow organically
One of the biggest hurdles with increased test automation is the lack of planning. With the fast-paced delivery model, most teams do not have a separate test planning phase. This gap causes many anti-patterns in the implementation. Agile is about the test and learn. The teams must start with a plan and a strong measuring capability implemented from day one.
The constant measuring helps in diagnosing the problems in the process and mitigate them. We cannot anticipate what will come in the way from the beginning, but measuring can highlight code redundancy, test proportions, flakiness, and the gap in testing. Keeping close control on test automation design helps in having a robust, scalable, and reliable framework.
Know what went wrong and where
One of the factors that require attention during the test automation framework design is its traceability. The automated tests must have a way to either bringing in application logs and test results in a single test reporting pipeline, or a unique identifier that can trace all the events during the test’s lifecycle and the after-effects. It will help in increased observability of the tests and a reduced test analysis effort 📈
It’s about conversations and interactions
Agile is all about conversation, faster feedback loops, and so must be the tests. It is a problem while integrating different components – the lack of an automated process for interactions between the integrating parties. Consumer-driven contract testing helps in bridging that gap between the provider and consumer systems.
These tests automatically share consumer expectations to the provider – a decisive feedback loop that prevents the breaking integration between the parties. It is also a solution to stub validation, as the consumer expectations get validated at the provider end.
Spin up an environment at the click of a button
X-axis scaling of scale-cube talks about horizontal capability – one of the most critical aspects of microservice design. Most of the modern microservice systems are containerized applications.
The deployment of the applications on new infrastructure needs to be automated so that the teams can spin up or bring down new environments on-demand basis. This automation helps in faster testing cycles in a cost-optimized manner.
“A journey is a person in itself; no two are alike” – John Steinbeck
Nothing is perfect; it is always a journey towards perfection. The software architecture keeps changing and rotating very fast these days. So, it is a challenge to keep up the quality with the best test automation design.
Observability is the key – watch, discover, act. This journey makes the system somewhat near perfect! I hope you enjoyed reading, happy testing 💫