When I look back at the first architecture I designed years ago, it is clear how many integration points we have now compared with the classic LAMP stack: Linux, Apache, MySQL and Python/PHP/Perl.
Nowadays, the data our application generates are the most important asset we have and that information has to be shareable and usable by machine learning algorithms, third-party applications. Even internally, a lot of teams need a fraction of those data to improve the services you offer. Think about the sales team, they need a curated subgroup of information to be effective during their daily job, same for your support team. As more as they can read about the interaction between a customer with the application, the more helpful they can be with the customer itself.
That’s why we tend to build or on-board services that integrate with the main application to enrich the fruition of the data.
Our application servers API used by 3rd parties in some cases, we outsource our login system to services like Auth0 because it gives us an admin UI, an API and a lot of the features that we have to build in order to make our project usable from teams that are not programmers.
All those integration points forced developers to build abstractions to avoid vendor lock-in. If you have to generate tickets for your sales team in SalesForce you have to be sure that tomorrow you won’t be locked by the use of that particular platform, those layers have to be tested.
Unit tests are a solid way to check what a function returns based on the input. You can inject errors policy to simulate a real environment where nothing goes all the time as planned.
Side by side with unit tests, you can write integration tests, where you do not have to mock all the dependencies for the function you are testing, but you can use the dependency itself. To battle testing the abstract layer you wrote for storing the history of the customer login, you can create a mock and validate the abstract. However, if you have to test how it performs with different databases like MySQL, etcd, Cassandra or even more, you have to close a bug that only happens with a particular MySQL version, it is useful to run your code using that.
Provisioning and spinning up the environment for your integration tests is a challenge. In theory, you need to have a dedicated environment to get enough isolation avoiding collisions between tests deleting, modifying and checking the same data. You also have to design a system that allows parallelization because integration tests tend to be a bit slow.
Today almost every continuous integration infrastructure has Docker running into it. Speaking about Docker, it providers isolation and it is a very effective way to spin up applications quickly.
TestContainers
TestContainers is an open-source community that provides a set of libraries across languages that leverage the Docker API to write integration tests. It is popular in Java and I am the maintainer for the Golang version, you can find the list of supported languages on GitHub.
In 2019 I gave a talk at DockerCon about testcontainers, but it is a vibrant community, we have a lot to say about it again! 🤗
You can think about testcontainers as a library that wraps the Docker SDK in your own language to interact with the Docker API, but in a way that is friendly when writing tests.
For example, the first test starts a container in the background called Riuk, its responsibility is to clean up containers created by testcontainers when your test suite runs in order to purge all the dead unused containers, keeping your environment clean. But more in general, the API provided for the library well covers the testing use case.
The next two chapters below will contain a set of snippets from the official documentation and from open source projects that use testcontainers for their test suite in Go and Java.
TestContainers Java
public class ElasticsearchStorageRule extends ExternalResource { |
I got this piece of code from a popular open-source project in Java called Zipkin. It is a tracer and it supports different databases where to store the traces used by your application. Like Cassandra, MySQL, Postgress, ElasticSearch or in memory.
The test suite is always the same, what changes is the concrete implementation for the storage system that it runs. This example uses ElasticSearch and as you can see the before() function creates the container that will be used by the tests.
new GenericContainer(image)
.withExposedPorts(ELASTICSEARCH_PORT)
.waitingFor(new HttpWaitStrategy().forPath("/"));
container.start();
GenericContainer is the object that wraps the Docker API, as you can see it accepts a docker image and a waitingFor strategy that will tell you when a container is ready to handle the traffic.
At this point in time from the container, you can get the IP and the PORT to configure the ElasticSearch client.
container.getContainerIpAddress() |
The container GenericContainer object serves those two functions because in order to avoid IP and port collision, testcontainers automatically use and map the port you asked for with a random one available in this way you can parallelize your tests, having running at the same time more ElasticSearch containers.
TestContainers Golang
// "github.com/testcontainers/testcontainers-go" |
This code comes from kube-profefe, a Kubernetes operator and kubectl plugin that I am writing:
ctx := context.Background()
req := testcontainers.ContainerRequest{
Image: "profefe/profefe:git-10551f2",
ExposedPorts: []string{"10100/tcp"},
WaitingFor: wait.ForLog("server is running"),
}
profefeC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
})
The GenericContainer function creates the container using the specified request, that as you can see, it sets a specific Docker image, the ports to expose and it also has the WaitingFor strategies. In this case, I am using the ForLog strategy that waits for a logline from the container before moving on declaring the container as running.
defer profefeC.Terminate(ctx)
With this single line, you are keeping your environment clean, removing the container when your tests end. As I wrote before, there is a container running in a background called Riuk that does the cleanup, but you also have the option to Terminate the container if you prefer or if you need to.
ip, err := profefeC.Host(ctx)
if err != nil {
t.Error(err)
}
port, err := profefeC.MappedPort(ctx, "10100")
if err != nil {
t.Error(err)
}
client := NewClient(Config{
HostPort: fmt.Sprintf("http://%s:%d", ip, port.Int()),
UserAgent: "testcontaners",
}, http.Client{})
At this point, we can get the Host and Port (for the same reason explained in the Java example above) and we can configure the profefe client to point it to the running container.
Conclusion
The ability to spin up and control your environment from the test is a powerful approach because you are using code. One of the most diffused alternatives is docker-compose, but the WaitStrategy, for example, gives you very granular control over the lifecycle of the container. You do not need to hack a loop that checks if and when the container is ready to handle the traffic.
For the fact of having code around you can also attempt to recover from a provisioning issue retrying the container creation, avoiding flakiness that is another huge pain that makes integration tests difficult to trust and maintain.
We just scratched the surface for this topic, there are a lot of gems not explained here about those libraries. Let me know on twitter if you would like to read more about this topic @gianarb, and share your comments below! 😎