Test Automation Pipelines for Modern CI Flows

The web and the mobile application development and release cycles have evolved dramatically over the past years. With the rising demand for agility, everything is about speed and reliable releases 🏃‍♂️

Continues integration (CI) and sometimes Continues Deployment (CD), are almost always required for agile software release cycles. Modern CI flows rely immensely on solid test automation that in most cases used as a gatekeeper for software releases.

In this tutorial we will dive deep into 4 best practice methods to create automation pipelines for modern CI flows within minutes, by utilizing next generation tools that are available nowadays in the test automation market.

For convenience, we are going to use the TestProject free test automation platform. However, it can be replaced with any other tool of your choice since the concepts I’ll be sharing in this tutorial also apply for pure Selenium, Appium, or any other modern automation tool or library.

So, grab your favorite tools & let’s get started 🔨📐🚀🔍

Table of Contents: Automation Pipelines for Modern CI Flows

1. Parallel Execution with Docker Containers
2. On-Premise Execution for Sensitive Platforms
3. API Integration with Any 3rd Party Tool
4. Zero-configuration and Orchestration by the Platform 
5. Conclusions

---

1. Parallel Execution with Docker Containers

Automation pipelines that utilize Docker containers are very popular these days and for a good reason 🐳 Containers allow us to easily scale for parallel execution, they are easy to deploy and low footprint by nature. A docker container is a self-sustainable package that includes all the dependencies and software required to run standalone. Since containers are executing everything in a sandboxed environment, they are OS agnostic and always run exactly the same regardless of the host OS and/or configuration, for instance, between development and staging environments. Here you can read more about docker containers. Hence with anything related to scaling with test automation and parallel execution, containers are the perfect candidates for deployment and execution of tests.

TestProject’s Agent makes automation deployment extremely easy for you, since it’s already bundled with the latest Selenium & Appium drivers and built-in reporting capabilities. The TestProject Agent docker container can be used in two ways:

1. Permanent execution engine – In this scenario, the agent is registered once and can be used to execute tests or jobs from the TestProject web application.
2. Ephemeral instance – In this scenario, the agent is started to perform a specific task (i.e., run a job), and will self terminate upon completion.

If you are utilizing TestProject’s hybrid cloud to store your test artifacts – this process becomes even simpler. TestProject’s Agent will deploy all of the test cases for you, run them and report the execution results automatically. All you will need to provide is the jobs ID that contains the tests to execute, and an API key for authorization purposes. That’s it! 🤯

This diagram describes the automation pipeline for parallel execution with Docker containers:

2. On-Premise Execution for Sensitive Platforms

On-premise pipelines typically use internal version control repositories such as self-hosted GitLab or BitBucket servers. The automation artifacts are stored alongside the code or in a dedicated repository, allowing the CI to build and execute against the testing/staging environments. With on-premise CI flows you can also utilize dockers containers or dedicated VMs. One of the main challenges of on-premise automation is to deploy and configure the required tools for executing your tests, such as: Selenium browser drivers, Appium server for mobile testing, automation frameworks, data-driven files, etc.

Starting from TestProject v2.0 it is now possible to build on-premise CI flows while benefiting from the TestProject Agent that is bundled with all of the drivers and configurations needed for your web and mobile automation to run.

On top of that now you can now save your codeless test scripts locally in your version control alongside your code. Deploy and run them using an integrated CLI tool with a simple command:

testproject-agent run MyFirstTest.yaml 

The TestProject Agent will do all the hard work for you, by running the tests and creating rich HTML reports.

This diagram describes the automation pipeline for on-premise CI flow:

3. API Integration with Any 3rd Party Tool

Modern automation platforms usually provide an API that allows you to trigger automation pipelines from a 3rd party tool. TestProject is no different and provides a rich RESTful API for triggering test automation from anywhere, even from your command line! 😎

TestProject will do all the heavy lifting for you: deploy the tests, run them, report the results and notify the pipeline upon completion ✅

The following examples use the “curl” command line utility for making the API calls to TestProject. The <PLACEHOLDERS> from the examples should be replaced with values from your TestProject account.

Running a Job

curl -X POST "https://api.testproject.io/v2/projects/<PROJECT_ID>/jobs/<JOB_ID>/run" -H "accept: application/json" -H "Authorization: <API_KEY>" -H "Content-Type: application/json" -d "{}"

Running a Test

curl -X POST "https://api.testproject.io/v2/projects/<PROJECT_ID>/tests/<TEST_ID>/run" -H "accept: application/json" -H "Authorization: <API_KEY>" -H "Content-Type: application/json" -d "{}{ \"agentId\": \"<TARGET_AGENT_ID\", \"browser\": \"Chrome\", \"queue\": true, \"testParameters\": { \"ApplicationURL\": \"https://example.testproject.io\" }, \"projectParameters\": { \"Name\": \"John Doe\" }}"

💡 This is applicable for Jenkins, Github Actions, TeamCity, CircleCI, Azure Pipelines, Bamboo and propriety build scripts. The complete TestProject API documentation is available here: https://api.testproject.io/docs/v2/.

This diagram describes the API integration CI flow:

4. Zero-configuration and Orchestration by the Platform

TestProject provides a complete scheduling and execution mechanism. To implement a CI flow with TestProject all you need to do is create jobs! A Job is an aggregation of the following information:

1. Target Agent (the execution engine)
   • Local or remote physical agent.
   • Virtual agent to run on 3rd party device farms, such as Sauce Labs or BrowserStack.
2. Target platforms: Browser(s) or mobile device(s)/emulator(s)/simulator(s).
3. Data source: A CSV file with the set of parameters for data-driven tests.
4. Schedule: When to run the job (as seen in the image below). You can run it on-demand (manually), set it to run just one time or use the built-in scheduler for recurring executions at your preferred timing. Simply pick a time and date and TestProject will schedule the job execution for you and make sure it gets executed at the selected time frames.
   
5. Notifications: You can define job notifications to inform yourself and your team when a job starts execution and when it completes. You can also choose to be notified only for failed executions. By using any of TestProject’s built-in notification options: email notifications, slack integration, or webhooks – once you set up the type of notification you want, you will receive them with a nice summary of the execution, including a link to the full execution report.
6. Custom capabilities: If you need to execute your test with special conditions on your target browser or device, you can define custom capabilities. These are key-value fields that provide additional configuration to the target browser or mobile device.

Conclusions

As test automation pipelines become more powerful, they take a much more significant place in modern CI flows. In most cases, the pipeline will involve more than one cloud or a combination of clouds & on-premise topologies.

Today’s automation frameworks have definitely evolved, and allow Testers, SDETs and Developers to build automation pipelines quickly and test products without spending much time in scripting or tooling, and without heavily involving DevOps/infrastructure IT engineers.

So, what’s your go-to strategy for achieving modern CI flows? Share with us in the comments below! 😉