In the advancement of the Agile era, frequent releases and faster feedback have become vital for software teams. To keep this cycle constantly up and running nothing could beat automation, and this necessity leads us to the topic of CI/CD. However, another challenge is performing continuous testing and rectifying major bugs before the application goes to production. In the last few years, we have talked a lot about Test Automation which significantly reduces the testing time by automating repetitive tests by making sure your stable features are untouched due to the integration of new changes. Here is a short overview of this interesting challenge and our CI/CD life before adapting TestProject.
Life Before TestProject – CI/CD with Bitrise
We included our automated tests as part of our CI/CD lifecycle and delivered continuous testing feedback to our team. However, things were getting difficult overtime when it came to managing test infrastructure, device management, and report sharing mechanisms within the team. Some of the solutions which we implemented during our early days of setting test automation pipelines were:
- Using Bitrise: We were using Bitrise as a CI platform for all our mobile apps, so we decided to run all our Appium tests on Bitrise itself. However, it has its own limitations such as we had to install and set up an Appium server, creating and launching emulators, waiting for emulators to be ready for execution, and no single dashboard for reports after test execution.
- Using AWS Device Farm: After trying out the above approach, we planned to move the test execution part to AWS Device Farm to take advantage of a wide range of emulators and better test infrastructure support. However, this approach required an additional code refactor to match the device farm’s acceptance criteria. Even after this implementation device ready waiting time and reporting were still an issue for us.
Mission TestProject Begins 🚀
Some of the key use cases which we wanted to cover by introducing TestProject into our automation stack were:
- Integrate TestProject SDK into our current Bitrise codebase: We were running a Cucumber based project with Java + TestNG + Maven for our mobile automation. To make it more compatible with TestProject we started with the integration of TestProject OpenSDK for Java and moved our project from Maven to Gradle (this was a personal choice, but Maven support is also available with TestProject). All the integration information is available on the official GitHub of TestProject.
- Run TestProject enabled codebase from Bitrise CI: TestProject currently doesn’t provide direct integration with Bitrise, however, the best part is they provide you with a bunch of REST APIs which can be executed as a part of any CI workflows.
API Generation Stage for CI Integration
Before you try out any of these APIs please make sure you generate an API key from your TestProject account.
1. Upload Test Package API
For the execution of your coded test, you need to pass your codebase in the form of a .JAR file. Here we need to use the POST API for uploading .JAR file to the TestProject cloud:
POST:https://api.testproject.io/v2/projects/{projectId}/test-packages/{packageId}
Authorization: Test Project API key (header)
fileName: Name of the JAR file (Query Param)
platform: Platform name iOS or Android (Query Param)
The next thing is to run the tests, but before that, we need to add those tests in a TestProject job, which we managed to complete in two phases:
2. Use Get Test IDs API
Get the list of test IDs that need to be added to the particular job. Use the below API that will help you to get your test IDs.
GET:https://api.testproject.io/v2/projects/{projectId}/test-packages/{packageId}/tests
Authorization: Test Project API key (header)
3. Update TestProject Job API
Once you get the Test IDs, you need to add those tests to your TestProject job.
POST:https://api.testproject.io/v2/projects/{projectId}/jobs/{jobId}/tests
Authorization: Test Project API key (header)
Body:
{
"position":0,
"tests":[
"{$TEST_ID_1}"
"{$TEST_ID_2}"
"{$TEST_ID_n}"
]
}
4. Run TestProject Job API
Finally, it’s time we tell TestProject to run our job on Sauce Labs. We will be using the run job API to define our agent (Sauce Labs) ID to run our tests.
POST:https://api.testproject.io/v2/projects/{projectId}/jobs/{jobId}/run
Body:
{
agentId: "{SAUCE_LAB_AGENT_ID}"
}
Architecture
Add TestProject to your CI Pipeline
For the CI platform, we are using Bitrise here as part of our Organization’s adopted CI/CD platform. However, the entire flow is more towards calling APIs at different stages. We can configure something similar to other CI/CD platforms like Jenkins, CircleCI, Buddybuild, etc.
1. Clone your Repository
Clone your test automation code base to Bitrise using this step in a folder at run time or you can directly clone into the Bitrise source directory as well.
2. Set JAVA_HOME to JAVA 11
This step is required because stacks on Bitrise might use JAVA 8 as the default JAVA version. For TestProject OpenSDK to work, we need to switch the JAVA version to 11.
jenv global system export JAVA_HOME="$(jenv prefix)" envman add --key JAVA_HOME --value "$(jenv prefix)"
3. Generate Test Package as JAR file
Since we need to upload our coded project in JAR, we will be running another script step using the Gradle command.
cd PROJECT_DIRECTORY gradle wrapper echo Generating .JAR file....... ./gradlew clean jar
4. Upload JAR file to TestProject
We will be calling the POST API to upload the JAR file in the form of a CURL command.
curl --location --request POST "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/test-packages/$TEST_PROJECT_PACKAGE_ID?fileName=$TEST_JAR&platform=Android" \ --header "Authorization: $TEST_PROJECT_AUTH_TOKEN" \ --form 'name=@"/$PROJECT_DIR/build/libs/APPLICATIO_NAME.jar"' \ --form 'content-type="application/octet-stream"'
5. Update TestProject Job with Test IDs
As mentioned earlier, we need to update our job with the test IDs we want to run with that job.
i) Get your Test IDs:
TEST_ID1=$(curl -X GET "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/test-packages/$TEST_PROJECT_PACKAGE_ID/tests" -H "accept: application/json" -H "Authorization: $TEST_PROJECT_AUTH_TOKEN" | jq -r '.[0].id') echo "" TEST_ID2=$(curl -X GET "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/test-packages/$TEST_PROJECT_PACKAGE_ID/tests" -H "accept: application/json" -H "Authorization: $TEST_PROJECT_AUTH_TOKEN" | jq -r '.[1].id') echo ""
ii) Update TestProject Job:
curl --location --request POST "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/jobs/$TEST_PROJECT_JOB_ID/tests" \
--header 'Content-Type: application/json' \
--header "Authorization: $TEST_PROJECT_AUTH_TOKEN" \
-d '{"position":0,"tests":["'"$TEST_ID1"'","'"$TEST_ID2"'"]}'
6. Run your Job on SauceLabs
This will be the final step of our workflow, where we will be executing our TestProject job on Sauce Labs. Let’s use another CURL command to run the job.
curl -X POST "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/jobs/$TEST_PROJECT_JOB_ID/run" -H "accept: application/json" -H "Authorization: $TEST_PROJECT_AUTH_TOKEN" -H "Content-Type: application/json" -d "{ \"agentId\": \"$TEST_PROJECT_AGENT_ID\"}"
❗ Important Note: We are uploading our APK to AWS S3 bucket from the build stage of our workflow and configured the S3 bucket path as Appium capability in our code.
cap.setCapability(MobileCapabilityType.APP, AWS_S3_BUCKET_APK_PATH);
Summary
Well, that’s how we managed to set up an E2E CI/CD pipeline using TestProject for our mobile tests. Overall my key takeaways from this journey are:
- TestProject helped us to offload the entire test automation infrastructure, which we used to set up before running our tests. This directly reduced the overall pipeline execution time to half on Bitrise. Now, our pipeline only focuses on building the project and passing the bundle to TestProject.
- Secondly, reports were one of the bottlenecks for us once the test execution was completed, but, with TestProject we can now share better dashboard and execution results with our team on a single place for our multiple apps, where each team can see their test execution results.
- I could clearly see that “TestProject is the future” and really beneficial for individual contributors who generally have to deal with multiple projects. I appreciate the TestProject team for providing such a great platform 😊
Happy Testing!