My Recent Obsession with Hermetic Tests

Over the years, after dealing with test flakiness due to the dependencies surrounding them, I started searching for answers. One day, my mentor sent me an article that discussed hermetic tests. It was the first time I heard the term, but the concept seemed very intriguing, so I started researching it 📖

What are hermetic tests?

A hermetic test is a test that is completely self-sufficient. It is fully independent, therefore every time a test is run, you are absolutely sure that a failure is a real failure, and had nothing to do with a dependency going wrong.

The way that it works is you set up a test by injecting mock data or completely mocking your dependency. While this looked like a perfect test for the problems we’ve been encountering with flaky tests, the way to achieve it meant a fundamental change to the way the software was developed.

Some context

We decided to implement it for the mobile app that we were working on 📲 This time, we wrote our mobile app tests on Espresso for Android, and XCUI tests for iOS. They share similar patterns and architectures from the testing point of view. For this article, we will discuss the android UI tests and how we managed to achieve a hermetic tests pattern.

The architecture of mobile apps, in general, meant that there were two levels of software dependencies. The first was the API it was talking to, and the second was SDK dependencies. So we will see how we managed to mock these two dependencies.

Android case study

The recent app that we were working on, used the Hilt Android library in its architecture. Hilt is primarily used for dependency injection. It also used a retrofit library for making network calls.

Mocking the API

This was the easiest part since for both mobile and web apps there were so many libraries that helped mock network calls. We set up test data and left it as part of the UI test package, and let the MockWebServer library mock them.

private val server: MockWebServer = MockWebServer() 
private val MOCK_WEBSERVER_PORT = 8000

@Before
fun init() {
    server.start(MOCK_WEBSERVER_PORT) 
    server.apply {
        enqueue(
            MockResponse()
            .setBody(MockResponseFileReader("jsonplaceholder_success.json")
            .content))
        }
    }
}

From the above example, we can see how we are able to mock an API call using a mock test data file 📂

Now that we are over the easy bit, let us look at the second part where we try to mock the SDK dependencies. This is where it gets interesting in a good way. In my own experience, forcing hermetic testability has made it mandatory to use the best practices while software development. The reason being, it fundamentally forces the development to be modular and have independent components that can easily be mocked or faked.

With the android hilt architecture, the way we implemented mock dependencies was using the @InstallIn annotation:

@UninstallModules(AnalyticsModule::class)
@HiltAndroidTest
class SettingsActivityTest {

    @Module
    @InstallIn(SingletonComponent::class)
    abstract class TestModule {

        @Singleton
        @Binds
        abstract fun bindAnalyticsService(
            fakeAnalyticsService: FakeAnalyticsService
        ): AnalyticsService
    }
}

In the above example, you can see that the actual implementation of the analytics module has been removed using the annotation @UninstallModules. Then the mock implementation was injected into the UI tests using the @InstallIn annotation.

The result

Now that we have mocked the two main dependencies before even the test runs, the app opens in the exact state that you want it. So any test you run fully tests the core logic of the implementation rather than the complete app along with the dependencies.

Counter argument

There is a really valid argument around not following hermetic tests pattern but testing the dependencies, too. I agree with where they are coming from, but when you get to a point where having UI tests don’t add value because of the flakiness that the dependencies produce, I feel this test pattern certainly adds so much value.

On top of that, there are scenarios where there is user input required due to the nature of the workflows (eg. hardware dependencies). Using a hermetic test pattern there would definitely help you test the complete core workflows, including failure scenarios, by implementing hermetic tests.

Was it worth it? 💯 

Conclusion

The example that has been provided in this article is for mobile apps. Although, the hermetic test pattern can work really well for any UI test automation framework. For example, with web apps, you can consider the hermetic test pattern as an extension of the storybook for react.

As long as you can inject your API and state into the UI and construct tests around it, you will get a lot of value out of it and have tests that run super fast, with less flakiness ✅ I highly recommend giving this a try with your existing UI test automation frameworks. Good luck!