Writing robust test scripts is a crucial matter for test automation. It is absolutely important to make sure that the automated scripts are doing exactly what they are supposed to do, validating the same features that a tester would if he/she was running the test manually, therefore achieving the same results ✅
If the results from the automated tests are different from those obtained by a manual tester, then they are invalid, not catching existing issues in the system, or even failing when the result should actually be a pass.
The cause of those issues may be a wrong assumption made when the script was created, or maybe some previously executed test changed the system’s state and is influencing the subsequent scripts.
Fortunately, there are ways to avoid falling into those traps when writing tests. This article will describe them with examples written in the JUnit framework, presenting the anatomy of the scripts, their execution flow, and what is the responsibility of the methods with JUnit annotations. Following the tips from this article, you’ll be able to improve the robustness of your test suites 💪
In this article, we will be testing the features of database operations in a system. In order to simulate that, I created the following Java program with a very simplified structure, which performs insert, select and delete operations in an SQLite database:
import java.io.File;
import java.sql.*;
import java.util.Objects;
public class DatabaseExample {
private static final String SQLITE_DB = "database.db";
private Connection getConnection() throws SQLException {
return DriverManager.getConnection("jdbc:sqlite:" + SQLITE_DB);
}
public int createDatabase() {
try (Connection connection = getConnection();
Statement statement = connection.createStatement()) {
return statement.executeUpdate("CREATE TABLE IF NOT EXISTS Users (name string, email string UNIQUE)");
} catch (SQLException e) {
e.printStackTrace();
return 0;
}
}
public boolean dropDatabase() {
return new File(SQLITE_DB).delete();
}
public User getUserByEmail(String email) {
try (Connection connection = getConnection();
PreparedStatement stmt = connection.prepareStatement("SELECT * FROM Users WHERE email = ?");) {
stmt.setString(1, email);
try (ResultSet rs = stmt.executeQuery()) {
if (rs.next()) {
String userName = rs.getString("name");
String userEmail = rs.getString("email");
return new User(userName, userEmail);
}
}
} catch (SQLException e) {
e.printStackTrace();
}
return null;
}
public int insertUser(User user) {
try (Connection connection = getConnection();
PreparedStatement stmt = connection.prepareStatement("INSERT INTO Users VALUES(?, ?)")) {
stmt.setString(1, user.name);
stmt.setString(2, user.email);
return stmt.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
return 0;
}
}
public int removeUserByEmail(String email) {
try (Connection connection = getConnection();
PreparedStatement stmt = connection.prepareStatement("DELETE FROM Users WHERE email = ?")
) {
stmt.setString(1, email);
return stmt.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
return 0;
}
}
}
class User {
public String name;
public String email;
public User(String name, String email) {
this.name = name;
this.email = email;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
User user = (User) o;
return Objects.equals(name, user.name) && Objects.equals(email, user.email);
}
}
Test script anatomy
With our sample system ready to be used, let’s start writing our test class that will perform the validations. The first thing we have to do is create a new Java class and add a new method with the annotation @Test. After that, let’s call a method from the Assertions class name assertTrue, passing true as its argument.
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
public class TestDatabase {
@Test
public void myFirstTest() {
Assertions.assertTrue(true);
}
}
Congratulations, you’ve just created your first test! The Assertion class contains the methods that will check if the expected conditions are actually satisfied, passing, or failing the test. In this example, the Assertion is just checking if the argument received is true, then it will always pass. We will talk more about Assertion in the next sections of this article.
JUnit annotations
The tests must be annotated with the @Test annotation in order to inform that this is the method responsible to perform the validations. We can add as many tests we want to the same class, but we cannot control the order in which they will be executed. So, we need to pay attention to write tests that are independent of each other.
Let’s add some more useful validations to our test script. The test must insert a new user into the database and check if the operation was performed successfully. To do that I’ll replace the method myFirstTest and add a new one named testAddUser.
The method insertUser must add the user to the database and then return the number of users that were added, then we are expecting it to return 1 if successful or 0 otherwise. Let’s pass this condition as the Assertion argument in order to perform the validation:
@Test
public void testAddUser() {
DatabaseExample db = new DatabaseExample();
db.createDatabase();
User user = new User("First User", "[email protected]");
int usersAdded = db.insertUser(user);
Assertions.assertTrue(usersAdded == 1, "One new user added");
}
Now execute the test and see that they passed. Great! 🔥
But what happens if we execute them again, without any change? They are failing now, how is that possible? If you noticed in the database, the email field is defined as unique, so when we ran the test for the first time, the database was empty, so the new user was added correctly.
But when we run the same test again, it fails because there is already a user with the same email. The problem is because we are not cleaning the changes made during the test, so the database state is unknown when the test is being executed.
Suppose you are testing an access control system and your first test validates if a password can be changed. If you change the password during the test and don’t revert this change, then all subsequent tests will not be able to authenticate correctly and therefore won’t be able to execute correctly.
You can add some code to revert the changes at the end of your test, but what would happen if it fails to reach this line of code? That’s why there are the annotations @AfterEach and @AfterAll.
The annotations @AfterEach and @AfterAll inform that the method must be executed after each test or after all tests in a class respectively. They are always executed, no matter if the test methods passed, failed, or even threw some exception before reaching the end. That is why this is the perfect place to revert any change we did during the tests, not influencing the results of the next ones.
In order to revert the change we made in our test, we need to remove the user previously added to the database. So let’s create a new method named tearDown and annotate it with @AfterEach. Also, let’s convert some local variables from the test method into attributes in order to reuse them in the tearDown method. Now, this is what our class will look like:
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
public class TestDatabase {
private DatabaseExample db = new DatabaseExample();
private User user = new User("First User", "[email protected]");
@Test
public void testAddUser() {
db.createDatabase();
int usersAdded = db.insertUser(user);
Assertions.assertTrue(usersAdded == 1, "One new user added");
}
@AfterEach
public void tearDown() {
db.removeUserByEmail(user.email);
}
}
Now when our test finishes, the user added by it will be removed, and then the database will return to the state in which no user was added with that email 📩 If you run it again, it will still fail because the user is only removed after the test execution. Fortunately, similarly to @AfterEach and @AfterAll, there are also the annotations @BeforeEach and @BeforeAll, which are executed before each test or before all tests in a class respectively.
There it is also the perfect place to create the database which currently is being created inside the test method. We just need to create the database once, so let’s use the @BeforeAll annotation. We will also add another method to completely remove the database after running all tests with the annotation @AfterAll:
import org.junit.jupiter.api.*;
public class TestDatabase {
private static DatabaseExample db = new DatabaseExample();
private User user = new User("First User", "[email protected]");
@BeforeAll
public static void createDatabase() {
db.createDatabase();
}
@Test
public void testAddUser() {
int usersAdded = db.insertUser(user);
Assertions.assertTrue(usersAdded == 1, "One new user added");
}
@AfterEach
public void tearDown() {
db.removeUserByEmail(user.email);
}
@AfterAll
public static void removeDatabase() {
db.dropDatabase();
}
}
Please note that both removeDatabase and createDatabase are static, this is because @AfterAll and @BeforeEach annotations require the methods to be static.
Let’s add two more tests to our class, one to validate a query and another one to validate removal. Both of them require at least one user to be inserted into the database. We could use the one from the testAddUser, but as mentioned before we cannot control the order that the tests are executed. Then we need to create a setUp method that inserts a different user to be used in those validations.
Also, once this user will be removed from one of the tests, we need to guarantee that it will always be available in the database before each test, then we need to annotate the setUp method with the @BeforeEach. We are also adding more validations to the testAddUser in order to really validate if the user was inserted in the database, not trusting only in the returned value:
import org.junit.jupiter.api.*;
public class TestDatabase {
private static DatabaseExample db = new DatabaseExample();
private User user = new User("First User", "[email protected]");
private User user2 = new User("Second User", "[email protected]");
@BeforeAll
public static void createDatabase() {
db.createDatabase();
}
@BeforeEach
public void setUp() {
boolean userNotFound = db.getUserByEmail(user2.email) == null;
if (userNotFound) {
db.insertUser(user2);
}
}
@Test
public void testAddUser() {
int usersAdded = db.insertUser(user);
Assertions.assertTrue(usersAdded == 1, "One new user added");
User foundUser = db.getUserByEmail(user.email);
Assertions.assertEquals(foundUser, user, "User was found in DB");
}
@Test
public void testGetUser() {
User foundUser = db.getUserByEmail(user2.email);
Assertions.assertEquals(foundUser, user2, "User was found in DB");
}
@Test
public void testRemoveUser() {
int deletedUser = db.removeUserByEmail(user2.email);
Assertions.assertTrue(deletedUser == 1, "One user deleted");
User foundUser = db.getUserByEmail(user2.email);
Assertions.assertNull(foundUser, "User was not found in DB");
}
@AfterEach
public void tearDown() {
db.removeUserByEmail(user.email);
}
@AfterAll
public static void removeDatabase() {
db.dropDatabase();
}
}
With more tests in the same class, we can have a better idea about what is the execution flow of a JUnit test script. The methods with the @BeforeAll annotation run once before all the others, then it is time for the methods with @BeforeEach annotations to be executed, followed by the ones with @Test and @AterEach subsequently.
These three methods are repeated as many times as the number of tests in the class. Finally, the methods with the @AfterAll annotation are executed just once. The diagram below illustrates the flow:
Conclusion
Knowing the execution flow of a test script is essential to write robust test scripts that are less prone to contain errors. It is important to have in mind that automated test scripts are one tool, and like any other tool, we have to know how to use them properly in order to achieve the best results.
So next time you write a test script, remember to configure all pre-conditions in the setup methods with the annotations @BeforeAll and @BeforeEach, perform the validation in the test methods annotated with @Test, and perform the clean up in tear down methods annotated with @AfterAll and @After Each.
Please leave your comment for any feedback you might have about this article 📘