Be Descriptive With Your Test Failures

Descriptive test failures can save time and money. While the return on investment may not be immediate, future developers (possibly yourself), will appreciate the effort. Tests that catch legitimate failures are a blessing 😇 However, a non-descriptive error requires debugging to find the real issue.

Short-circuited assertions

A short-circuited assertion occurs when the first assertion fails and subsequent ones do not execute. The following example is a simplified version of a common scenario. It uses supertest with jest to test an API, but the same idea can be applied elsewhere.

app.get("/user/:id", function (req, res) {
  // ...logic
  res.status(403).send("SOME_VALUABLE_ERROR_CODE");
});

test("a user can be fetched", (done) => {
  request(app)
    .get("/user/1")
    .end(function (err, res) {
      expect(res.status).toEqual(200);
      expect(res.body.name).toEqual("Slugathor");
      done();
    });
});

The test above expects the call to always succeed and may have historically passed for a long time. However, when a change causes it to suddenly fail, it leaves a vague message that could become a headache for whoever troubleshoots 🤕

Diagnosing in a local environment

test("a user can be fetched", (done) => {
  request(app)
    .get("/user/1")
    .expect(200)
    .end(function (err, res) {
      if (err) {
        console.log(res.body);
        throw err;
      }
    });
});

💡 Logging the whole response object, while verbose, would allow inspecting other issues, like invalid headers.

After discovering the error code, the investigating developer can pinpoint the issue. Unfortunately, some people view console.log statements as pollution and will remove them before committing code. Doing that only recreates the issue and loses time for the next person.

Diagnosing in a Continuous Integration (CI) environment

Rewriting the test

There are other ways to write the test more descriptively.

Separating assertions

The following example separates the assertions into separate tests, so one assertion failing does not skip the subsequent.

describe("/user/:id.", () => {
  let response;

  beforeEach((done) => {
    request(app)
      .get("/user/1")
      .end(function (err, res) {
        if (err) {
          throw err;
        }

        response = res;

        done();
      });
  });

  test("returns a 200 status code", () => {
    expect(response.status).toEqual(200);
  });

  test("returns the expected name", () => {
    expect(response.body).toEqual({ name: "Slugathor" });
  });
});

expect.extend({
  toHaveOkBody(response, expectedBody) {
    if (response.status === 200 && response.body === expectedBody) {
      return { pass: true };
    }

    return {
      message: () =>
        `Unexpected response: ${JSON.stringify(response, null, 4)}`,
      pass: false,
    };
  },
});

test("a user can be fetched", (done) => {
  request(app)
    .get("/user/1")
    .end(function (err, res) {
      if (err) {
        throw err;
      }

      expect(res).toHaveOkBody({ name: "Slugathor" });
      done();
    });
});

Uncaught exceptions

Sometimes assertions work with a happy path but mask the error message when failing. Regardless of the framework (Chai/Jest), Expect APIs provide descriptive failures on their own. Those messages are meaningless in the following example, which has an unexpected result.

it("should fetch a user", (done) => {
    request(app)
      .get("/user/1")
      .send()
      .then((res) => {
        expect(res.body).to.deep.equal({ name: "Slugathor"});
        done();
      });
  });

What happened?

Test frameworks (Mocha in this example) wrap each test, catching exceptions that are thrown from expect statements. By using the done callback, the framework continues execution until it’s called. Since the expect statement throws, the callback is skipped. This results in the appearance of the test failing from a timeout, masking the real assertion. Unfortunately, this error could trick a developer into thinking the target service went down, and they’ll potentially waste time on the wrong investigation path.

How to avoid uncaught exceptions

Forcibly failing tests is a good practice when first writing them. For example, changing the assertion to “Slugathor2” will present the message above.

💡 Using Test-Driven Development will prevent this every time.

Other ways involve writing the test differently. There isn’t much need for the done callback with promises, but legacy code haunts many developers. Here are a few examples of a better format:

it("should fetch a user", () => {
  // Exceptions will be caught by mocha because it has a surrounding catch block
  return request(app)
    .get("/user/1")
    .send()
    .then((res) => {
      expect(res.body).to.deep.equal({ name: "Slugathor" });
    });
});

it("should fetch a user", async () => {
  // Using await causes exceptions to occur on the same code path
  const res = await request(app).get("/user/1").send();
  expect(res.body).to.deep.equal({ name: "Slugathor" });
});

it("should fetch a user", () => {
  // The end function in supertest handles uncaught exceptions
  request(app)
    .get("/user/1")
    .end((err, res) => {
      if (err) {
        throw err;
      }

      expect(res.body).to.deep.equal({ name: "Slugathor" });
    });
});

Conclusion