Have you ever been in a painter’s house that is in need of paint, or in a carpenter’s house that has unfinished trim, or seen a mechanic’s car in need of repairs? Why do you think this happens? Maybe they are too busy doing work that ‘pays the bills’. Or maybe they don’t want to spend more time doing the things they did all day at work, at home as well. Whatever it is, it seems that we often don’t apply the skills we are best at to the things in our own control.
The same thing happens to testers. We are really good at finding problems with things, and yet we complain. We complain that our automation is full of problems. We are great at finding problems for other people, but we forget to apply our skills to finding the problems that are in our tests. Our tests are broken and we need to do something about it. We need to test our tests!
Here are a few ideas to help you get started:
Flaky tests
I’ll start with the one we all know and complain about: The test that fails but then passes on re-run. We do things like automatically re-run tests so they pass, or comment out some flaky calls, but what if you were trying to test your tests? What could you do? One idea would be to invest in some instrumentation that would help you better record the state of the system or the tests when this happens. Could you find patterns in the timing, situations or steps that cause flakiness? Could you identify tests that fail together and clean up or eliminate them? Can you test your tests?
Green Tests
Now here is one you might not think of as a problem, but if we are testing our tests we need to think with a different mindset. The tests that always pass – Are they helpful? Could they demonstrate problems? I was reviewing some tests a while ago and was looking at a test that had been running for three years and had never failed. When I looked into why, I noticed that the test did not have any assert statements in it. Essentially, the test was only checking for crashes. No wonder it never failed! It wasn’t checking anything. Are your tests deceiving you? Can you test your tests?
Red Tests
Let’s not forget about those tests that fail at the slightest provocation. You make a minor tweak in the product and the test fails. It is easy to get caught up in the hamster wheel of keeping the passing percentage high, but you have to put your tester hat on. Zoom out and look at the big picture! Is there something going on here that you can address? Are there particular kinds of tests that you can clean up so that they are more robust? Are there certain areas of the product that don’t need as many tests to cover (or that you should wait to add automated tests for)? Look for patterns. Tests your tests!
Of course, there are many other patterns you could look for. So put on your tester hat, start brainstorming and I’m sure you will be able to come up with plenty more. Hopefully these are enough thoughts to get you starting on thinking about what it might look like to test your tests.
Please share your thoughts in the comment section below 😎