logo logo

Trick-or-Treat, Speed, & Coverage

Trick-or-Treat, Speed, & Coverage

It was Halloween 1995, and I was in my last year of elementary school. At the beginning of the day, my friend and I had a “brilliant” idea to maximize our trick-or-treat returns 🍬🍭 We decided that walking from house to house was a fool’s errand. Instead, we would take our bicycles and cover much more ground in the process. You can imagine what happened next…

When school let out, we hopped on our bikes and headed to the nearest residential neighborhood 🚴‍♂️ It only took a few seconds for my friend to say, “Oh, we still have to get off our bikes and knock at each door. This doesn’t really speed things up, does it? We’ll actually end up covering less ground.” Following this recognition, we dropped off our Roadmasters and went back to the “old way” of trick-or-treating 🎃😄

I have had similar experiences in test automation over the years. A “brilliant idea” can quickly become a fool’s errand, which can cause frustration and lost productivity.

For example, I was recently given the task of automating several workflows for a medical communication app. One of these workflows included searching and filtering a user directory. Using TestProject, I quickly built a few tests that I subsequently included in my regression suite.

Part of what helped me avoid false failures at first was the inclusion of a few explicit waits (pauses) between steps. Then, at some point along the way, I had a “brilliant idea” to increase the speed of my tests. I decided that I would remove the pauses, not accounting for fluctuating server speeds and other variables. Very quickly, my tests began failing, and I realized that I had been too ambitious.

Testing is a constant balancing act between speed and coverage. Automation vastly increases the rate at which we can execute tests, but there are times when the pursuit of rapidity can undermine how much code is actually executed.

In the case above, I sacrificed coverage for speed, and I paid for it with many false test failures. Thankfully, this was a minor problem that could be easily amended. However, there are times when dead-end shortcuts require time-consuming work to fix, especially when using “coded” automation.

Still, we all need the freedom to try a “brilliant idea” every now and then. I love TestProject’s Test Recorder because it allows me to experiment with my ideas without committing to a major refactoring process. I have the liberty to adjust wait times, remove steps, use alternative locators, designate a different element action, and much more. If my idea turns out to actually be brilliant (increasing speed without sacrificing coverage), then it’s a win for everyone. Yet, if the idea is a dud, I haven’t lost too much time chasing a wild goose 🦆

One other thing. It’s interesting how different environmental conditions can transform bad ideas into good ideas. This year, in order to maintain social distancing, many neighborhoods are asking families to place tables/stations with pre-sealed treats closer to the street. Now, the bicycle idea really might be more efficient!

Likewise, with recent improvements in the web framework of my app under test and the inclusion of an Adaptive Wait feature in TestProject, I have been able to remove those explicit pauses without any subsequent false failures. This has opened the door for me to cover much more ground in the same amount of time or less.

So, my advice to you? Hold on to those “brilliant ideas.” You never know when they might come in handy 😉

Happy Halloween! 👻🦇🎃

About the author

Beau Brown

Hey, I’m Beau Brown. I wear a lot of hats. I’m a software tester, Presbyterian Minister [PC(USA)], hip-hop artist, writer, and teacher. At heart, I’m a learner (thus all the vocations). I enjoy speaking and writing on a variety of issues, including mental health, religion, race, hip-hop, education, adoption, and technology.

Leave a Reply