“Have you handled exceptions in your test automation code? What kind of information is the exception handling mechanism providing to you? Is that information useful to the people who are analyzing the test execution results?”
A few years back, in a team meeting, those were the questions asked by one of our colleagues who joined our team from a different team. The person was quite experienced in Test Automation and had also worked in product development for several years. The test automation framework that we were developing at that time was still in its nascent stages and over the years it matured a lot. But dealing with exceptions efficiently was something that we all agreed to take care of diligently from the start while writing code for the framework. Thus, at that time, we could explain to him the exception handling mechanism that we had in place and how it was providing value.
Over the years, I have seen a lot of test automation code – in the public domain (like GitHub, blogs), in projects, in tutorials, workshops, and from people. Most of the time, exception handling was not used or shown. In places where it was there, it was not implemented efficiently, and more often than not, it was just used for the sake of it or to just bypass some condition to continue the execution flow.
Table of contents
- Handle Exceptions responsibly and don’t put Exception-Handling blocks on everything
- Handle Checked Exceptions by re-throwing them as runtime instances
- Use Custom Exceptions to provide useful business-friendly messages
- Don’t forget to close connections or clean up resources
- Don’t just catch and ignore the caught exception
- Try to be as specific as possible while catching the Exception
- Don’t catch Assertion Errors
- Conclusion
Can you relate and recall the number of places where you have seen this?
Catch the exception and just print the stack trace…?
try {
// some code…
} catch (Exception e) {
e.printStackTrace();
}
Or catch the exception and just ignore it…?
try {
// some code…
} catch (Exception e) {
// do nothing…
}
Handling unexpected events (exceptions) efficiently is an important skill that a test automation developer must have. Exceptions may occur during the compilation of the automation code or during runtime when the automation code lines are getting executed. Exceptional events during runtime may take place due to the use of wrong logic/computation in the code, due to unexpected inputs being driven into the code, due to the automation libraries not able to perform or complete some desired action (e.g. locating a web element) or it can be due to some other software/hardware issues.
These events disrupt the normal flow of instructions in the tests and if they are not handled properly (failed gracefully or a recovery mechanism is put in place), then it may lead to unexpected issues at unexpected points in the framework. This will make it difficult for the developers to analyze test execution results, or to debug and fix issues. Furthermore, it may also result in false positive and false negative execution results. When exceptional events happen, most of the programming languages that we use to write our test automation code create objects which contain information about the exception, including its type, cause, and the state when the event occurred. The exception is then thrown to the runtime system which then looks for some code (exception handler) to handle it. If no exception handler is found, then the execution prints the call stack up to the code where the event took place and terminates.
In this article, I will go through some of the good practices that you can keep in mind while writing code to handle exceptions in your framework. I have considered Java while writing these points, but most of them apply to other programming languages too.
1) Handle Exceptions responsibly and don’t put Exception-Handling blocks on everything
Many times I have seen people putting exception-handling blocks (e.g. try-catch in Java/JavaScript or try-except in Python) to everything under all the functions because they were told to handle exceptions. Use exception-handling blocks in parts of the test automation code where you can take some action or process an exception if it is thrown. The actions can comprise – wrapping the raised exception to another exception and throw the latter, logging the exception in some logging framework or utility for further analysis, jumping to a different part of the code to perform some recovery operation, and so on.
If you think catching the exception for that part of the code will help you or the team in some way, then catch it. Otherwise, let it bubble up to the level where it can be dealt with by the framework itself. Also, don’t log an exception and then re-throw the same exception from the catch block. If this is done, then chances are the error message/stack-trace for the same exception will appear multiple times in the logs, and debugging will become difficult.
2) Handle Checked Exceptions by re-throwing them as runtime instances
Java has the concept of “Checked Exception” where the exception is checked at compile-time and needs to be either caught (using try-catch) or declared (using “throws” clause with the method signature) in the method in which it is thrown. When a method calls another method and the invoked method throws checked exceptions, the calling method doesn’t need to know about or deal with it or let it propagate as “checked” throughout the call hierarchy (which will result in multiple “throws” clause declaration throughout the framework).
Instead, you can catch the specific checked exceptions inside the catch blocks, wrap them with new RuntimeException instances, throw them and let the calling methods decide how they want to deal with those runtime instances.
public static void main(String[] args) {
File file = new File("EnvironmentConfig.properties");
FileInputStream fileInputStream = null;
try {
fileInputStream = new FileInputStream(file);
} catch (FileNotFoundException e) {
throw new RuntimeException(e);
}
Properties prop = new Properties();
try {
prop.load(fileInputStream);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
3) Use Custom Exceptions to provide useful business-friendly messages
Handling standard exceptions efficiently can provide us useful information about the caught exceptions, their type, cause, and the stack traces. But there will be situations where you would want to customize the exception handling mechanism in such a way that whenever an exception is thrown, it is wrapped into a “Custom Exception” and a framework-specific, user or business-friendly message will get generated and logged along with the original exception type, cause, and stack trace.
You can create multiple Custom Exception classes at the framework level (as subclasses of the RuntimeException) and you can use them wherever you would want to provide some useful customized message. These custom exceptions can also provide the flexibility to add attributes, store additional information like error codes, and provide utility methods that can be used to handle or present the exception.
public class FrameworkException extends RuntimeException {
private Integer errorCode;
public FrameworkException () {
}
public FrameworkException(String message) {
super(message);
}
public FrameworkException(String message, Throwable cause) {
super(message, cause);
}
public FrameworkException(Throwable cause) {
super(cause);
}
public FrameworkException(String message, Throwable cause, ErrorCodes errorCode) {
super(message, cause);
this.errorCode = errorCode.getCode();
}
public FrameworkException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) {
super(message, cause, enableSuppression, writableStackTrace);
}
}
enum ErrorCodes {
VALIDATION_PARSE_ERROR(123);
private int code;
ErrorCodes(int code) {
this.code = code;
}
public int getCode() {
return code;
}
}
public void wrapWithCustomException(){
try{
// some code…
}catch(NoSuchElementException e){
throw new FrameworkException(“A business-friendly message describing the caught NoSuchElementException: ” , e);
}
}
4) Don’t forget to close connections or clean up resources
If you are dealing with resources (e.g. with FileInputStream) or connections (e.g. JDBC connection), ensure that they are closed at the end of their processing (irrespective of whether the processing is successful or not). Many times I have seen that the resources or connections are opened inside the “try” block and are closed at the end of the same “try” block. This works fine if no exception is thrown inside the try block. But if an exception is thrown before the close statements are reached, then the opened resources and connections will not get closed.
To ensure that all the opened resources and connections are closed and are not dependent on whether an exception is thrown or not, place the cleanup code performing the “close” actions inside the “finally” block (which will always get executed). Also, don’t put them in the “catch” blocks because then you have to re-write the same closing statements. Another way of dealing with this automatically in Java is to use “try-with-resources” statement (for resources that implement the “AutoCloseable” interface) instead of using “try-catch-finally”.
try{
// connections or resources opened
}catch(){
// some code…
} finally{
// connections or resources closed
}
5) Don’t just catch and ignore the caught exception
I have also seen code snippets where the try-catch blocks were used and inside the catch blocks, nothing was done. So if any exception is thrown, then the “catch” block will just catch and ignore it. This is a definite anti-pattern and doing this can affect execution in various ways. If this anti-pattern is used and any raised exception causes some unexpected behavior, then the developer will have a very difficult time finding the root cause for that unexpected behavior. Inside the “catch” blocks, at least a single log message should be written which will inform that something unexpected has happened or the default stack trace should be logged.
public void ignoreException(){
try{
// some code…
}catch(NoSuchElementException e){
log.error(“Something unexpected has happened”);
}
}
6) Try to be as specific as possible while catching the Exception
Exceptions have pre-defined classes and those classes have their superclasses. While creating exception handling code, if the exception handling mechanism is designed in such a way that it handles at the superclass level, then it may turn out difficult to trace the exact exception which caused the unexpected behavior. Thus, while catching exceptions, it is preferred to be as specific as possible with the exception for which the exception handling code is written.
For example, don’t catch “Throwable” or “Exception” when the exception which you are looking to handle is “NoSuchElementException”. “Throwable” is the superclass of all exceptions and errors and if used in the catch clause, it will catch all exceptions and errors which may result in hiding some serious problems.
Don’t use:
try{
// some code
}catch(Throwable e){
// some code
}
Be specific and use:
try{
// some code
}catch(NoSuchElementException e){
// some code
}
7) Don’t catch Assertion Errors
For test automation, this is a serious mistake and we should never be doing it. We use assertions for validations and if any validation fails, the framework should expect the assertions to throw and log Assertion Errors. I have seen instances where code has been written in such a way that if an assertion fails, the catch block catches the Assertion error (at the “Error” level) and does nothing.
Sometimes, these things are implemented intentionally to continue the execution flow, make the tests “Pass” and the execution report “Green”. The objective of automated execution is to provide feedback on the state of the application-under-test and not to just generate “Green” execution reports. Though you can use try-catch blocks to catch Assertion errors, don’t do it and let the assertions take place and log their results, whether passed or failed, with cause and messages.
Conclusion
Those are some of the good practices for Exception Handling that you can implement in your test automation framework and make the mechanism more efficient by avoiding the anti-patterns.
Sometimes, Exception Handling is criticized heavily because if not implemented correctly they can hide the assertion errors or any other important test automation event, can create unexpected test execution exits from unexpected points, or behave like the “GOTOs” which can make the automation code logic very complex (Spaghetti code). But if implemented correctly, exception handling can turn out to be highly beneficial for people.