When I was pursuing a degree in software engineering, I had a class specific to the concepts of software testing. In this class, it was the first time I had encountered the term “code coverage” and “test coverage”, and to be honest, most of us thought of them to be the same.
How could you have tests that cover the complete code but not the complete functionality of the application? Later on, when I started working as a software engineer, I found out that the confusion I had faced in college was still present with most of the employees. And to my surprise, most of the testers too.
Code coverage and test coverage may sound similar, but their underlying concept, the purpose of existence, and implementation are entirely different 🔍 One aims at the “quantitative” aspect of the software code, and the other the “qualitative” aspect of it.
But to understand how they differ from each other, we first have to understand what are they, and why are they important. By defining the reasoning, I am confident that you would be able to differentiate between the two even before I try to do so at the end of this post.
Code Coverage vs. Test Coverage – Table of Contents
- What is code coverage?
- What it test coverage?
- Code coverage vs. test coverage
- Conclusion & personal opinion
What is code coverage?
To understand code coverage, let’s start with a simple piece of code that prints “You are an adult” or “You are a child” based on your age. I am assuming the code for input is already known because it does not affect our code coverage part.
if (age > 18):
print("You are an adult")
else:
print("You are a child")
Now, let’s consider I have a test case that gives the input to the age variable as 21. When it’s time to execute the if block, it runs line by line.
- Line 1: if (age > 18).
- This is true and hence the if block is entered.
- Line 2: print (“You are an adult”).
- This is a simple print statement and hence printed on the console.
Now since the if block was entered, the program exits after the print statement. So the code checked for two lines of code out of four, which means 50% of the code. In simple words, I covered just half of the code in my program, hence my code coverage is 50%.
This is just what the code coverage is – how much code have you covered through your tests. If I am covering 100% of the code, I can say that I have executed each line of the code and it did not give any errors.
So the question that arises here is, if I’m covering 100% of the code, can I say that I have tested (emphasizing the “tested” because it’s important) the code completely and it works fine? Or can I say that the software quality is as expected? 🤔 Let’s hold this thought here and I leave you to ponder over it for a couple more sections.
Benefits of code coverage
Using the code coverage methodology on the source code delivers the following benefits:
✅ How much code you tested
When projects are smaller, code coverage is easier to follow even manually by reading the code. However, as the project starts to take the shape, this becomes much harder, or even manually impossible.
In the code written in the above section, if the else block is never executed, we never know if it will produce the correct output or not. Code coverage removes that uncertainty and we can be sure that even if we don’t know about the functionality, we know that each line of code is executing through our tests.
✅ Bug finder
The previous point states that code coverage can help us determine how many lines of code we are running through our tests. Hence, as a result of that, we can conclude that the more code coverage we have, the more bugs we would find as more lines are being executed.
✅ Ease of development
Code coverage detects continuously how much coverage you are getting in your code. Therefore, if used parallel to the development, the final testing becomes much easier.
Types of code coverage
What we discussed in the introductory section about code coverage is a conditional block to determine the coverage of the code. In testing terms, it is referred to as branch coverage, since a decision-making block becomes a root of a branch, where one branch means the condition returned true and the other one as false.
Similar to this, there are different types of code coverage in software testing. Since all of them aim at different parts of the code, practically, we make use of a combination of them:
1️⃣ Branch coverage
Testing for code coverage when the code is a decision-making statement (if-else, switch, etc.) is called branch coverage. Branch coverage makes sure that every branch statement is executed at least once. The detailed analysis using branch coverage is done in the previous section.
2️⃣ Function coverage
As branch coverage ensures the execution of each condition in a branch, the function coverage ensures that each function is called and executed at least once. The function coverage also ensures that different types of arguments are passed into the function to test the legibility of the code.
3️⃣ Loop coverage
Loop coverage ensures that the code enters a loop. It validates whether the loop is running as it should or not. But, the problem with loops is that it hardly contains any local variable that can be initialized and taken care of in the end, like in a function.
Loop works on variables that it shares with the program, and depending on the variable’s value, it might behave differently. Therefore, a lot of the time you might come across different definitions of loop coverage, i.e. when you can say that you have covered this loop.
Some say that entering the loop once is enough, some define loop coverage as entering it at least once, and some define it as entering more than once. It completely depends on the tester and the project he is testing.
4️⃣ Condition coverage
A testing method similar to branch coverage (also referred to as decision coverage), is the condition coverage. A condition coverage method determines the validation of subexpressions and variables inside the conditional statements. For example, consider the following statement:
if(age > 18 and gender == "Male")
In the above statement, we are concerned about the variable validations “age” and “gender“, and the expression (and) that makes this statement complete. For every instance, you will select one of the four conditions:
- age > 18, gender is Male
- age > 18, gender is Female
- age < 18, gender is Male
- age > 18, gender is Female
A branch coverage differs here as in that case we are not concerned about the variables. If any of the conditions are met and the code goes inside the conditional block, it is enough.
5️⃣ Statement coverage
The statement coverage is very simple. It validates if all the executable statements (i.e. code statements and not comments etc.) in the code are executed at least once or not.
6️⃣ Finite state machine coverage
A finite state machine coverage (also referred to as FSM coverage in short), is the most complex type of code coverage in software testing. To understand FSM coverage, we must know FSM basics.
A finite state machine represents states and paths. A code is always at one of the states given in the FSM at any point in time. When we execute code further ahead from this state, the machine might go to a different state or remain in the same state, depending on the code execution.
So, FSM defines the behaviour of the machine in the most concise and expressive way there is. You can take reference from the following example:
Here the arrows are the paths and the circles are the states. FSM coverage aims at covering the complete FSM and validating the code at each state and path (including reverse paths). If you can cover all of them, we say that we have done 100% FSM coverage. Since FSM defines the machine behaviour at different points, achieving 100% FSM coverage is often the goal of the testers.
Tools to perform code coverage
In this section, we have listed various code coverage tools according to the programming language to help you choose the best one:
🔨 JaCoCo – Java
JaCoCo is a code coverage tool that uses Java programming language to produce the code coverage output which may look as follows:
JaCoCo is triggered by JUnit tests and produces the output file in the target directory. However, the output is in binary format.
🔨 Coverage.py – Python
Coverage.py is a python based code coverage tool that can be installed through the pip install coverage command. The tool when executed on the code tells the tester about the parts of code that were executed, that did not execute, and those that could have been executed but didn’t. Coverage.py uses tracing hooks from Python’s standard library.
🔨 Testwell CTC++
Testwell CTC++ is a C++-based code coverage testing tool that does not come with the installation package. The tool has to be downloaded and installed from a third-party website, but is compatible with all compilers in C++.
The added advantage of Testwell C++ over other code coverage tools is that it tells you about how many times each line of code has been executed instead of just telling whether a line is executed or not.
What does 100% code coverage signify?
Code coverage tells you the code that has been executed in percentage terms. So, a 100% code coverage would mean that all the code lines are executed by our designed test cases 💯 But practically, what does that signify? Does that mean we have the highest quality of code and the most rigorous test cases?
A lot of the time you would find 100% code coverage defined as an achievement on various resources on the Internet. These may also ask you to always aim for 100% code coverage. But I think 100% code coverage might give you a false sense of security about your tests as well as your product.
It may fire back later when the end-user uses the application. A good example I read in a book that can prove our assumption with contradiction goes as follows- Let’s say we have a function:
def myFunc(num1, num2):
return num1 * num2
And a test function:
def test():
assert myFunc(1,1) == 1
The test achieves 100% coverage above. Now let’s say the function’s operation is changed to division instead of multiplication inadvertently. This test would still pass in that case. Hence, 100% code coverage has no relation to the code quality or software quality. Relying on this could mean passing on bugs to production that are fatal for the software.
With that being said, code coverage does help in determining if each line of code executes, and if there are any bugs in the branches or syntax etc., they can be easily pointed out.
Therefore, it is alright to keep the number higher (80% maybe), but writing test cases just for achieving 100% code coverage should be avoided. If you get 100% code coverage anyway, it is well and good for all.
What is test coverage?
The code coverage part helps you analyze if each code in the application is being executed by our tests or not. Test coverage is a bit different. The test coverage defines whether our tests are covering the functionality of the application or not.
For example, if a branch statement is one functionality, you don’t need to count on each statement here. If you are able to hit the branch, you have tested the functionality. Test coverage, similar to code coverage, is also measured in percentage terms.
So, if you have 100 test cases and 90 of them got executed while testing the application functionality, it means we get 90% test coverage. If you are looking to increase the number, you just have to look at the functionality that you missed apparently and write test cases for it.
Here, we conclude that 90% code coverage and 90% test coverage can mean totally different things in software testing. Hence, using them interchangeably is a bad practice that can lead to misunderstanding regarding the quality of the software.
Benefits of test coverage
Using test coverage in the testing phase of SDLC provides the following benefits:
✅ Fewer defects
Test coverage can determine the areas in the application that are not working properly. The greater the test coverage, the better chances of finding the hidden defects in the application are (given that there are no repetitive or inefficient cases). As a result, very very few bugs seep into the production.
✅ Help ease out the development
Using test coverage can help in faster development as the developers are able to identify the part of code that is not working at an earlier stage. This can help to avoid piling up of bugs later, that will cost extra time and money to the organization.
✅ High return on investment
The previous point states how test coverage can help save time and costs at later stages of release. As a result of this, we will find our investment returns to be much faster, which benefits the complete organization.
Apart from these, you may also find test coverage benefitting your test report, test cycles (and time to test the app), and other criteria such as exit criteria.
Types of test coverage – techniques
By now you must have got the gist of test coverage and its importance in software testing. But just defining test coverage as a method of verifying the product’s functionality is a very wide term.
Product functionality may constitute a variety of things such as functional requirements, risk minimization, etc.. So, we need to divide test coverage accordingly to let the team know what our focus was during the testing:
1️⃣ Requirement coverage
The first and most important type of test coverage is requirement coverage. Requirement coverage aims at covering all the user requirements, which are brought by the client or planned by the team.
A failure in testing requirement coverage can mean that either you are pushing a faulty application, or that some requirement may never have been implemented at all. This could be devastating to the reputation if the end-user finds the application faulty or cannot find the promised feature.
2️⃣ Product coverage
Product coverage is another type of test coverage that works on all the complications of the product and verifies if it can handle complex scenarios. For example, testing the behavior of the application in various situations (such as various network conditions), or testing if the product meets its end goal or not.
3️⃣ Boundary value coverage
As the name suggests, boundary value coverage is similar to boundary value analysis in manual/automation testing. Boundary value coverage analyzes the boundary values of the application such as in the input fields.
For example, if you have an input field that takes only positive values, testing it for negative values and very large positive values would explore the boundary situations.
It is always considered a best practice to use this type of test coverage mixed with other ones. This is because if your application starts accepting values that it shouldn’t, the consequences could propagate to other parts of the system such as the database, which could be really costly.
4️⃣ Risk coverage
The risk coverage focuses on the things that could be a risk to the system or the application, and hence to the user. While all other types of test coverage can list down a few common things, risk coverage depends completely on the type of application you are building.
For a very simple tic-tac-toe app, you probably would not need to go through this. But, if your application concerns the personal data of the user, risk coverage becomes very important. For example, for a banking application, it would be a risk to the system if you logged out on one tab, but others can still open your information.
Tools to perform test coverage
Similar to code coverage, we need expert tools to perform test coverage. But they are not separate tools as we looked upon in the code coverage section. Test coverage is about executing tests aimed at specific goals.
So if your goals are risk coverage, a risk analysis tool would work. If your goals are functionality-oriented, unit testing tools might be the best choice. Once your test coverage type is decided, just choose the appropriate tool in your comfortable language. Some of the tool’s names are given below:
🔨 JUnit– For Java-loving people.
🔨 PyUnit– For Python-loving people.
🔨 PerlUnit– For Perl-loving people.
🔨 TestUnit– For Ruby-loving people.
And there are more of such tools for other languages. The bottom line is, through these tools you can predict your test coverage (by checking the output of how many tests were executed) and combine it with code coverage for a perfect analysis.
Code coverage vs. test coverage
| Point of consideration | Code coverage | Test coverage |
| What is it? | Code coverage is the measure of covering your code lines through test cases | Test coverage is the measure of covering your application functionality and behavior through test cases |
| How is it performed? | White-box: on the source code | Black-box: run directly on the application |
| Type | Branch, function, loop, condition, statement, FSM | Requirement, product, boundary value, risk |
| Tools | JaCoCo, Testwell CTC++, Coverage.py etc. | JUnit, PyUnit, TestUnit etc. |
| 100% coverage | Should not be aimed and is not necessary | Can be aimed |
Conclusion & personal opinion
Test coverage and code coverage are two terms that often confuse people. While they don’t have anything in common as such, I believe the only confusing aspect here is their names and working mechanisms. Both are executed on the application, and both require test cases to run.
However, code coverage does not necessarily aim at the product quality or the application functionality in any way. It just aims at finding bugs in the software lines. Test coverage is a bit different and focuses purely on the application’s behavior and finding defects.
So, it is not a choice of which one to choose, but rather how much time should you give to each of them. Both would contribute in their own way to the software quality. If you just perform test coverage on a branch and not the code coverage for example, you might miss the bug in another part that could affect the functionality.
With that being said, it is also important to note that 100% code coverage should not be aimed in any phase of SDLC or release cycles. It would just waste the time and resources, and we may miss out on important testing parts when short on time.
I hope this post enlightened you about code coverage and test coverage in the simplest way possible.
Happy testing 💢