Open Source Image Comparison Addon for your Test Automation

During automation, UI elements in the application can change their appearance. It can be a change of color, size or even its presence or visibility. Validating those changes during manual testing is relatively easy, whereas validating them during automation is not such a straightforward task.

The new open source Image Comparison Addon by TestProject will help you to achieve just that. Let’s get started! 

Table of content

  1. Prerequisites
  2. Resources
  3. Compare images with UI elements (Example A)
    1. Step #1: Take a screenshot of the element in its expected state
    2. Step #2: Compare the element’s current state with the saved screenshot
  4. Compare two locally stored images (Example B)
  5. Summary

Prerequisites

  1. Sign up to TestProject here. Don’t worry, it’s completely free!
  2. Login to TestProject and install the TestProject Agent (it incorporates all of the tools and drivers you need for your test automation, in one small desktop local component). Watch this short video to see how easily you can setup your Agent.
  3. Install the Image Comparison Addon from here.

Resources

  1. The source code of the TestProject Image Comparison addon is hosted in GitHub: https://github.com/testproject-io/addons/tree/master/image-comparison
  2. The demo site we used for this tutorial: https://testproject-io.github.io/addons/image-comparison-demo-site/.

 

Compare images with UI elements (Example A)

In this example, we will work with a simple website that contains 6 clickable squares. Clicking on any of these squares will immediately change their colors to blue. To validate this functionality, we will use the Image Comparison addon!

Step #1: Take a screenshot of the element in its expected state

First, we will take a screenshot of the ‘unordered list’ element that contains the squares at its final expected state (when all the squares are blue). We will use the element action called “Take element screenshot” for saving the screenshot. This element action takes two input parameters:

  1. Path: This is a local path on your computer where you want to save the screenshot.
  2. ImgName: The name you want to give the screenshot (if you leave it blank, the name will be: ‘screenshot’ and will include the current timestamp).

This element action also returns one output parameter:
ResultFullPath: This output parameter will be assigned with the full path to the saved screenshot in your computer (e.g. “D:\Documents\screenshots\expectedState.jpg”).

The video below is an example of using the element action called “Take element screenshot”:

This is our “expectedState” screenshot that was taken (as described above):

Image comparison addon_expected state

Step #2: Compare the element’s current state with the saved screenshot

Now that we have a screenshot of the expected state of the element, we can test its functionality (change of color). In order to get a different “current state”, let’s click on any of those six squares and compare the new current state of the element with the saved screenshot from the previous step (described above in step #1 as “final state”). This time, we will use the element action called “Compare image with UI element”. This element action takes four input parameters

  1. ImagePath: A local path to the saved screenshot from step #1 (e.g. “D:\Documents\screenshots\expectedState.jpg”).
  2. Threshold: The maximal value of the difference between the images (in percentage) that you want to have. For example, 0% means that you are not willing to accept any kind of difference, whereas 15% means that you are willing to accept up to 15% of difference. 
  3. ResultImagePath: This is a local path on your computer where you want to save the screenshot.
  4. ImgName: The name you want to give the image (if you leave it blank, the name will be: ‘comparison_result’ and will include the current timestamp).

This element action also returns one output parameter:
FullResPath: This output parameter will be assigned with the full path to the saved difference image in your computer (e.g. “D:\Documents\screenshots\actualState.jpg”).

The video below is an example of using the element action called Compare image with UI element”: 

Now, let’s check the output of the exact same scenario, but this time – we will fail the test.
In this case, we will compare the initial state (where all six squares have different colors) against the final state (where all six squares have the same blue color) and demonstrate the difference:

This is our “initialState” screenshot that was taken (as described above):

Image comparison addon_initial state

This is our “finalState” screenshot that was taken (as described above). In red you can see the highlighted areas that demonstrate where there are differences between the two:

Image comparison addon_final state

Compare two locally stored images (Example B)

In this example, we will also be working on the same colored squares website.
This time, we will use the action called “Compare two locally stored images“. This action takes two locally stored images and compares them, as opposed to the element action “Compare image with UI element” (that we used in Example A above) that compares an element with another image.

The action “Compare two locally stored images” takes five input parameters:

  1. ImgA: The local path to the first image (e.g. “D:\Documents\screenshots\imageA.jpg”).
  2. ImgB: The local path to the second image (e.g. “D:\Documents\screenshots\imageB.jpg”).
  3. Threshold: The maximal value of the difference between the images (in percentage) that you want to have. For example, 0% means that you are not willing to accept any kind of difference, whereas 15% means that you are willing to accept up to 15% of difference.
  4. ResultImagePath: This is a local path on your computer where you want to save the comparison result image.
  5. ImgName: The name you want to give the image (if you leave it blank, the name will be: ‘comparison_result’ and will include the current timestamp).

This action returns one output parameter:
FullResPath: This output parameter will be assigned with the full path to the saved comparison result image in your computer (e.g. “D:\Documents\screenshots\ComparisonResult.jpg”).

So, in this example, we would like to visually validate that the “RESET” button on our website does indeed reset the color of the squares and changes their colors. We will take two screenshots of the ‘unordered list’ element: Before clicking the “RESET” button and after clicking on it. Then, using the “Compare two locally stored images”, we will validate that they are not identical:

The comparison step failed since we defined the threshold to be zero, in order to be sure that the two images are different. The difference in percentages in this example is 27.34%. In the screenshots below you can see the two compared images along with the comparison result image that highlights the differences in red:

Before clickng the “RESET” button:

Image comparison addon_before

After clickng the “RESET” button:

Image comparison addon_after

Comparison result image:

Image comparison addon_compare results

Summary

In this tutorial, we created a real example of using image comparison in test automation. Our challenge was to validate the change of the colors in the square elements. Without image comparison, this task would have been difficult. To simplify your effort, we used the Image Comparison addon by TestProject and learned how to use it in your automated tests.

Now it’s your turn! Go ahead and use this great addon in your own test cases and share your experience in the comments below!  💡 

 

TestProject Test Automation Tool