Spot the Difference with Visual Regression Testing

We all know the importance of automation testing during the product life cycle. In this article, we will discuss special automated test cases that help optimize the testing process known as Visual Regression Testing.

Visual Regression Testing allows us to make visual comparisons between the correct (baseline) versions of the site with new versions. It’s a series of tests you create to run through your UI while taking screenshots, that alerts you when the original and current versions have differences. It is an alternative approach to testing web pages. Instead of just making sure that some element or text value is present, the test actually opens the page and checks to see if that specific block appears exactly as you want it to appear.

Visual Regression Testing Benefits
Visual Regression Testing Workflow
What is BackstopJS?
Why BackstopJS?
Getting Started with BackstopJS
Summarizing Guide for Creating and Maintaining Visual Regression Tests using BackstopJS

Visual Regression Testing Benefits

Reference(Prod) vs Test(Test env) Comparison
Support for multiple viewports
Easily scan the entire site
Helpful error reporting for UI issues
Easy to set up and run the tests
Lower workload for testers
Helps to overcome “Change Blindness”

The main benefit is that is simply helps to save a lot of time! ⏳

Visual Regression Testing Workflow

Reference: Take & store reference/baseline screenshots of the website.
Tests: After making changes, run tests that take new screenshots to compare against reference screenshots.
Approve: If tests passed, approve the test version which will be stored as a new reference.

There are many tools to perform visual regression testing, and even TestProject has a cool Addon for image comparison with a step by step tutorial here. However, in this article I will demonstrate another simple and quick way to perform visual tests, using BackstopJS.

What is BackstopJS?

BackstopJS is a framework written in JavaScript that helps to automate visual regression tests easily and fast, and provides easy-to-configure test parameters for a variety of viewport sizes and pass/fail conditions.

BackstopJS is a combination of the following components:

Chrome Headless: Headless browser
- It is useful for rendering a DOM, styling elements, doing AJAX requests via command line.

ChromyJS/Puppeteer: Simulate user behaviour
- Scripting tool which helps to interact with the page rendered by PhantomJS. Eg: User actions like entering texts, clicks, etc.

ResembleJS: Spots the difference
- The “diffing” tool. Comparing two images and determines the difference in pixel between them.

Why BackstopJS

Ease of use & configuration
Reliability
Integration with JS task runners and CI systems
Detailed browser reports with visual diffs

Let’s see it in action… 👉

Getting Started with BackstopJS

How BackstopJS works

During initial execution, BackstopJS will take screenshots of the application and those will be stored as baseline images. Whenever you change something on the website and run BackstopJS again, it takes a new image and compares it with the original image (baseline). It is easy to install and configure for testing at various URLs and different screen sizes.

If there are no differences between the images, the test will return “PASS”, else, the test will return “FAIL” and a new image will be generated, showing the difference between the two images compared.

How to install BackstopJS

BackstopJS can either be installed globally or locally in your project. To install, run this command:

Note: I recommend the global installation of BackstopJS on your machine, which will make testing easier.

$ npm install -g backstopjs

How to configure BackstopJS

Create a test project: Download the project with some sample content that I attached hereby, and unzip it. The folder structure will be as follows:

You can create a default configuration file in your current working directory. Run this command in your project’s directory:

$ backstop init

This command will create a folder called backstop_data and will generate a backstop.json file. This json file will act as the config for our visual regression tests. The project structure after init will be as follows:

The default backstop.json configuration file created will be:

{
   "id":"backstop_default",
   "viewports":[
      {
         "label":"phone",
         "width":320,
         "height":480
      },
      {
         "label":"tablet",
         "width":1024,
         "height":768
      }
   ],
   "onBeforeScript":"puppet/onBefore.js",
   "onReadyScript":"puppet/onReady.js",
   "scenarios":[
      {
         "label":"BackstopJS Homepage",
         "cookiePath":"backstop_data/engine_scripts/cookies.json",
         "url":"https://garris.github.io/BackstopJS/",
         "referenceUrl":"",
         "readyEvent":"",
         "readySelector":"",
         "delay":0,
         "hideSelectors":[

         ],
         "removeSelectors":[

         ],
         "hoverSelector":"",
         "clickSelector":"",
         "postInteractionWait":0,
         "selectors":[

         ],
         "selectorExpansion":true,
         "expect":0,
         "misMatchThreshold":0.1,
         "requireSameDimensions":true
      }
   ],
   "paths":{
      "bitmaps_reference":"backstop_data/bitmaps_reference",
      "bitmaps_test":"backstop_data/bitmaps_test",
      "engine_scripts":"backstop_data/engine_scripts",
      "html_report":"backstop_data/html_report",
      "ci_report":"backstop_data/ci_report"
   },
   "report":[
      "browser"
   ],
   "engine":"puppeteer",
   "engineOptions":{
      "args":[
         "--no-sandbox"
      ]
   },
   "asyncCaptureLimit":5,
   "asyncCompareLimit":50,
   "debug":false,
   "debugWindow":false
}

Configuring backstop.json

Open the backstop.json file in your preferred editor. Replace the following definitions in viewports, since we will be running the tests in 3 different viewports:

"viewports":[
   {
      "label":"Phone",
      "width":960,
      "height":480
   },
   {
      "label":"Tablet",
      "width":660,
      "height":480
   },
   {
      "label":"Desktop",
      "width":1024,
      "height":768
   }
]

Next, we’ll configure our test scenarios. Edit the following parameters in the “scenarios” tag:

“label”: Description for our test (Visual Regression Test Demo)
“url”: Destination “URL” (For our test we will use index.html)
“referenceUrl”: Reference “URL” (For our test we will be using index.html).
“selectors”: Replace the default [] to [“html”]
which sets backstop to take a screenshot of the entire HTML. Same we can specify with an array of selectors to capture screenshot only for that elements.

A full list of commands & explanations can be found in the official documentation.

After the updates, our scenario section will be as follows:

"scenarios":[
   {
      "label":"Visual Regression Test Demo",
      "cookiePath":"backstop_data/engine_scripts/cookies.json",
      "url":"/dist/index.html",
      "referenceUrl":"/dist/index.html",
      "readyEvent":"",
      "readySelector":"",
      "delay":0,
      "hideSelectors":[

      ],
      "removeSelectors":[

      ],
      "hoverSelector":"",
      "clickSelector":"",
      "postInteractionWait":0,
      "selectors":[
         "html"
      ],
      "selectorExpansion":true,
      "expect":0,
      "misMatchThreshold":0.1,
      "requireSameDimensions":true
   }
]

How to work with BackstopJS

After finishing the basic settings of the tests, let’s look at the commands for working with BackstopJS.

Run the following command to take reference screenshots which we will use to compare while running the tests:

$ backstop reference

It creates a series of baseline/reference images for future tests in the “backstop_data/bitmaps_reference/ <timestamp>/” folder. The folder structure before and after backstop reference is as follows:

❗Note: You could also run tests without building any reference files, and after the failure, use the test result as references with approving the command which I will be explaining later.

Since we have all our reference screenshots, we can start tests using this command:

$ backstop test

It takes a new set of screenshots and compares them with reference set, then opens a report. New images will be generated in “backstop_data/bitmaps_test/<timestamp>/ ” folder and a report comparing the most recent reference screenshots and the current ones will be generated in “backstop_data/html_report” folder.

If all test cases are passed, you can update the reference screenshots with this command:

$ backstop approve

It sets the last screenshots as the new reference, accepting them as the new canonical version to test against next time.

BackstopJS Reports

After we run the tests, BackstopJS creates HTML reports in the “backstop_data / html_report / index.html” folder which will show a detailed comparison of the images.

The reports will be as follows:

Now, let’s spot the difference after changing something in the page.

Let’s make the following changes to the “dist/style.css” file:

Change the alignment of heading “Visual Regression Demo Test Screen” to left by changing text-align: left; in the .top class.
Change the alignment of heading “Grid section” in left to center by changing text-align: center; within the #row2> h2.
Change the alignment of the footer text “Nithin” to right by changing text-align: right; to the .footer class.
Update the footer image in the index.html file to https://imgur.com/0uZBDW5.jpg which has the logo color change from aqua to pink.

Let’s run the test again to see how it spots the difference by running the following command:

$ backstop test

This time the test will be FAILED!!!

We can check the differences from the report generated:

The report will have a “Scrubber” option which helps us to recognize the changes in a comparison mode.

Summarizing Guide for Creating and Maintaining Visual Regression Tests using BackstopJS

Correct implementation of visual tests can increase the quality of your product by reducing UI bugs. Visual Automated tests help to detect even the tiniest difference in the application’s UI. In addition, it helps to make our work more efficient. With visual tests in place, QA Engineers just need to verify the functional aspect rather than spending time to identify UI issues.

This is just a basic introduction to Visual regression testing, using BackstopJS. I invite you all to spot the differences in an automated way with BackstopJS 😎
Happy Learning!