logo logo

Spot the Difference with Visual Regression Testing

main post image

We all know the importance of automation testing during the product life cycle. In this article, we will discuss special automated test cases that help optimize the testing process known as Visual Regression Testing.

Visual Regression Testing allows us to make visual comparisons between the correct (baseline) versions of the site with new versions. It’s a series of tests you create to run through your UI while taking screenshots, that alerts you when the original and current versions have differences. It is an alternative approach to testing web pages. Instead of just making sure that some element or text value is present, the test actually opens the page and checks to see if that specific block appears exactly as you want it to appear.

Table of Contents

  1. Visual Regression Testing Benefits
  2. Visual Regression Testing Workflow
  3. What is BackstopJS?
  4. Why BackstopJS?
  5. Getting Started with BackstopJS
  6. Summarizing Guide for Creating and Maintaining Visual Regression Tests using BackstopJS

Visual Regression Testing Benefits

  • Reference(Prod) vs Test(Test env) Comparison
  • Support for multiple viewports
  • Easily scan the entire site
  • Helpful error reporting for UI issues
  • Easy to set up and run the tests
  • Lower workload for testers
  • Helps to overcome “Change Blindness

The main benefit is that is simply helps to save a lot of time! ⏳

Visual Regression Testing Workflow

  • Reference: Take & store reference/baseline screenshots of the website.
  • Tests: After making changes, run tests that take new screenshots to compare against reference screenshots.
  • Approve: If tests passed, approve the test version which will be stored as a new reference.

Visual Regression Testing Workflow

There are many tools to perform visual regression testing, and even TestProject has a cool Addon for image comparison with a step by step tutorial here. However, in this article I will demonstrate another simple and quick way to perform visual tests, using BackstopJS.

What is BackstopJS?

BackstopJS is a framework written in JavaScript that helps to automate visual regression tests easily and fast, and provides easy-to-configure test parameters for a variety of viewport sizes and pass/fail conditions.

BackstopJS is a combination of the following components:

  • Chrome Headless: Headless browser
    • It is useful for rendering a DOM, styling elements, doing AJAX requests via command line.
  • ChromyJS/Puppeteer: Simulate user behaviour
    • Scripting tool which helps to interact with the page rendered by PhantomJS. Eg: User actions like entering texts, clicks, etc.
  • ResembleJS: Spots the difference
    • The “diffing” tool. Comparing two images and determines the difference in pixel between them.

Why BackstopJS

  • Ease of use & configuration
  • Reliability
  • Integration with JS task runners and CI systems
  • Detailed browser reports with visual diffs

Let’s see it in action… 👉

Getting Started with BackstopJS

How BackstopJS works

During initial execution, BackstopJS will take screenshots of the application and those will be stored as baseline images. Whenever you change something on the website and run BackstopJS again, it takes a new image and compares it with the original image (baseline). It is easy to install and configure for testing at various URLs and different screen sizes.

If there are no differences between the images, the test will return PASS, else, the test will return FAILand a new image will be generated, showing the difference between the two images compared.

How to install BackstopJS

BackstopJS can either be installed globally or locally in your project. To install, run this command:

Note: I recommend the global installation of BackstopJS on your machine, which will make testing easier.

$ npm install -g backstopjs

How to configure BackstopJS

Create a test project: Download the project with some sample content that I attached hereby, and unzip it. The folder structure will be as follows:

Project Folder

You can create a default configuration file in your current working directory. Run this command in your project’s directory:

$ backstop init

init backstopJS

This command will create a folder called backstop_data and will generate a backstop.json file. This json file will act as the config for our visual regression tests. The project structure after init will be as follows:

Folder structure after backstop init

The default backstop.json configuration file created will be:

{
   "id":"backstop_default",
   "viewports":[
      {
         "label":"phone",
         "width":320,
         "height":480
      },
      {
         "label":"tablet",
         "width":1024,
         "height":768
      }
   ],
   "onBeforeScript":"puppet/onBefore.js",
   "onReadyScript":"puppet/onReady.js",
   "scenarios":[
      {
         "label":"BackstopJS Homepage",
         "cookiePath":"backstop_data/engine_scripts/cookies.json",
         "url":"https://garris.github.io/BackstopJS/",
         "referenceUrl":"",
         "readyEvent":"",
         "readySelector":"",
         "delay":0,
         "hideSelectors":[

         ],
         "removeSelectors":[

         ],
         "hoverSelector":"",
         "clickSelector":"",
         "postInteractionWait":0,
         "selectors":[

         ],
         "selectorExpansion":true,
         "expect":0,
         "misMatchThreshold":0.1,
         "requireSameDimensions":true
      }
   ],
   "paths":{
      "bitmaps_reference":"backstop_data/bitmaps_reference",
      "bitmaps_test":"backstop_data/bitmaps_test",
      "engine_scripts":"backstop_data/engine_scripts",
      "html_report":"backstop_data/html_report",
      "ci_report":"backstop_data/ci_report"
   },
   "report":[
      "browser"
   ],
   "engine":"puppeteer",
   "engineOptions":{
      "args":[
         "--no-sandbox"
      ]
   },
   "asyncCaptureLimit":5,
   "asyncCompareLimit":50,
   "debug":false,
   "debugWindow":false
}

Configuring backstop.json

Open the backstop.json file in your preferred editor. Replace the following definitions in viewports, since we will be running the tests in 3 different viewports:

"viewports":[
   {
      "label":"Phone",
      "width":960,
      "height":480
   },
   {
      "label":"Tablet",
      "width":660,
      "height":480
   },
   {
      "label":"Desktop",
      "width":1024,
      "height":768
   }
]

Next, we’ll configure our test scenarios. Edit the following parameters in the “scenarios” tag:

  • label”: Description for our test (Visual Regression Test Demo)
  • url”: Destination “URL” (For our test we will use index.html)
  • referenceUrl”: Reference “URL” (For our test we will be using index.html).
  • selectors”: Replace the default [] to [“html”]
    which sets backstop to take a screenshot of the entire HTML. Same we can specify with an array of selectors to capture screenshot only for that elements.

A full list of commands & explanations can be found in the official documentation.

After the updates, our scenario section will be as follows:

"scenarios":[
   {
      "label":"Visual Regression Test Demo",
      "cookiePath":"backstop_data/engine_scripts/cookies.json",
      "url":"/dist/index.html",
      "referenceUrl":"/dist/index.html",
      "readyEvent":"",
      "readySelector":"",
      "delay":0,
      "hideSelectors":[

      ],
      "removeSelectors":[

      ],
      "hoverSelector":"",
      "clickSelector":"",
      "postInteractionWait":0,
      "selectors":[
         "html"
      ],
      "selectorExpansion":true,
      "expect":0,
      "misMatchThreshold":0.1,
      "requireSameDimensions":true
   }
]

How to work with BackstopJS

After finishing the basic settings of the tests, let’s look at the commands for working with BackstopJS.

Run the following command to take reference screenshots which we will use to compare while running the tests:

$ backstop reference

It creates a series of baseline/reference images for future tests in the “backstop_data/bitmaps_reference/ <timestamp>/” folder. The folder structure before and after backstop reference is as follows:

Before and after backstop reference command execution

 

❗Note: You could also run tests without building any reference files, and after the failure, use the test result as references with approving the command which I will be explaining later.

Execution of backstop reference

 

Since we have all our reference screenshots, we can start tests using this command:

$ backstop test

It takes a new set of screenshots and compares them with reference set, then opens a report. New images will be generated in “backstop_data/bitmaps_test/<timestamp>/ ” folder and a report comparing the most recent reference screenshots and the current ones will be generated in “backstop_data/html_report” folder.

Folder structure after backstop test

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Passed Test

If all test cases are passed, you can update the reference screenshots with this command:

$ backstop approve

It sets the last screenshots as the new reference, accepting them as the new canonical version to test against next time.

BackstopJS Reports

After we run the tests, BackstopJS creates HTML reports in the “backstop_data / html_report / index.html” folder which will show a detailed comparison of the images.

The reports will be as follows:

Passed Test Report

Now, let’s spot the difference after changing something in the page.

Let’s make the following changes to the “dist/style.css” file:

  1. Change the alignment of heading “Visual Regression Demo Test Screen” to left by changing text-align: left; in the .top class.
  2. Change the alignment of heading “Grid section” in left to center by changing text-align: center; within the #row2> h2.
  3. Change the alignment of the footer text “Nithin” to right by changing text-align: right; to the .footer class.
  4. Update the footer image in the index.html file to https://imgur.com/0uZBDW5.jpg which has the logo color change from aqua to pink.

Let’s run the test again to see how it spots the difference by running the following command:

$ backstop test

This time the test will be FAILED!!!

Failed Tests

We can check the differences from the report generated:

Failed Test Report

The report will have a “Scrubber” option which helps us to recognize the changes in a comparison mode.

Scrubber View


Summarizing Guide for Creating and Maintaining Visual Regression Tests using BackstopJS

Correct implementation of visual tests can increase the quality of your product by reducing UI bugs. Visual Automated tests help to detect even the tiniest difference in the application’s UI. In addition, it helps to make our work more efficient. With visual tests in place, QA Engineers just need to verify the functional aspect rather than spending time to identify UI issues.

This is just a basic introduction to Visual regression testing, using BackstopJS. I invite you all to spot the differences in an automated way with BackstopJS 😎
Happy Learning!

Creating and Maintaining Visual Regression Tests using BackstopJS

References:

Automating CSS Regression Tests

https://github.com/garris/BackstopJS

https://garris.github.io/BackstopJS/

https://www.npmjs.com/package/backstopjs

 

Avatar

About the author

Nithin

An ISTQB certified professional with 6+ years of experience in the field of IT with focus on Quality Assurance (Automation & Manual) of web and mobile based (native & hybrid technologies) applications. Dedicated and hard-working individual with good communication and team-building skills & also good in managing multiple tasks in a pressured environment. Believes in doing high quality, thorough work and clear, honest, straight forward communication with Co-workers and superiors. Senior QA Automation Engineer at Fave. More articles by him can be found here : synapse-qa.com

Join TestProject Community

Get full access to the world's first cloud-based, open source friendly testing community. Enjoy TestProject's end-to-end test automation Platform, Forum, Blog and Docs - All for FREE.

Join Us Now  

Leave a Reply

popup image

Selenium for Teams

Sharing and distributing Selenium tests has never been so easy! With TestProject's FREE Selenium based platform, you can finally create awesome tests with the freedom to collaborate with your team effortlessly.
Sign Up Now right arrow
FacebookLinkedInTwitterEmail