Cross-platform visual regression testing

Orde Saunders' avatarPublished: by Orde Saunders

A follow up to my earlier article on CSS refactoring using visual diffing specifically dealing with cross-platform issues.

The tool I've used for visual regression testing is Wraith which, like many (most?) command line browser tools uses Phantom as a browser. Whilst Phantom is incredibly useful it is a distinct browser in it's own right and should be treated as such - i.e. it's not Chrome and none of your users are going to experience exactly what you get from Phantom.

However, if we're staying with Phantom then should we expect it to produce the same visual result across different platforms? If we take a more familiar example of Chrome or Firefox then, whilst the functionality is broadly equivalent between platforms, we're used to seeing visual differences between "desktop" and "mobile" versions of the same browser. What we're not so used to thinking about is the differences between different "desktop" platforms (e.g. Chrome on Windows vs. Chrome on Linux) so let's list off a few that we could reasonably expect:

  • Native UI widgets - in particular form controls
  • Installed fonts - in particular different system fonts and metrics
  • Display drivers - this can affect rendering

With the detailed comparison we're doing during visual diffing of rendered pages we should be expecting differences to show between different environments - in fact it should be more surprising if we don't.

The key with visual diffing when refactoring or regression testing is to compare like with like, we want to be comparing before and after from, ideally, the same machine. For example, this could either be your local development environment or your continuous integration server.

This means the workflow will be something along these lines:

  1. Take reference images of current "master"
  2. Make changes
  3. Compare changes to reference images

There may well be situations were you're prepared to lower the diff threshold so that it can tolerate the differences created by cross-platform use of a browser but that's a much broader topic and I suggest you check out Automated Responsive Design Testing for more information.