Skip to main content
Version: Current

Compare with a threshold

By default, Happo treats any two screenshots that have pixels that aren't the same as a diff, even if the diff is only a single pixel. This is by design, and can strike many as an odd limitation. Why not allow some small differences? There are two reasons we've made Happo strict about differences. The first one comes from our experience doing screenshot testing over the last decade — it's just inherently hard to know when a subtle diff is okay. The second reason is performance. Comparing bitmaps is expensive, so we want to avoid that as much as possible. Happo processes hundreds, even thousands of diffs every minute. Having to dig into the actual pixels would mean comparisons would run a lot slower. So instead we diff based on technique called "bitmap hashing". Each screenshot is assigned a hash based on its contents (the pixels). We compare hashes, not screenshots.

Deep comparisons​

In cases where the default bitmap hashing comparison is too limiting, you can tell Happo to deep-compare screenshots. A shallow diff (bitmap hash comparisons) is then first applied, and any diffs remaining after that will be deep-compared. Set the behavior of the deep comparison with the deepCompare option in happo.config.ts, or pass it in the comparison API call. Since happo 7, a config without deepCompare compares with { compareThreshold: 0.002, diffAlgorithm: 'color-delta' }.

To find settings that fit your test suite, try them on the compare thresholds page, on mock diffs or your own projects' diffs, and copy the deepCompare config from there.

Projects also have deep compare settings of their own on happo.io, which account admins change from the "Diff threshold" column of the projects admin page. These are deprecated: they apply only to comparisons made without deep compare settings, from happo before version 7, from a config with deepCompare: false, or from an API call that passes none.

There are five settings that control deep comparisons, outlined below.

Diff algorithm​

The algorithm used to compare pixels. The default is color-delta, which computes the perceptual color difference between pixels. You can also choose ssim (Structural Similarity Index), which compares structural patterns rather than individual pixel colors. ssim is experimental and may produce different results depending on your screenshots.

With the color-delta algorithm, a deep comparison leaves out pixels that look like anti-aliasing: the smoothing along the edges of text and shapes, which browsers and operating systems draw a little differently from one run to the next. A diff where only those pixels changed is not a diff once deep comparisons are on, at any color delta threshold. Happo ignores such a diff even in a project with no color delta threshold, as long as it uses color-delta without blur, a pixel threshold or whitespace removal, and no pixel changed only its transparency. It shows up among the ignored diffs, marked as anti-aliasing.

Color delta threshold​

The color delta threshold option

The color delta threshold, compareThreshold in the Happo config, controls how different two pixels are allowed to be. With the ssim algorithm, the same option is called the SSIM threshold, and it applies to the whole image instead: the images are equal when their structural difference (1 minus their SSIM) is below the threshold. Use the slider to change the threshold value. The right side gray box will change color to visualize what the threshold value means for pixel comparisons. It will update its color to the largest possible difference for the current threshold value.

Pixel threshold​

The pixel threshold option

In certain cases, a small amount of pixels can cause a diff despite the majority of pixels being the same. With the pixel threshold, ignoreThreshold in the Happo config, you can tell Happo to ignore a small amount of pixels. This is useful if you for instance have images with noise in them, causing a few pixels to have a large color difference.

Apply blur​

The blur images option

If your screenshots are high in contrast, you can blur the images slightly before comparing them. This can help smooth out some rough edges that can otherwise cause diffs. Note that blurring images when comparing them does nothing to the original screenshots -- it's simply a preprocessing step for the deep comparison.

Ignore whitespace​

When enabled, whitespace-only differences are ignored during deep comparison. This can help reduce diffs coming from spurious content shifts, especially when there's extra space at the bottom of the screenshots.

If you can get away with not enabling these settings at all, that is always best. If not, start with a low color delta threshold (compareThreshold) to begin with: 0.002. Leave other options turned off.

The pixel threshold (ignoreThreshold) is mostly used if you have graphical content (images, canvases, maps, etc) in your screenshots and it's hard to control noise. If you notice spots with few or single-pixel diffs, set the pixel threshold to something like "Allow 5 pixels to be above the color delta threshold in a 1000×500 pixel image". The underlying value is a float, so large images will allow more pixels to be ignored and small images will allow fewer.

Apply blur should most often be turned off as it does affect performance a little. Only turn it on if you have high contrast screenshots where element edges contain diffs.