Spurious/flaky diffs

An important factor when constructing a good screenshot testing setup is to keep the number of flaky diffs to a minimum. A flaky diff (i.e. spurious diff or false positive) is when Happo finds a visual difference that isn't caused by a change in the code. These may involve (but are not limited to):

• image loading
• font loading
• requests to third-party servers
• asynchronous behavior (e.g. components fetching data)
• animations
• random data, counters, etc
• dates, timestamps, etc
• stack traces
• state an earlier story left on the page
• CSS whose result depends on the order stylesheets load in

Happo tries to take care of as many of these as possible, automatically. For instance, the following tasks are performed before taking the screenshot:

• wait for images (including background images and srcset)
• wait for custom fonts
• wait for asynchronous data fetching (XHR, window.fetch)
• freeze CSS animations (on their last frame by default, see freezeAnimations) and transitions
• stop SVG animations
• show animated GIFs and videos on their first frame
• ask for reduced motion, except in iOS Safari (see prefersReducedMotion)

Happo doesn't block network requests unless you ask it to: see Blocking external requests.

When you want to see how something moves rather than one frozen frame of it, animated snapshots step the animation through set times instead, so each frame comes out the same on every run.

Tracking a flake down

A flaky diff is worth a few minutes before you dismiss it: the cause is usually something you can fix once, for every snapshot it affects.

1. Compare the two images. Note what moved, changed size or changed color, and whether other snapshots in the report show the same change. A change spread over many components usually comes from something they share: a font, an image, a shared component's styles.
2. Look at what was rendered. View source shows what each image was rendered from and, depending on your integration, lets you download its assets or static package to reproduce it locally. Re-generate snapshot renders it again: if it comes out different again, the snapshot isn't stable on its own.
3. Read the logs. The report's logs page, under View logs at the bottom of its sidebar, shows what the run loaded and what timed out or was still loading when a screenshot was taken. If you use an AI assistant, Happo's MCP server can read the logs of one snapshot in each run, to find what differed.
4. Fix the cause with one of the tips below.
5. Report what's left as flake. A diff you can't attribute to a change can be reported as flaky, so Happo ignores that before and after pair from then on. It comes back once the baseline changes, so fix the cause when you can. For tiny differences such as anti-aliasing, a color delta threshold (compareThreshold) or whole-pixel font sizes are the better tools.

Tips & tricks

In some cases however, Happo can't automatically detect things that cause flaky diffs. Here are some tips & tricks that you might find useful when dealing with flaky diffs.

Dates and timestamps

If you have dates/timestamps, either injecting a fixed new Date('2024-05-23T08:28:02.446Z') into your component or freezing time via something like mockdate or Sinon.js can help.

Random data and generated ids

Anything that differs on every render — Math.random(), generated names and avatars, ids that show up in the output, counters that keep counting across stories — gives a new screenshot every run. In Chrome and Edge, Happo seeds Math.random(), so it returns the same sequence from the start of each page. That doesn't make a story's values stable, though: stories that share a page draw from one sequence, so a story's values change when the stories before it do, and Firefox and Safari aren't seeded. Render fixed data in your stories instead. If you generate it with a library like Faker, give it a fixed seed (faker.seed(123)) so it produces the same values every time.

External data

If a component depends on external data (e.g. via an API), consider splitting out the data fetching from the visual component and test the visual component without data fetching, injecting the data needed to render it.

Blocking external requests

Anything your snapshots load from a third party — fonts, scripts, images, analytics, ads — changes when that third party changes, loads slowly when it is slow, and fails when it is down. Each of those is a flaky diff waiting to happen. The allowedHostnames target option blocks every request to a hostname that isn't on its list, before it leaves the browser:

happo.config.ts

import { defineConfig } from 'happo';

export default defineConfig({
  targets: {
    chrome: {
      type: 'chrome',
      viewport: '1024x768',
      // Block every external request.
      allowedHostnames: [],
    },
  },
});

An empty list is the goal for a self-contained test suite: everything your snapshots need comes from your own package, and nothing outside it can change them. If your snapshots still need a few hostnames, list just those, like ['fonts.gstatic.com']. Leaving the option unset blocks nothing. The page being rendered and everything in your uploaded package are always allowed, but with the pages integration your pages load over the network too, so their hostnames have to be on the list. The option has no effect on ios-safari and ipad-safari targets, which can't block requests: they log that it was ignored, and their snapshots still depend on whatever they load.

To find out what your snapshots reach for, run once and read the logs: every run logs the external hostnames it requested, whether or not a list is set, and the report's logs page opens with that list. Once a list is in force, it splits into what was allowed and what was blocked. If a snapshot is missing something after you set one, see Missing fonts, images, or other external assets.

CI merge commits and mixed baselines

You can sometimes see diffs where the "after" image includes changes from a later commit on the main/base branch. The most likely cause is that the CI system is creating a dynamic merge commit (PR head merged with the current main tip) and Happo is using that merge instead of the PR head itself.

In GitHub Actions for instance, a way to fix this is to tell the checkout step to use a specific ref, like this:

- uses: actions/checkout@v6
  with:
    ref: ${{ github.event.pull_request.head.sha || github.ref }}
    fetch-depth: 100

GitLab's merged results and merge train pipelines run on a merge of the merge request into its target branch. The happo command files the screenshots under the merge request's head (CI_MERGE_REQUEST_SOURCE_BRANCH_SHA), so check that commit out before running it:

happo:
  script:
    - git checkout "${CI_MERGE_REQUEST_SOURCE_BRANCH_SHA:-$CI_COMMIT_SHA}"
    - npx happo

Bitbucket's pull request pipelines merge the destination branch into the pull request before their steps run. The happo command files the screenshots under the pull request's head (BITBUCKET_COMMIT), so check that commit out before running it:

pipelines:
  pull-requests:
    '**':
      - step:
          script:
            - git checkout "$BITBUCKET_COMMIT"
            - npx happo

The happo command warns when the checked-out commit isn't the head on GitHub Actions, GitLab CI and Bitbucket Pipelines.

Other CI systems may do the same kind of merge-by-default (sometimes called "merge refs" or "merge builds"). Look for settings that control whether the job checks out a synthetic merge commit vs the PR head SHA, and switch to the PR head SHA when you want deterministic Happo diffs.

Font loading issues

Font loading can be a common source of flaky diffs, especially when using external font CDNs. While Happo automatically waits for custom fonts to load, there are several scenarios where font loading can still cause inconsistencies:

CDN throttling and rate limiting

External font CDNs may throttle or rate-limit requests coming from the same IP or IP range. This is particularly problematic for Happo since it generates significant traffic to font URLs during screenshot testing. When throttling occurs, fonts may take longer to load or fail to load entirely, causing the browser to fall back to system fonts.

Symptoms may include:

• Inconsistent font rendering between test runs
• Fallback fonts being used instead of custom fonts

Solutions:

1. Use locally hosted fonts: The most stable solution is to host fonts locally within your testing environment. This eliminates external dependencies and ensures consistent font loading. Then set allowedHostnames without the CDN on it, so nothing can still load from there.
2. Disable throttling for font URLs: Contact your CDN provider to disable rate limiting for font URLs from your Happo testing environment. Using a custom header on outgoing requests using the outgoingRequestHeaders option could make CDN configuration simpler.

Fonts declared inside stories

Happo waits for fonts to load before taking a screenshot, but takes fonts it has already seen to be available. A @font-face rule declared inside a story, rather than globally, can leave a snapshot taken before its font arrives. Declare your @font-face rules globally, in a stylesheet every story loads, rather than in the stories.

Text whose edges change between runs

Sometimes text shows up in a diff although nothing about it moved: the glyphs are in the same place in both images, and only the shading along their edges is a little different. This is anti-aliasing noise. The browser smooths each glyph's edges with partly colored pixels, and how much of each pixel a glyph covers depends on exactly where the text sits. Given the same page, Chrome and Edge draw text the same way every time. When the edges differ, the text landed a fraction of a pixel away from where it was in the other run, too little to see as a move. That usually comes from sizes that don't add up to whole pixels, or from something that arrived at a different time, such as a font or an image above the text. Happo already ignores a diff where only pixels it recognizes as anti-aliasing changed, even without a color delta threshold, but a change in shading that is a bit larger can still get through.

Symptoms may include:

• On the comparison page, the changed pixels hug the outlines of the letters, and the text doesn't shift when you flip between before and after.
• Only a few pixels changed, and each by a small amount: a slightly lighter or darker gray, not text appearing where there was background.
• If you use Happo's MCP server, get_diff_measurements reports a small largest color difference (maxColorDelta), well below what a moved letter gives (see below).

Solutions:

1. Use whole-pixel font sizes and line heights. A size like 0.7rem works out to 11.2px with the default 16px root font size, and its line height to a fraction too. Fractions add up from line to line, so text lower on the page can end up close to halfway between two pixels, where the slightest difference between runs rounds it one way or the other. You don't have to leave rem for this: pick values that come out whole, like font-size: 0.75rem; line-height: 1rem (12px and 16px) instead of font-size: 0.7rem, and round spacing that doesn't, for example with round(0.5lh, 1px). That stops fractions from adding up down the page, so lines don't drift towards where rounding flips.
2. Make sure everything has loaded. A font, image or other asset that arrives late can move the text after it by a fraction of a pixel. The report's logs show what was still loading when a screenshot was taken.
3. Raise the color delta threshold slightly. A color delta threshold (compareThreshold) a little above the noise lets the edge shading through while still catching real changes. See below for how to pick one.

Choosing a threshold

The color delta threshold (compareThreshold) is how far a single pixel's color may move before it counts as changed, from 0 (identical) up to 1. Black to white scores about 0.93. Happo measures that distance in the YIQ color space, which weighs brightness more than hue, and the measure is quadratic: for a change between two grays, a difference of d levels (out of 255) scores about (d / 264)². Doubling a difference quadruples its score, so a few hundredths of threshold cover quite a lot:

Color delta threshold	A gray may change by up to
0.002	about 12 levels
0.03	about 46 levels
0.05	about 59 levels

Real changes to text score far higher than that. When dark text on a light background moves by one pixel, pixels that were dark glyph become background and the other way round, and those pixels score around 0.7, so any of these thresholds still catches it.

Faint lines are where a threshold starts to miss things. A light 1px border or divider that disappears only changes those pixels from light gray to the background color, which can score around 0.04: a threshold of 0.03 still catches it, but 0.05 does not.

To pick a threshold:

1. Look at the noise diffs you want to get rid of, and note their largest color difference, which Happo's MCP server reports as maxColorDelta.
2. Set the threshold just above that, and no higher.
3. Check it against the faintest change you would want to catch: the lightest border, divider or background tint in your UI, against what it sits on. If that change scores below your threshold, fix the text with whole-pixel sizes instead.

AVIF images

Avoid using images served in AVIF format. These are known to render in a non-deterministic way which will cause small but significant changes in pixel output. Use WEBP, PNG or JPG instead. If your images are served by a CDN, it's possible that they are automatically converted to AVIF even if the original image was of a different format.

Animations

Happo freezes most types of animations (e.g. CSS transitions & animations, SVG animations, etc). But if you have animations controlled from JavaScript, you need to find a way to disable them for the Happo test suite.

Examples of animations Happo can stop automatically:

• CSS transitions, e.g. .hero-img { transition: opacity 0.3s }
• CSS animations, e.g. .nav-menu { animation: fade-in 0.3s }
• SVG animations, e.g. <rect><animate attributeName="rx" values="0;5;0" dur="10s" /></rect>

Example of an animation that you need to disable yourself:

const element = document.getElementById('some-element-you-want-to-animate');
let start;

function step(timestamp) {
  if (start === undefined) {
    start = timestamp;
  }
  const elapsed = timestamp - start;

  const shift = 0.1 * elapsed;
  element.style.transform = `translateX(${shift}px)`;
  if (shift < 200) {
    requestAnimationFrame(step);
  }
}

if (!window.matchMedia('(prefers-reduced-motion: reduce)').matches) {
  // The browser isn't asking for reduced motion -- start the animation!
  requestAnimationFrame(step);
}

prefers-reduced-motion media query

One way to avoid flaky diffs caused by animations is to disable animations for browsers that signal they prefer reduced motion. By default, Happo is configured to prefer reduced motion.

Animated images

Happo shows animated GIFs on their first frame, but it can't pause an animated WebP or APNG <img>: it keeps playing, and can be caught on a different frame each run. Swap it for a still image in your stories.

Stories that share a page

In the Storybook integration, Happo renders many stories one after another in the same page instead of loading the page again for each one. Anything a story leaves behind is still there for the stories after it, and which stories share a page changes between runs: adding or removing stories, or a different number of chunks, moves them around. A story can then render differently depending on what ran before it. Common culprits:

• state kept outside the component: a store, a cache, a module-level variable, a mocked API's responses
• elements a story adds outside its root, like modals, toasts and tooltips rendered in a portal on document.body, that are never removed
• classes or styles added to <html> or <body>, and global styles a CSS-in-JS library injects
• timers and listeners that keep running after the story is gone
• CSS that only some stories load, which changes the order of the page's stylesheets (see below)

Reset what you can in a decorator that wraps every story, or clean up when a story unmounts. If a story needs a clean page, navigatePerStory gives every story its own page load, at the cost of slower runs.

Styles that depend on load order

When two rules with the same specificity set the same property on one element, the one that loads last wins. If the order your stylesheets load in can change, so can your screenshots. A common way to end up there is a shared component that takes a className and also styles its own root element:

// Tooltip.jsx: the root gets the tooltip's own class and the one passed in
<div className={`${styles.tooltip} ${className}`}>…</div>

/* Tooltip.module.css */
.tooltip {
  display: flex;
  justify-content: center;
}

/* ImpactScale.module.css, passed in as <Tooltip className={styles.component}> */
.component {
  display: inline-flex;
}

Many Storybook builds add a CSS module's styles to the page the first time a story imports it: webpack's style-loader does, and @storybook/nextjs uses it. Since stories share a page, whether Tooltip.module.css lands before or after ImpactScale.module.css depends on which stories rendered earlier on it, and the element above is centered in one run and not in the next.

Symptoms may include:

• An element that moves or changes size in a diff, although nothing about it changed
• The same component flipping back and forth between two looks across runs, in several stories at once
• A diff that comes and goes depending on which stories ran with it

Solutions:

1. Keep a shared component's own styles from competing with the class passed to it. Wrap them in :where(), which gives them no specificity, so any class passed in wins: :where(.tooltip) { display: flex; }. A cascade layer works too, but by precedence rather than specificity: a rule in a layer loses to any unlayered rule, and to rules in layers declared after its own. So it only helps if the class passed in is unlayered, or in a later layer. Either way, check that no global element selector (div { … }) now overrides them.
2. Raise the specificity of the rule that should win. Repeating the class (.component.component { display: inline-flex; }) is enough. This fixes one element at a time, so prefer the first solution when a component is used in many places.
3. Load each story on a fresh page. navigatePerStory gives every story its own page load, so the order no longer depends on the stories before it. It makes runs slower and hides the conflict rather than removing it, and your app can still run into it in production.

Stack traces

If you have components that throw errors, consider catching them and rendering a normalized error message. This is especially useful if you have components that render stack traces, which are likely to change between test runs.

One way to normalize these stack traces in a React component is to use an error boundary. Here's an example:

import { ErrorBoundary } from 'react-error-boundary';

// https://github.com/bvaughn/react-error-boundary?tab=readme-ov-file#errorboundary-with-fallbackrender-prop
const fallbackRender = ({ error }) => {
  // We need to sanitize ports, asset hashes, and line/col numbers from
  // the stack trace to make the Happo diffs stabilized.
  error.stack = error.stack.replace(
    /http:\/\/localhost:\d{4}.*?:\d+:\d+/g,
    'http://localhost:1234/path-to-file.1234abcd.bundle.js:1234:56',
  );

  throw error;
};

export const MiscFailing = () => (
  <ErrorBoundary fallbackRender={fallbackRender}>
    <ComponentThatThrows />
  </ErrorBoundary>
);