WebPageTest filmstrip diff automation for LCP

A byte budget can pass while the page still feels slower, because two builds can transfer identical bytes yet paint the hero in a different frame. WebPageTest closes that gap: it records a frame-by-frame filmstrip and computes Speed Index and LCP, so you can diff the visual progress of a candidate build against a stored baseline and see exactly which frame the media appears in. This guide, part of Monitoring & Regression for Media Delivery under CDN & Edge Media Delivery, shows how to drive the WebPageTest API from Node.js, pull the filmstrip and metrics, and turn a visual regression into a red CI check. It is the render-timeline complement to Lighthouse CI budget enforcement.

Prerequisite checklist

Warning: the public WebPageTest API is rate-limited and runs on shared agents whose queue depth varies through the day. For a gate that runs on every PR, use a private/self-hosted WPT instance or a paid API tier; otherwise queue time makes the job flap on timeouts rather than on real regressions.

What the filmstrip tells you that a budget can’t

Speed Index is the average time at which visible parts of the page are painted — it integrates the filmstrip’s visual-completeness curve over time. A lower Speed Index means content appears earlier. LCP marks the single frame where the largest element (usually the hero image) finishes painting. Diffing these between builds catches regressions that leave byte weight untouched: a hero that lost its fetchpriority hint, a lazy-loading attribute that crept onto the LCP image, or a render-blocking script inserted above it. None of those change total image bytes, but all of them push the hero’s paint into a later frame.

Baseline versus candidate filmstrip diff Two rows of filmstrip frames at 0.5-second intervals. The baseline row shows the hero image appearing by the 1.8s frame; the candidate row shows a blank hero until the 2.9s frame, marking an LCP regression the byte budget did not catch. Baseline LCP 1.8s 0.5s 1.0s 1.5s 1.8s ✓ hero 2.5s 3.0s Candidate LCP 2.9s 0.5s 1.0s 1.5s 1.8s ✗ blank 2.5s 2.9s ✓ hero Same bytes, +1.1s to hero paint — a regression only the filmstrip reveals

Exact solution

Step 1 — Submit a test and poll for completion

WebPageTest’s REST API is two calls: runtest.php submits and returns a testId; jsonResult.php returns the full result once the run finishes. Request multiple runs so you can take the median.

// wpt-run.mjs — submit a WebPageTest run and resolve with its JSON result.
const WPT_HOST = 'https://www.webpagetest.org';
const API_KEY = process.env.WPT_API_KEY;

export async function runTest(url) {
  const params = new URLSearchParams({
    url,
    k: API_KEY,
    f: 'json',                 // return JSON, not the HTML result page
    runs: '5',                 // FIVE runs so we can take a stable median
    fvonly: '1',               // first-view only; repeat-view doubles cost, not needed for LCP
    location: 'Dulles:Chrome', // PIN the location — a different agent = different numbers
    connectivity: '4G',        // PIN the network profile; must match the baseline exactly
    lighthouse: '1',           // also compute Lighthouse metrics (gives a clean LCP)
    video: '1',                // REQUIRED to get filmstrip frames back
  });

  // Submit the test.
  const submit = await fetch(`${WPT_HOST}/runtest.php?${params}`).then(r => r.json());
  if (submit.statusCode !== 200) {
    throw new Error(`WPT submit failed: ${submit.statusText}`);
  }
  const { testId, jsonUrl } = submit.data;

  // Poll jsonResult until statusCode 200 (complete). 100/101 = queued/running.
  for (let attempt = 0; attempt < 60; attempt++) {
    const res = await fetch(jsonUrl).then(r => r.json());
    if (res.statusCode === 200) return res.data;        // done
    if (res.statusCode >= 400) throw new Error(res.statusText);
    await new Promise(r => setTimeout(r, 5000));         // wait 5s between polls
  }
  throw new Error(`WPT test ${testId} did not complete in time`);
}

The two-call shape hides a three-party conversation: your CI job never talks to the test agent directly, it polls the API while the agent works through the queue. statusCode is the only progress signal you get — 100 queued, 101 running, 200 complete — and it is returned with HTTP 200 in every case, so you must branch on the body, not the transport status.

WebPageTest submit-and-poll sequence A sequence diagram with three lifelines: the CI job, the WebPageTest API, and a test agent. The CI job posts to runtest.php and receives a testId; the API queues the run on the agent; the CI job polls jsonResult and gets statusCode 100 queued, then 101 running; the agent uploads frames and traces; the final poll returns statusCode 200 with the complete result, and the CI job extracts the median first view. CI job (Node) WebPageTest API Agent Dulles:Chrome 4G POST runtest.php?f=json&runs=5 video=1 · location + connectivity pinned 200 { testId, jsonUrl } queue 5 first-view runs GET jsonResult.php statusCode 100 — queued poll again after 5 s statusCode 101 — running video frames + trace uploaded GET jsonResult.php statusCode 200 — complete + data extract data.median.firstView SpeedIndex · LCP · videoFrames

Step 2 — Extract the median metrics and filmstrip

WebPageTest already nominates a median run (by Speed Index by default). Pull its metrics and the filmstrip frames from the result tree.

// wpt-extract.mjs — reduce a raw WPT result to the numbers we gate on.
export function extractMetrics(data) {
  // data.median.firstView is the run WPT chose as representative.
  const fv = data.median.firstView;

  return {
    testId: data.testId,
    // Speed Index (ms) — integral of visual completeness; lower is better.
    speedIndex: fv.SpeedIndex,
    // LCP (ms). WebPageTest exposes it as chromeUserTiming.LargestContentfulPaint
    // or as the lighthouse LCP; prefer the Chrome trace value when present.
    lcp: fv['chromeUserTiming.LargestContentfulPaint']
         ?? fv['lighthouse.LargestContentfulPaint'],
    // Visual Complete (ms) — when the above-fold stopped changing.
    visualComplete: fv.visualComplete,
    // Filmstrip: array of { time, image, VisuallyComplete } frames.
    frames: (fv.videoFrames ?? []).map(f => ({
      timeMs: f.time,
      progress: f.VisuallyComplete,   // 0–100 visual completeness at this frame
      image: f.image,                 // URL of the frame thumbnail
    })),
  };
}

Step 3 — Diff candidate against baseline

Compare medians and flag the frame where visual progress diverges. A tolerance absorbs normal agent noise; anything beyond it is a real regression.

// wpt-diff.mjs — compare a candidate run to a stored baseline.
export function diffRuns(baseline, candidate, tolerance = { si: 0.10, lcp: 0.10 }) {
  const siDelta = (candidate.speedIndex - baseline.speedIndex) / baseline.speedIndex;
  const lcpDelta = (candidate.lcp - baseline.lcp) / baseline.lcp;

  // Find the first frame where the candidate is >5pp less visually complete
  // than the baseline at the same timestamp — that's where the hero slipped.
  let firstRegressionFrame = null;
  for (const base of baseline.frames) {
    const cand = candidate.frames.find(f => f.timeMs === base.timeMs);
    if (cand && base.progress - cand.progress > 5) {
      firstRegressionFrame = { timeMs: base.timeMs, baseline: base.progress, candidate: cand.progress };
      break;
    }
  }

  const regressed = siDelta > tolerance.si || lcpDelta > tolerance.lcp;
  return { regressed, siDelta, lcpDelta, firstRegressionFrame };
}

Step 4 — Wire it into CI

# .github/workflows/wpt-filmstrip.yml
name: WebPageTest filmstrip diff
on:
  pull_request:
    branches: [main]

jobs:
  filmstrip:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }

      - name: Diff candidate deploy against production baseline
        run: node ci/wpt-gate.mjs
        env:
          WPT_API_KEY: ${{ secrets.WPT_API_KEY }}
          # PREVIEW_URL is the PR's deploy; BASELINE_URL is current production.
          # Both must resolve publicly so WPT agents can reach them.
          PREVIEW_URL: ${{ steps.deploy.outputs.preview-url }}
          BASELINE_URL: https://media-delivery.com/products/
// ci/wpt-gate.mjs — the glue script the job runs.
import { runTest } from './wpt-run.mjs';
import { extractMetrics } from './wpt-extract.mjs';
import { diffRuns } from './wpt-diff.mjs';

const baseline = extractMetrics(await runTest(process.env.BASELINE_URL));
const candidate = extractMetrics(await runTest(process.env.PREVIEW_URL));
const diff = diffRuns(baseline, candidate);

console.log(`Speed Index: ${baseline.speedIndex} -> ${candidate.speedIndex} (${(diff.siDelta * 100).toFixed(1)}%)`);
console.log(`LCP:         ${baseline.lcp} -> ${candidate.lcp} (${(diff.lcpDelta * 100).toFixed(1)}%)`);
if (diff.firstRegressionFrame) {
  console.log(`First slipped frame at ${diff.firstRegressionFrame.timeMs}ms:`,
    `${diff.firstRegressionFrame.baseline}% -> ${diff.firstRegressionFrame.candidate}%`);
}
// Non-zero exit fails the check when the candidate regressed beyond tolerance.
process.exit(diff.regressed ? 1 : 0);

Verification steps

1. Confirm the API returns a median and filmstrip

# Submit a one-off run and confirm the result contains videoFrames.
# jq should print a non-empty array of frame timestamps.
curl -s "https://www.webpagetest.org/runtest.php?url=https%3A%2F%2Fmedia-delivery.com%2F&k=$WPT_API_KEY&f=json&runs=3&video=1&location=Dulles:Chrome&connectivity=4G" \
  | jq -r '.data.testId'

2. Prove the gate catches a real regression

Deploy a preview with loading="lazy" deliberately added to the LCP image and run the gate against it. The Speed Index and LCP deltas must exceed tolerance and the job must exit non-zero, with firstRegressionFrame naming the frame where the hero slipped. If it passes, your tolerance is too loose or the baseline and candidate used different location/connectivity values.

3. Confirm baseline and candidate used identical conditions

# Both result JSONs must report the same location and connectivity, or the
# diff is meaningless. Compare the two testInfo blocks.
jq '.data.testInfo | {location, connectivity}' baseline.json candidate.json

Expected metric deltas for a healthy build:

Metric Acceptable delta vs baseline Interpretation
Speed Index within ±10% agent noise, not a regression
LCP within ±10% hero paints at the same frame
First slipped frame none no frame lost visual progress
Visual Complete within ±10% above-fold stabilizes at the same time

Common mistakes and fixes

1. Trusting a single run

Anti-pattern: submitting runs=1 and comparing that number to the baseline.

Effect: WebPageTest agents are shared and their timings vary run-to-run by 10–20%. A single run diffs the noise, not the build, so the gate flaps green and red on identical code.

Fix: request runs=5 and compare the median run WebPageTest selects (or compute your own median). This mirrors the median-of-N rule used in Lighthouse CI budget enforcement and is non-negotiable for a stable gate.

2. Inconsistent location or connection profile

Anti-pattern: the baseline ran on Dulles:Chrome / Cable weeks ago; the candidate runs on London:Chrome / 4G today.

Effect: the diff conflates a network/geography change with a code change. A “regression” that is really just a slower connection profile blocks a clean PR.

Fix: pin location and connectivity as constants shared by both runs, assert they match (Verification step 3), and re-baseline whenever you deliberately change them. Latency and bandwidth dominate LCP, so they must be held constant.

3. Omitting video=1

Anti-pattern: requesting metrics without the video flag.

Effect: WebPageTest returns Speed Index and LCP but no videoFrames, so the filmstrip diff has nothing to compare and firstRegressionFrame is always null — you lose the visual evidence that is the whole point.

Fix: always pass video=1. It is what generates the frame captures the filmstrip diff reads.

4. Stale or absent baseline

Anti-pattern: comparing every candidate to a hard-coded baseline JSON committed a year ago.

Effect: slow drift accumulates undetected, or every PR looks like a huge regression because the baseline predates a legitimate redesign.

Fix: refresh the baseline from current production on a schedule (e.g. nightly) and store it as an artifact keyed by the production commit. The baseline should track the last-known-good production build, not a frozen snapshot.

5. Gating on LCP alone

Anti-pattern: asserting only on LCP and ignoring Speed Index and the frame diff.

Effect: a regression that delays secondary above-fold media (a second image, a background) without moving the single largest element passes, even though the page visibly loads slower.

Fix: gate on Speed Index and LCP and the first-slipped-frame check. LCP is one element; Speed Index and the filmstrip capture the whole above-fold render. For the field-side confirmation of an LCP regression, pair this with tracking LCP field data with the CrUX API.

Each signal has a blind spot, and the blind spots are not the same shape — which is the whole argument for running a byte budget and a filmstrip diff side by side rather than picking one. Map the regressions you actually ship against the signals that would catch them before deciding what to gate on:

Which signal catches which regression A coverage matrix. Rows are five regressions: hero bytes grew, lazy loading added to the LCP image, the fetchpriority hint dropped, a second above-fold image delayed, and below-fold gallery images bloated. Columns are the byte budget, Speed Index, LCP, and the filmstrip frame diff. The byte budget misses every timing-only regression, LCP misses regressions to non-largest elements, and only the byte budget catches below-fold bloat, which every render-timing signal misses. Regression shipped Byte budget Speed Index LCP Filmstrip frame diff Hero image bytes grew by 400 KB ✓ ✓ ✓ ✓ loading="lazy" crept onto the LCP image ✗ ✓ ✓ ✓ fetchpriority hint dropped from the hero ✗ ~ ✓ ✓ Second above-fold image delayed ✗ ✓ ✗ ✓ Below-fold gallery images bloated ✓ ✗ ✗ ✗ ✓ caught ~ sometimes caught ✗ missed Byte budgets are blind to timing-only regressions; render-timing signals are blind to below-fold weight.

Tradeoff: filmstrip runs are far slower and costlier than a Lighthouse budget — five runs across two URLs, each queued on a shared agent, can take minutes. Reserve filmstrip diffing for the handful of templates where render timing is the product (landing pages, article heroes) and let the cheaper byte budget guard the rest.