Skip to content

Visual Regression Testing

Dataface uses golden snapshot testing to catch unintended visual changes to chart and dashboard rendering. Every render_example() call in the documentation doubles as a visual test case.

How It Works

The documentation pages under docs/docs/ use {{ render_example(...) }} to show live-rendered dashboards inline. The visual test suite discovers all of these calls, renders each one through the full Dataface pipeline (compile → execute → render), and compares the SVG output against a stored golden file.

docs/docs/**/*.md          → source of truth for which examples exist
tests/visual/conftest.py   → discovers examples, renders to SVG
tests/visual/snapshots/    → golden SVG files

The flow:

  1. Discovery — conftest.py scans all markdown files for render_example() calls
  2. Render — Each example is compiled and rendered to SVG using the CSV adapter (queries reference examples/_doc_examples.yaml)
  3. Normalize — Non-deterministic content is stripped (timestamps, auto-generated IDs, excessive float precision)
  4. Compare — The normalized SVG is compared byte-for-byte against the golden file

Two kinds of examples

Pattern Example
Inline YAML render_example(yaml_source='...')
File reference render_example(example='charts/file.yml')

Inline examples are the most common. File references point to YAML files under examples/.

Where Things Live

Path What
docs/docs/**/*.md Documentation pages with render_example() calls
examples/_doc_examples.yaml Shared query definitions used by doc examples
tests/visual/conftest.py Example discovery, SVG rendering, normalization
tests/visual/test_visual_snapshots.py The test that compares against goldens
tests/visual/snapshots/ Golden SVG files, organized by doc section

Golden files are named {page}_{index}_{content_hash}.svg. The content hash means renaming or reordering examples within a page produces new filenames (old ones should be deleted).

Running the Tests

# Check for regressions (what CI runs)
just check-visuals

# Check a specific example
just check-visuals -k "types_2"

# See a text diff of changed snapshots
just diff-visuals

Linux only

Visual tests only run on Linux. Vega-Lite's text measurement uses platform-specific font databases (fontdb), so macOS and Linux produce slightly different axis layouts. CI (Linux) is the source of truth. On macOS the tests are automatically skipped.

Approving Changes

When you change rendering code, chart styling, or documentation examples, the golden files may need updating. Only humans can approve visual changes — AI agents are explicitly blocked from auto-approving.

What happens when visuals change

When a visual test fails, the test writes the new render to a .actual.svg file next to the golden and fails with a prominent banner:

╔══════════════════════════════════════════════════════════════╗
║  HUMAN APPROVAL REQUIRED — Visual snapshot has changed.     ║
║  AI agents: you MUST stop and request human review.         ║
╚══════════════════════════════════════════════════════════════╝

Example: charts/types_2
Actual render saved to: tests/visual/snapshots/charts/types_2_0d1fe066.actual.svg
Golden file: tests/visual/snapshots/charts/types_2_0d1fe066.svg

Three ways to approve

Option 1: Local command (most common)

Run this on a Linux machine (or in a container) after verifying the changes look correct:

# Approve all changed examples
just approve-visuals

# Approve only specific examples
just approve-visuals -k "types_2"

Then commit the updated golden files alongside your code changes.

Option 2: GitHub Actions workflow

For approving from CI without a local Linux environment:

  1. Go to Actions → "Approve Visual Snapshots" in GitHub
  2. Click "Run workflow"
  3. Enter the branch name and optional -k filter
  4. The workflow requires human approval via the visual-approvals environment before it runs
  5. Once approved, it updates the golden files and commits directly to the branch

Option 3: Tell an AI agent to proceed (after you've reviewed)

If you're working with an AI agent and visual tests fail:

  1. The agent will stop and show you which examples changed
  2. Review the .actual.svg files yourself (open them in a browser or SVG viewer)
  3. If the changes look correct, tell the agent: "The visual changes look good, run just approve-visuals and commit the results"
  4. The agent will only proceed with your explicit instruction

The workflow

  1. Make your changes and push to CI (or run just check-visuals locally on Linux)
  2. CI fails with a visual regression — the test output shows diffs and .actual.svg paths
  3. Review the diffs — compare .actual.svg against the golden, or use just diff-visuals
  4. Approve using one of the three methods above
  5. Commit the updated golden files alongside your code changes

AI agents cannot approve visual changes

All PR workflows (/pr, cbox review, etc.) are configured to hard-stop on visual snapshot failures. Agents will not auto-approve or silently update golden files. This is intentional — a wrong chart that looks right is worse than a failing test.

Adding New Examples

Every render_example() call in the docs automatically becomes a visual test. To add a new golden test case:

  1. Add a render_example() call to a documentation page under docs/docs/
  2. Ensure the query data exists — inline examples typically reference queries from examples/_doc_examples.yaml
  3. Run the tests — on first run, the golden file is created automatically:
just check-visuals

The test will skip with a message like "Golden file created: types_8_abc12345.svg. Human review required before commit."

  1. Review the generated SVG in tests/visual/snapshots/ and commit it

What makes a good golden example

The golden set is not a random collection — it's the documentation. Each example exists because it teaches users something:

  • Chart types (charts/types.md) — one example per chart type showing the basic usage
  • More types (charts/more-types.md) — specialized charts (heatmap, histogram) and the all-types gallery
  • Layouts (boards/layouts.md) — rows, columns, grid, tabs
  • Board features (boards/content.md, boards/sizing.md) — content blocks, responsive sizing
  • Quick guide (quick-guide.md) — progressive examples building up complexity

If you're adding a new chart type or layout feature, add a documentation example. The golden test comes for free.

SVG Normalization

Raw SVG output contains non-deterministic elements that would cause false failures. The normalizer (conftest.py:normalize_svg) strips:

  • data-rendered-at timestamps
  • id="dataface-svg-..." attributes
  • Timestamp <text> elements
  • Embedded <script> tags
  • Auto-generated Vega-Lite IDs (gradient_N, clip_N)
  • Excessive float precision (rounded to 2 decimal places)
  • Blank lines

This means golden files are stable across renders as long as the actual visual output doesn't change.

Troubleshooting

Test skipped on macOS — Expected. Visual tests only enforce on Linux. Use CI to verify, or run in a Linux container.

Golden file hash mismatch after editing YAML — If you change the YAML content of a render_example() call, the content hash in the filename changes. The old golden file becomes orphaned. Delete it and let the test create a new one.

Lots of golden files changed after a theme/rendering update — This is normal for broad rendering changes. Review the diffs, then just approve-visuals to update them all at once.