feat(atomic-a11y): add vitest a11y reporter

[y-lakhdar]

TL;DR

Custom Vitest Reporter that captures axe-core accessibility results from Storybook tests and produces a structured JSON report per component.

Context

This PR adds the core reporting engine: a custom Vitest Reporter that hooks into test results, extracts axe-core accessibility violations, and builds a structured JSON report per component.

Shard merging (for parallel CI runs) was extracted to a separate PR (#7137) to keep this focused on the reporter itself.

What this PR does

npm run test:storybook [--shard=N/M]
        │
        ▼
  Vitest + Storybook addon-a11y
  (render each story → run axe-core → attach AxeResults to test meta)
        │
        ▼
  VitestA11yReporter
  ├─ onTestCaseResult():  axe tags → WCAG IDs, accumulate per component
  └─ onTestRunEnd():  build report → write JSON
        │
        ▼
  reports/
  ├── a11y-report.json                 ← single run
  ├── a11y-report.shard-1.json         ← sharded run (CI)
  └── ...

Wiring

The reporter is wired into packages/atomic/vitest.config.js as a custom reporter for the Storybook test project.

How to review

Start from the consumer, then trace into the engine.

Step 1 — Wiring (1 min)

Read packages/atomic/vitest.config.js first. This is where VitestA11yReporter is instantiated with its options (outputDir, packageJsonPath). This shows how the reporter plugs into Vitest — it's just a reporter in the reporters array.

Step 2 — Reporter lifecycle (core, ~10 min)

Open vitest-a11y-reporter.ts. This is the entrypoint class — everything else serves it. Follow these two methods:

1. onTestCaseResult(testCase) — called by Vitest for each test. Read top-to-bottom:

   • Filters for Storybook projects only (project.name.startsWith('storybook'))
   • Tries to extract axe results from test metadata (meta.reports)
   • Extracts component name, category, framework (→ storybook-extraction.ts)
   • Deduplicates by story ID, then accumulates violations/passes/criteria

2. onTestRunEnd(...) — called once at the end. Delegates to buildA11yReport() (→ report-builder.ts), serializes to JSON, writes to disk.

Step 3 — Report assembly (~5 min)

Open report-builder.ts. This transforms the Map<string, ComponentAccumulator> into the final A11yReport:

• buildComponents() — converts mutable accumulators → sorted A11yComponentReport[]
• buildCriteria() — inverts the component→criteria mapping into criteria→components
• Reads package metadata for axe-core/storybook versions

Step 4 — Supporting modules (skim, ~5 min)

These are leaf dependencies — review as needed:

Order	File	What to look for
4a	axe-integration.ts	Type guard for axe results, WCAG tag parsing logic
4b	storybook-extraction.ts	Regex patterns for component/category/framework detection
4c	shard-resolution.ts	CLI --shard flag parsing
4d	summary.ts	Summary computation (straightforward arithmetic)
4e	reporter-utils.ts	Package.json reading, date formatting, criterion metadata lookup

Step 5 — Types & shared (~2 min)

• src/shared/types.ts — all report interfaces (A11yReport, A11yComponentReport, A11yCriterionReport, A11ySummary)
• src/shared/constants.ts — defaults and sentinel values
• src/index.ts — public API surface (what gets exported from the package)

Key design decisions to look for

• Never throws from reporter hooks: onTestCaseResult and onTestRunEnd catch all errors and console.warn — a reporter crash must never break the test run
• Accumulator pattern: ComponentAccumulator uses Set<string> for deduplication, converted to plain arrays at report build time
• WCAG tag parsing: wcag143 → 1.4.3 (positional digit extraction, not a lookup table)

Try it

cd packages/atomic
pnpm build

# run storybook tests (single shard)
pnpm run test:storybook

View the JSON report in packages/atomic-a11y/reports/a11y-report.json

PR Chain (3 of 7)

#	PR	Branch	Description
1	#7111	feat/a11y-package-scaffold	Package scaffolding
2	#7122	feat/a11y-shared-foundation	Shared types, constants, utilities
3	#7123	feat/a11y-reporter	VitestA11yReporter + wiring ← this PR
4	#7137	feat/a11y-merge-shards	Shard merging for parallel CI runs
5	#7124	feat/a11y-openacr	OpenACR report generator
6	#7125	feat/a11y-scripts	CLI scripts
7	#7117	feat/a11y-ci-integration	Weekly a11y scan workflow

KIT-5469

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af3262b8f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Create parent directories for resolved output paths

The reporter resolves each output file path but only creates this.outputDir, so a custom outputFilename containing subdirectories (e.g. reports/a11y.json) causes writeFile to fail with ENOENT and the report is silently skipped after warning. Directory creation should target path.dirname(outputPath) for each file being written.

Useful? React with 👍 / 👎.