dev-tools-observer

Zero-dependency TypeScript browser SDK that intercepts all outgoing HTTP requests (fetch + XHR), hashes sensitive values, deduplicates similar requests by endpoint shape, and batches structured data to a configurable reporting server. The goal is to automatically map which API endpoints a front-end uses — and in what order — so that end-to-end tests can be generated automatically.

Quick start

import { ObserverSDK } from 'dev-tools-observer';

ObserverSDK.init({
  endpoint: 'https://your-ingest-server.com/ingest',
  orgId: 'my-org',      // optional, forwarded as-is to the server
  flushInterval: 5000,  // ms between automatic flushes, default 5000
  debug: false,         // set true to console.log every captured request
});

Call ObserverSDK.destroy() to remove interceptors and stop the flush timer (e.g. in test teardown).

Commands

npm run build          # tsup → dist/ (ESM + CJS + IIFE bundles)
npm run observe        # launch headed Chrome with SDK injected into every page
npm run serve:dev      # start a local ingest server on http://localhost:3737/ingest
npm test               # vitest run (jsdom environment)
npm run test:watch     # vitest watch mode
npm run test:coverage  # v8 coverage report
npm run typecheck      # tsc --noEmit (strict, zero errors required)

npm run observe opens a Chrome window and injects the SDK into every page you visit. Captured batches are forwarded to DT_ENDPOINT (default http://localhost:3737/ingest) via a Playwright CDP bridge — no gzip or keepalive issues. Environment variables:

DT_ENDPOINT=http://localhost:3737/ingest  # ingest server URL
DT_FLUSH_MS=5000                          # ms between automatic flushes
DT_ORG_ID=default                         # X-Org-Id header sent with every batch

---

Architecture

fetch() / XHR call
       │
       ▼
 interceptor-fetch.ts / interceptor-xhr.ts
   • blocks analytics domains + own endpoint (anti-recursion)
   • reads request method, URL, headers, body
   • reads response status, headers, body (via response.clone())
   • fires onCapture(RawRequestData) in background void-promise
       │
       ▼
 sdk.ts — onCapture()
   • parseUrl()       → normalizes path params, extracts query keys
   • hashQueryParams() → one hash per query param key
   • hashRecord()      → space-split hash for headers
   • hashRecord()      → space-split hash for path params
   • hashBody()        → JSON tree walk / form fields / text space-split
   • builds CapturedRequest
       │
       ▼
 queue.ts — EventQueue.push()
   • deduplicates by (method, domain, normalizedPath, sortedQueryKeyNames, graphqlOp)
   • increments count on duplicate
       │
       ▼
 (every flushInterval ms, or ObserverSDK.flush())
 sender.ts — sendBatch()
   • tries navigator.sendBeacon with gzip-compressed Blob
   • falls back to originalFetch POST with keepalive:true
   • uses originalFetch (saved before interceptor was installed) to bypass interception

---

Source files

File	Responsibility
src/index.ts	Public re-exports
src/sdk.ts	ObserverSDK static class — init / flush / destroy
src/types.ts	All interfaces (SDKConfig, RawRequestData, CapturedRequest, …)
src/blocklist.ts	Analytics domain blocklist + isBlocked()
src/hasher.ts	SHA-256, space-split hashing, JSON tree walker
src/interceptor-fetch.ts	window.fetch monkey-patch
src/interceptor-xhr.ts	XMLHttpRequest.prototype monkey-patch
src/queue.ts	EventQueue with deduplication
src/sender.ts	sendBeacon + fetch fallback, gzip via CompressionStream
src/url-parser.ts	URL decomposition, path parameter normalization, makeDedupeKey()

---

Hashing

All values are hashed with truncated SHA-256 (first 16 hex chars) via crypto.subtle. No raw values leave the browser.

Space-split hashing (headers)

Header values are split on spaces before hashing. "Bearer eyJ..." becomes "<hash(Bearer)> <hash(eyJ...)>". This preserves structural shape (a 2-part Bearer token stays distinguishable from a 1-part API key) without revealing either value. The ingest server can later correlate matching hash segments across endpoints.

Field path in stored data: requestHeaders.authorization.1 refers to the second space-split token of the authorization header.

JSON tree hashing

Request and response bodies parsed as JSON have every leaf value hashed individually. Objects and arrays keep their structure; keys are preserved; only the string/number leaf values are replaced with their hashes. Nulls and booleans are passed through unchanged.

Query param hashing

Each query param key maps to a single hash. Multi-value params (?tag=a&tag=b) are joined with a comma ("a,b") before hashing, producing one hash per key. The stored value is queryParams.tag = hash("a,b").

Path param hashing

Detected ID segments in the URL path are extracted as rawPathParams before hashing. Each one is hashed via hashRecord (which calls hashSpaceSplit — since IDs contain no spaces, this is equivalent to hashToken). Stored as pathParams[':param0'] = hash("123").

---

Path parameter normalization

parseUrl() calls normalizePathSegments() on the URL pathname before returning. Any segment that looks like a database ID is replaced with :param0, :param1, etc.

A segment is considered an ID if it matches any of these patterns:

Pattern	Examples detected
Pure numeric	123, 0, 9999999
UUID v4/v7 (case-insensitive)	550e8400-e29b-41d4-a716-446655440000
MongoDB ObjectId (24 hex chars)	507f1f77bcf86cd799439011
Hex string ≥ 8 chars (lowercase)	deadbeef, cafebabe1234
ULID (26 uppercase Crockford base32 chars)	01ARZ3NDEKTSV4RRFFQ69G5FAV
Base64url ID (≥ 8 chars, [A-Za-z0-9_-], contains both an uppercase letter and a digit)	dQw4w9WgXcQ, usr_2cHqf3kJL9OFjl

Human-readable slugs like my-product, v2, api, 2024-01-15, and plain lowercase words are intentionally not normalized.

The normalized path is what gets stored and used in the deduplication key, so /api/products/123 and /api/products/456 are stored as one endpoint: /api/products/:param0.

The raw extracted values (before hashing) are held in rawPathParams:

rawPathParams = { ':param0': '123' }

---

Deduplication

Two requests are considered the same endpoint shape if they share:

• HTTP method (uppercased)
• Domain (including port if non-standard)
• Normalized path (after ID replacement)
• Sorted query parameter key names (not values)
• GraphQL operationName (if present)

/users?page=1 and /users?page=2 are the same endpoint shape — they deduplicate and increment a counter. /users?page=1 and /users?page=1&sort=name are different shapes (different key sets).

---

What is and is not captured

Captured

• Method, domain, normalized path, query param key names
• Hashed query param values, hashed path param values
• Hashed request and response headers (space-split)
• Hashed request and response bodies (JSON tree walk, form fields, text space-split)
• Response status code
• Timestamp and duration

Blocked (never captured)

• Requests to analytics/tracking domains (Google Analytics, Mixpanel, Segment, Amplitude, Sentry, PostHog, Datadog, HotJar, Intercom, LaunchDarkly, …)
• Requests to the SDK's own reporting endpoint (anti-recursion)

Known limitation

URL strings inside JSON response bodies are hashed as opaque strings. A HATEOAS-style response like { "next": "https://api.example.com/items?cursor=TOKEN" } hashes the entire URL — the cursor=TOKEN inside it is not parsed. This means the system cannot detect the structural edge from that response field to a subsequent request's query string when the caller follows the URL directly. Individual cursor/token values passed via normal JSON fields are detected.

---

CapturedRequest shape

interface CapturedRequest {
  method: string;                       // "GET", "POST", …
  protocol: string;                     // "https"
  domain: string;                       // "api.example.com" or "localhost:3000"
  path: string;                         // normalized, e.g. "/users/:param0"
  queryParams: Record<string, string>;  // { page: hash("2") }
  pathParams: Record<string, string>;   // { ':param0': hash("123") }
  requestHeaders: Record<string, string>; // space-split hashed, lowercase keys
  requestBody: HashedBody | null;
  responseStatus: number;
  responseHeaders: Record<string, string>;
  responseBody: HashedBody | null;
  timestamp: number;
  duration: number;
  graphqlOperationName?: string;        // preserved unhashed
}

HashedBody is one of:

{ type: 'json';    data: unknown }               // recursive hash tree
{ type: 'graphql'; data: unknown; operationName: string | undefined }
{ type: 'form';    data: Record<string, string> } // each field hashed
{ type: 'text';    data: string }                 // space-split hashed
{ type: 'binary';  data: null }                   // no hash possible

---

TypeScript gotchas

• @tsconfig/strictest + tsup: tsup's CJS resolver cannot resolve the package by name. All strictest settings are inlined directly in tsconfig.json.
• Header keys are lowercased: always access them as raw.requestHeaders['content-type'], never 'Content-Type'.
• Closure narrowing: TypeScript narrows let variables assigned only inside closures to null. Pattern: use a { current: T | null } ref object, copy to const before calling.
• noUncheckedIndexedAccess: array/record access returns T | undefined. Use ?. and ??.
• exactOptionalPropertyTypes: T | undefined is distinct from an absent optional property.
• XHR tests: do not use vi.stubGlobal('XMLHttpRequest', FakeClass) — jsdom validates instances. Replace prototype methods with vi.fn() before installing the interceptor.
• XMLHttpRequest.DONE: use the literal 4 in source code — static properties may be undefined when the prototype is replaced in tests.