Skip to content

Proto-first API rebuild with sebuf code generation#106

Draft
SebastienMelki wants to merge 54 commits intomainfrom
feat/sebuf-integration
Draft

Proto-first API rebuild with sebuf code generation#106
SebastienMelki wants to merge 54 commits intomainfrom
feat/sebuf-integration

Conversation

@SebastienMelki
Copy link
Collaborator

Motivation

WorldMonitor currently has 56 hand-written API endpoint files in api/, each independently implementing fetch logic, error handling, CORS, request validation, and response typing. This means:

  • 56 places where CORS headers can be wrong or missing
  • 56 places where error handling is inconsistent
  • 56 files with hand-written TypeScript interfaces that can drift from actual API responses
  • Zero compile-time guarantees that request/response shapes match between client and server
  • Zero auto-generated API documentation

Every new upstream data source requires writing a new endpoint file from scratch, copying boilerplate from an existing one, and hoping the types stay in sync.

What this PR introduces

This PR lays the foundation for a proto-first API rebuild — defining every API contract in Protocol Buffers and generating type-safe TypeScript clients, server handler interfaces, and OpenAPI documentation from a single source of truth.

The toolchain: sebuf v0.7.0

sebuf v0.7.0 was released with WorldMonitor as the primary design target. It includes:

  • protoc-gen-ts-client — Generates typed client classes with injectable fetch, automatic JSON serialization, and proper error types (ValidationError, ApiError)
  • protoc-gen-ts-server — Generates framework-agnostic server handler interfaces using the Web Fetch API. Works natively on Node 18+, Deno, Bun, Vercel Edge, and Cloudflare Workers — all environments WorldMonitor targets
  • protoc-gen-openapiv3 — Generates OpenAPI v3 specs (YAML + JSON) from the same proto definitions
  • Route descriptorscreateXxxServiceRoutes() returns a RouteDescriptor[] that can be mounted on any HTTP framework, enabling a single catch-all Vercel function to replace all 56 individual files
  • Proto-defined validation — Field constraints (required, min/max length, ranges, patterns) defined once in proto, enforced in both client and server
  • Custom error hooksonError callback for mapping domain exceptions to HTTP responses, no more per-file try/catch boilerplate

What's in this branch

79 proto files defining 17 domain services + shared core types:

Domain Service RPCs Upstream Sources
seismology SeismologyService 1 USGS
wildfire WildfireService 1 NASA FIRMS
climate ClimateService 1 Open-Meteo / ERA5
conflict ConflictService 3 ACLED, UCDP, HAPI/HDX
displacement DisplacementService 1 UNHCR
unrest UnrestService 1 ACLED + GDELT
military MilitaryService 3 OpenSky, Wingbits, AIS
aviation AviationService 1 FAA, Eurocontrol
maritime MaritimeService 2 AIS, NGA
cyber CyberService 1 Feodo, URLhaus, OTX, AbuseIPDB, C2Intel
market MarketService 4 Finnhub, Yahoo Finance, CoinGecko
prediction PredictionService 1 Polymarket
economic EconomicService 3 FRED, World Bank, EIA
news NewsService 2 RSS feeds, Groq/OpenRouter
research ResearchService 3 arXiv, GitHub, Hacker News
infrastructure InfrastructureService 2 Cloudflare Radar, custom
intelligence IntelligenceService 4 Computed (CII, PizzINT, GDELT, AI)

Core shared types (proto/worldmonitor/core/v1/):

  • GeoCoordinates, BoundingBox — Validated WGS84 coordinates
  • TimeRange — Unix epoch millisecond intervals
  • PaginationRequest/PaginationResponse — Cursor-based pagination
  • SeverityLevel, CriticalityLevel, TrendDirection — Cross-domain enums
  • CountryCode — Validated ISO 3166-1 alpha-2
  • 13 typed ID wrappers (EarthquakeID, CyberThreatID, etc.)
  • GeneralError — Structured error types (RateLimited, UpstreamDown, GeoBlocked, MaintenanceMode)

Generated output (from make generate):

  • 17 TypeScript client classes (src/generated/client/)
  • 17 TypeScript server handler interfaces (src/generated/server/)
  • 17 OpenAPI v3 specs in YAML + JSON (docs/api/)

Benefits

1. Single source of truth

Every field, type, enum, and constraint is defined once in a .proto file. Client types, server types, and API docs are all derived — they cannot drift.

2. Type-safe end-to-end

The generated SeismologyServiceHandler interface forces handler implementations to accept the correct request type and return the correct response type. No more as any or mismatched interfaces.

3. Automatic API documentation

Every service gets a complete OpenAPI v3 spec generated from the proto definitions — no manual Swagger/OpenAPI authoring needed. These live in docs/api/ and update automatically on make generate.

4. Validation at the boundary

Field constraints (required, min_len, max_len, pattern, gte, lte) are defined in proto and enforced by the generated server. Invalid requests get a structured 400 response with specific field violations — no hand-written validation code.

5. Unified server runtime

Instead of 56 individual Vercel edge functions, the plan is a single api/[[...path]].ts catch-all that mounts all RouteDescriptor[] arrays. CORS, error handling, and request validation happen once in shared middleware.

6. Framework-agnostic

The generated server code uses the Web Fetch API (Request/Response). It runs identically on Vercel Edge, Vite dev server, and Tauri desktop — no framework coupling.

7. Incremental migration

Old api/*.js files keep working. Vercel routing gives specific files priority over the catch-all. Each domain migrates independently: write the handler, mount it, verify, delete the old file.

8. Generated clients for frontend

Instead of hand-writing fetch('/api/earthquakes') with manual type assertions, the frontend will use:

const client = new SeismologyServiceClient('/api/seismology/v1');
const { earthquakes } = await client.listEarthquakes({ minMagnitude: 4.0 });
// earthquakes is fully typed as Earthquake[]

9. Built-in Tauri compatibility

Generated clients accept an injectable fetch function, defaulting to globalThis.fetch. The existing Tauri fetch patch intercepts all /api/* paths — new sebuf paths (/api/seismology/v1/list-earthquakes) work automatically with zero changes to src-tauri/.

What's NOT in this PR (yet)

  • Server handlers — The handler implementations that actually call upstream APIs (Phase 2B)
  • Catch-all gateway — The unified api/[[...path]].ts routing (Phase 2B)
  • Frontend migration — Switching UI components to use generated clients (Phase 2C+)
  • Legacy file deletion — Old api/*.js files remain until each domain is migrated and verified

How to review

This is a contracts-only PR — no runtime behavior changes. The app continues to work exactly as before.

Quick validation

make lint      # buf lint — should pass with zero errors
make generate  # regenerate all TypeScript + OpenAPI from protos
npx tsc --noEmit  # verify generated TypeScript compiles

What to look at

  1. Proto conventions — Review a few domain protos (start with proto/worldmonitor/seismology/v1/ as the simplest). Check that field names, types, and comments make sense.
  2. Entity shapes — Compare proto messages against src/types/index.ts to verify the proto definitions capture the same data. For example, compare the Earthquake message in seismology/v1/earthquake.proto with the Earthquake interface in src/types/index.ts.
  3. Core types — Review proto/worldmonitor/core/v1/ for the shared building blocks (geo, pagination, time, severity, identifiers).
  4. Generated output — Spot-check a generated client (src/generated/client/worldmonitor/seismology/v1/service_client.ts) and server (src/generated/server/worldmonitor/military/v1/service_server.ts) to see the quality of generated TypeScript.
  5. OpenAPI specs — Open any docs/api/*.openapi.yaml to see the auto-generated API documentation.

What NOT to review in detail

  • Individual generated TypeScript files — these are machine output, they'll regenerate on any proto change
  • The .planning/ directory — internal planning docs, not shipped code

Architecture diagram

proto/worldmonitor/{domain}/v1/          ← You define contracts here
        │
        ├── make generate
        │
        ├─→ src/generated/client/{domain}/v1/service_client.ts   (frontend uses this)
        ├─→ src/generated/server/{domain}/v1/service_server.ts   (handler implements this)
        └─→ docs/api/{Domain}Service.openapi.yaml                (docs get this free)
                                                                  
api/server/worldmonitor/{domain}/v1/handler.ts  ← You write handlers here (Phase 2B)
        │
        └─→ api/[[...path]].ts  ← Single catch-all mounts all routes (Phase 2B)

🤖 Generated with Claude Code

SebastienMelki and others added 20 commits February 18, 2026 12:32
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… patterns

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…f.lock

- buf.yaml v2 with STANDARD+COMMENTS lint, FILE+PACKAGE+WIRE_JSON breaking, deps on protovalidate and sebuf
- buf.gen.yaml configures protoc-gen-ts-client, protoc-gen-ts-server, protoc-gen-openapiv3 plugins
- buf.lock generated with resolved dependency versions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- geo.proto: GeoCoordinates with lat/lng validation, BoundingBox for spatial queries
- time.proto: TimeRange with google.protobuf.Timestamp start/end
- pagination.proto: cursor-based PaginationRequest (1-100 page_size) and PaginationResponse
- i18n.proto: LocalizableString for pre-localized upstream API strings
- identifiers.proto: typed ID wrappers (HotspotID, EventID, ProviderID) for cross-domain refs
- general_error.proto: GeneralError with RateLimited, UpstreamDown, GeoBlocked, MaintenanceMode

All files pass buf lint (STANDARD+COMMENTS) and buf build with zero errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SUMMARY.md documents 2 tasks, 9 files created, 2 deviations auto-fixed
- STATE.md updated: plan 1/2 in phase 1, decisions recorded
- ROADMAP.md updated: phase 01 in progress (1/2 plans)
- REQUIREMENTS.md updated: PROTO-01, PROTO-02, PROTO-03 marked complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mestamp

User preference: all time fields use int64 (Unix epoch milliseconds)
instead of google.protobuf.Timestamp for simpler serialization and
JS interop. Applied to TimeRange and MaintenanceMode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test_item.proto with GeoCoordinates import and int64 timestamps
- Add get_test_items.proto with TimeRange and Pagination imports
- Add service.proto with HTTP annotations for TestService
- All proto files pass buf lint and buf build
… pipeline

- Add Makefile with generate, lint, clean, install, check, format, breaking targets
- Update buf.gen.yaml with managed mode and paths=source_relative for correct output paths
- Generate TypeScript client (TestServiceClient class) at src/generated/client/
- Generate TypeScript server (TestServiceHandler interface) at src/generated/server/
- Generate OpenAPI 3.1.0 specs (JSON + YAML) at docs/api/
- Core type imports (GeoCoordinates, TimeRange, Pagination) flow through to generated output
- Create 01-02-SUMMARY.md with pipeline validation results
- Update STATE.md: phase 1 complete, 2/2 plans done, new decisions recorded
- Update ROADMAP.md: phase 1 marked complete (2/2)
- Update REQUIREMENTS.md: mark PROTO-04 and PROTO-05 complete
… servers, and OpenAPI specs

Remove test domain protos (Phase 1 scaffolding). Add core enhancements
(severity.proto, country.proto, expanded identifiers.proto). Define all
17 domain services: seismology, wildfire, climate, conflict, displacement,
unrest, military, aviation, maritime, cyber, market, prediction, economic,
news, research, infrastructure, intelligence. 79 proto files producing
34 TypeScript files and 34 OpenAPI specs. buf lint clean, tsc clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Feb 18, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment Feb 18, 2026 4:42pm
worldmonitor-finance Ready Ready Preview, Comment Feb 18, 2026 4:42pm
worldmonitor-startup Ready Ready Preview, Comment Feb 18, 2026 4:42pm

Request Review

Prepare Phase 2B with full context file covering deliverables,
key reference files, generated code patterns, and constraints.
Update STATE.md with resume pointer.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SebastienMelki and others added 5 commits February 18, 2026 15:09
…pper)

- router.ts: Map-based route matcher from RouteDescriptor[] arrays
- cors.ts: TypeScript port of api/_cors.js with POST/OPTIONS methods
- error-mapper.ts: onError callback handling ApiError, network, and unknown errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implements SeismologyServiceHandler from generated server types
- Fetches USGS M4.5+ earthquake GeoJSON feed and transforms to proto-shaped Earthquake[]
- Maps all fields: id, place, magnitude, depthKm, location, occurredAt (String), sourceUrl

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SUMMARY.md with task commits, decisions, and self-check
- STATE.md updated: position, decisions, session info
- REQUIREMENTS.md: SERVER-01, SERVER-02, SERVER-06 marked complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SebastienMelki and others added 3 commits February 18, 2026 16:18
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SebastienMelki and others added 3 commits February 18, 2026 16:58
- Vendor sebuf/http/annotations.proto locally with Int64Encoding extension (50010)
- Remove buf.build/sebmelki/sebuf BSR dep, use local vendored proto instead
- Add INT64_ENCODING_NUMBER annotation to 34 time fields across 20 proto files
- Regenerate all TypeScript client and server code (time fields now `number` not `string`)
- Fix seismology handler: occurredAt returns number directly (no String() wrapper)
- All non-time int64 fields (displacement counts, population) left as string
- buf lint, buf generate, tsc, and sidecar build all pass with zero errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 2C-01-SUMMARY.md with execution results and deviations
- Update STATE.md: plan 01 complete, int64 blocker resolved, new decisions
- Update ROADMAP.md: mark 2C-01 plan complete
- Update REQUIREMENTS.md: mark CLIENT-01 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GSD planning docs use formatting that triggers MD032 -- these are
machine-generated and not user-facing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SebastienMelki and others added 4 commits February 18, 2026 17:11
…t and adapt all consumers to proto types

- Replace legacy fetch/circuit-breaker adapter with port/adapter wrapping SeismologyServiceClient
- Update 7 consuming files to import Earthquake from @/services/earthquakes (the port)
- Adapt all field accesses: lat/lon -> location?.latitude/longitude, depth -> depthKm, time -> occurredAt, url -> sourceUrl
- Remove unused filterByTime from Map.ts (only called for earthquakes, replaced with inline filter)
- Update e2e test data to proto Earthquake shape

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lean API_URLS config

- Delete api/earthquakes.js (legacy Vercel edge function proxying USGS)
- Remove /api/earthquake Vite dev proxy (sebufApiPlugin handles seismology now)
- Remove API_URLS.earthquakes entry from base config (no longer referenced)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 2C-02-SUMMARY.md with execution results
- Update STATE.md: phase 2C complete, decisions, metrics
- Update ROADMAP.md: mark 2C-02 and phase 2C complete
- Mark requirements CLIENT-02, CLIENT-04, CLEAN-01, CLEAN-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SebastienMelki and others added 8 commits February 18, 2026 17:49
- Add region (field 8) and day_night (field 9) to FireDetection proto
- Regenerate TypeScript client and server types
- Implement WildfireServiceHandler with NASA FIRMS CSV proxy
- Fetch all 9 monitored regions in parallel via Promise.allSettled
- Graceful degradation to empty list when API key is missing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Import createWildfireServiceRoutes and wildfireHandler in catch-all
- Mount wildfire routes alongside seismology in allRoutes array
- Rebuild sidecar-sebuf bundle with wildfire endpoint included

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 2D-01-SUMMARY.md with execution results
- Update STATE.md position to 2D plan 01 complete
- Update ROADMAP.md with 2D progress (1/2 plans)
- Mark DOMAIN-01 requirement complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add src/services/wildfires/index.ts with fetchAllFires, computeRegionStats, flattenFires, toMapFires
- Rewire App.ts to import from @/services/wildfires with proto field mappings
- Rewire SatelliteFiresPanel.ts to import FireRegionStats from @/services/wildfires
- Update signal-aggregator.ts source comment
- Remove api/firms-fires.js (replaced by api/server/worldmonitor/wildfire/v1/handler.ts)
- Remove src/services/firms-satellite.ts (replaced by src/services/wildfires/index.ts)
- Zero dangling references confirmed
- Full build passes (tsc, vite, sidecar)
- Create 2D-02-SUMMARY.md with execution results
- Update STATE.md: phase 2D complete, progress ~52%
- Update ROADMAP.md: phase 2D plan progress
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@koala73
Copy link
Owner

koala73 commented Feb 18, 2026

Massive work 😃 ! @SebastienMelki

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments