Proto-first API rebuild with sebuf code generation#106
Draft
SebastienMelki wants to merge 54 commits intomainfrom
Draft
Proto-first API rebuild with sebuf code generation#106SebastienMelki wants to merge 54 commits intomainfrom
SebastienMelki wants to merge 54 commits intomainfrom
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…f.lock - buf.yaml v2 with STANDARD+COMMENTS lint, FILE+PACKAGE+WIRE_JSON breaking, deps on protovalidate and sebuf - buf.gen.yaml configures protoc-gen-ts-client, protoc-gen-ts-server, protoc-gen-openapiv3 plugins - buf.lock generated with resolved dependency versions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- geo.proto: GeoCoordinates with lat/lng validation, BoundingBox for spatial queries - time.proto: TimeRange with google.protobuf.Timestamp start/end - pagination.proto: cursor-based PaginationRequest (1-100 page_size) and PaginationResponse - i18n.proto: LocalizableString for pre-localized upstream API strings - identifiers.proto: typed ID wrappers (HotspotID, EventID, ProviderID) for cross-domain refs - general_error.proto: GeneralError with RateLimited, UpstreamDown, GeoBlocked, MaintenanceMode All files pass buf lint (STANDARD+COMMENTS) and buf build with zero errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SUMMARY.md documents 2 tasks, 9 files created, 2 deviations auto-fixed - STATE.md updated: plan 1/2 in phase 1, decisions recorded - ROADMAP.md updated: phase 01 in progress (1/2 plans) - REQUIREMENTS.md updated: PROTO-01, PROTO-02, PROTO-03 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mestamp User preference: all time fields use int64 (Unix epoch milliseconds) instead of google.protobuf.Timestamp for simpler serialization and JS interop. Applied to TimeRange and MaintenanceMode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test_item.proto with GeoCoordinates import and int64 timestamps - Add get_test_items.proto with TimeRange and Pagination imports - Add service.proto with HTTP annotations for TestService - All proto files pass buf lint and buf build
… pipeline - Add Makefile with generate, lint, clean, install, check, format, breaking targets - Update buf.gen.yaml with managed mode and paths=source_relative for correct output paths - Generate TypeScript client (TestServiceClient class) at src/generated/client/ - Generate TypeScript server (TestServiceHandler interface) at src/generated/server/ - Generate OpenAPI 3.1.0 specs (JSON + YAML) at docs/api/ - Core type imports (GeoCoordinates, TimeRange, Pagination) flow through to generated output
- Create 01-02-SUMMARY.md with pipeline validation results - Update STATE.md: phase 1 complete, 2/2 plans done, new decisions recorded - Update ROADMAP.md: phase 1 marked complete (2/2) - Update REQUIREMENTS.md: mark PROTO-04 and PROTO-05 complete
… servers, and OpenAPI specs Remove test domain protos (Phase 1 scaffolding). Add core enhancements (severity.proto, country.proto, expanded identifiers.proto). Define all 17 domain services: seismology, wildfire, climate, conflict, displacement, unrest, military, aviation, maritime, cyber, market, prediction, economic, news, research, infrastructure, intelligence. 79 proto files producing 34 TypeScript files and 34 OpenAPI specs. buf lint clean, tsc clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Prepare Phase 2B with full context file covering deliverables, key reference files, generated code patterns, and constraints. Update STATE.md with resume pointer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…pper) - router.ts: Map-based route matcher from RouteDescriptor[] arrays - cors.ts: TypeScript port of api/_cors.js with POST/OPTIONS methods - error-mapper.ts: onError callback handling ApiError, network, and unknown errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implements SeismologyServiceHandler from generated server types - Fetches USGS M4.5+ earthquake GeoJSON feed and transforms to proto-shaped Earthquake[] - Maps all fields: id, place, magnitude, depthKm, location, occurredAt (String), sourceUrl Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SUMMARY.md with task commits, decisions, and self-check - STATE.md updated: position, decisions, session info - REQUIREMENTS.md: SERVER-01, SERVER-02, SERVER-06 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Vendor sebuf/http/annotations.proto locally with Int64Encoding extension (50010) - Remove buf.build/sebmelki/sebuf BSR dep, use local vendored proto instead - Add INT64_ENCODING_NUMBER annotation to 34 time fields across 20 proto files - Regenerate all TypeScript client and server code (time fields now `number` not `string`) - Fix seismology handler: occurredAt returns number directly (no String() wrapper) - All non-time int64 fields (displacement counts, population) left as string - buf lint, buf generate, tsc, and sidecar build all pass with zero errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 2C-01-SUMMARY.md with execution results and deviations - Update STATE.md: plan 01 complete, int64 blocker resolved, new decisions - Update ROADMAP.md: mark 2C-01 plan complete - Update REQUIREMENTS.md: mark CLIENT-01 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GSD planning docs use formatting that triggers MD032 -- these are machine-generated and not user-facing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t and adapt all consumers to proto types - Replace legacy fetch/circuit-breaker adapter with port/adapter wrapping SeismologyServiceClient - Update 7 consuming files to import Earthquake from @/services/earthquakes (the port) - Adapt all field accesses: lat/lon -> location?.latitude/longitude, depth -> depthKm, time -> occurredAt, url -> sourceUrl - Remove unused filterByTime from Map.ts (only called for earthquakes, replaced with inline filter) - Update e2e test data to proto Earthquake shape Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lean API_URLS config - Delete api/earthquakes.js (legacy Vercel edge function proxying USGS) - Remove /api/earthquake Vite dev proxy (sebufApiPlugin handles seismology now) - Remove API_URLS.earthquakes entry from base config (no longer referenced) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 2C-02-SUMMARY.md with execution results - Update STATE.md: phase 2C complete, decisions, metrics - Update ROADMAP.md: mark 2C-02 and phase 2C complete - Mark requirements CLIENT-02, CLIENT-04, CLEAN-01, CLEAN-02 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add region (field 8) and day_night (field 9) to FireDetection proto - Regenerate TypeScript client and server types - Implement WildfireServiceHandler with NASA FIRMS CSV proxy - Fetch all 9 monitored regions in parallel via Promise.allSettled - Graceful degradation to empty list when API key is missing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Import createWildfireServiceRoutes and wildfireHandler in catch-all - Mount wildfire routes alongside seismology in allRoutes array - Rebuild sidecar-sebuf bundle with wildfire endpoint included Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 2D-01-SUMMARY.md with execution results - Update STATE.md position to 2D plan 01 complete - Update ROADMAP.md with 2D progress (1/2 plans) - Mark DOMAIN-01 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add src/services/wildfires/index.ts with fetchAllFires, computeRegionStats, flattenFires, toMapFires - Rewire App.ts to import from @/services/wildfires with proto field mappings - Rewire SatelliteFiresPanel.ts to import FireRegionStats from @/services/wildfires - Update signal-aggregator.ts source comment
- Remove api/firms-fires.js (replaced by api/server/worldmonitor/wildfire/v1/handler.ts) - Remove src/services/firms-satellite.ts (replaced by src/services/wildfires/index.ts) - Zero dangling references confirmed - Full build passes (tsc, vite, sidecar)
- Create 2D-02-SUMMARY.md with execution results - Update STATE.md: phase 2D complete, progress ~52% - Update ROADMAP.md: phase 2D plan progress
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Owner
|
Massive work 😃 ! @SebastienMelki |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
WorldMonitor currently has 56 hand-written API endpoint files in
api/, each independently implementing fetch logic, error handling, CORS, request validation, and response typing. This means:Every new upstream data source requires writing a new endpoint file from scratch, copying boilerplate from an existing one, and hoping the types stay in sync.
What this PR introduces
This PR lays the foundation for a proto-first API rebuild — defining every API contract in Protocol Buffers and generating type-safe TypeScript clients, server handler interfaces, and OpenAPI documentation from a single source of truth.
The toolchain: sebuf v0.7.0
sebuf v0.7.0 was released with WorldMonitor as the primary design target. It includes:
protoc-gen-ts-client— Generates typed client classes with injectablefetch, automatic JSON serialization, and proper error types (ValidationError,ApiError)protoc-gen-ts-server— Generates framework-agnostic server handler interfaces using the Web Fetch API. Works natively on Node 18+, Deno, Bun, Vercel Edge, and Cloudflare Workers — all environments WorldMonitor targetsprotoc-gen-openapiv3— Generates OpenAPI v3 specs (YAML + JSON) from the same proto definitionscreateXxxServiceRoutes()returns aRouteDescriptor[]that can be mounted on any HTTP framework, enabling a single catch-all Vercel function to replace all 56 individual filesonErrorcallback for mapping domain exceptions to HTTP responses, no more per-file try/catch boilerplateWhat's in this branch
79 proto files defining 17 domain services + shared core types:
SeismologyServiceWildfireServiceClimateServiceConflictServiceDisplacementServiceUnrestServiceMilitaryServiceAviationServiceMaritimeServiceCyberServiceMarketServicePredictionServiceEconomicServiceNewsServiceResearchServiceInfrastructureServiceIntelligenceServiceCore shared types (
proto/worldmonitor/core/v1/):GeoCoordinates,BoundingBox— Validated WGS84 coordinatesTimeRange— Unix epoch millisecond intervalsPaginationRequest/PaginationResponse— Cursor-based paginationSeverityLevel,CriticalityLevel,TrendDirection— Cross-domain enumsCountryCode— Validated ISO 3166-1 alpha-2EarthquakeID,CyberThreatID, etc.)GeneralError— Structured error types (RateLimited, UpstreamDown, GeoBlocked, MaintenanceMode)Generated output (from
make generate):src/generated/client/)src/generated/server/)docs/api/)Benefits
1. Single source of truth
Every field, type, enum, and constraint is defined once in a
.protofile. Client types, server types, and API docs are all derived — they cannot drift.2. Type-safe end-to-end
The generated
SeismologyServiceHandlerinterface forces handler implementations to accept the correct request type and return the correct response type. No moreas anyor mismatched interfaces.3. Automatic API documentation
Every service gets a complete OpenAPI v3 spec generated from the proto definitions — no manual Swagger/OpenAPI authoring needed. These live in
docs/api/and update automatically onmake generate.4. Validation at the boundary
Field constraints (
required,min_len,max_len,pattern,gte,lte) are defined in proto and enforced by the generated server. Invalid requests get a structured400response with specific field violations — no hand-written validation code.5. Unified server runtime
Instead of 56 individual Vercel edge functions, the plan is a single
api/[[...path]].tscatch-all that mounts allRouteDescriptor[]arrays. CORS, error handling, and request validation happen once in shared middleware.6. Framework-agnostic
The generated server code uses the Web Fetch API (
Request/Response). It runs identically on Vercel Edge, Vite dev server, and Tauri desktop — no framework coupling.7. Incremental migration
Old
api/*.jsfiles keep working. Vercel routing gives specific files priority over the catch-all. Each domain migrates independently: write the handler, mount it, verify, delete the old file.8. Generated clients for frontend
Instead of hand-writing
fetch('/api/earthquakes')with manual type assertions, the frontend will use:9. Built-in Tauri compatibility
Generated clients accept an injectable
fetchfunction, defaulting toglobalThis.fetch. The existing Tauri fetch patch intercepts all/api/*paths — new sebuf paths (/api/seismology/v1/list-earthquakes) work automatically with zero changes tosrc-tauri/.What's NOT in this PR (yet)
api/[[...path]].tsrouting (Phase 2B)api/*.jsfiles remain until each domain is migrated and verifiedHow to review
This is a contracts-only PR — no runtime behavior changes. The app continues to work exactly as before.
Quick validation
What to look at
proto/worldmonitor/seismology/v1/as the simplest). Check that field names, types, and comments make sense.src/types/index.tsto verify the proto definitions capture the same data. For example, compare theEarthquakemessage inseismology/v1/earthquake.protowith theEarthquakeinterface insrc/types/index.ts.proto/worldmonitor/core/v1/for the shared building blocks (geo, pagination, time, severity, identifiers).src/generated/client/worldmonitor/seismology/v1/service_client.ts) and server (src/generated/server/worldmonitor/military/v1/service_server.ts) to see the quality of generated TypeScript.docs/api/*.openapi.yamlto see the auto-generated API documentation.What NOT to review in detail
.planning/directory — internal planning docs, not shipped codeArchitecture diagram
🤖 Generated with Claude Code