Background Job Runner

[charlieroth]

Why

Need a reliable worker that executes deferred work, retries on failure, and updates job state. Enables fetching and extraction to run outside the request path

Description of Done

• Single process worker polls the jobs table and executes due jobs
• Concurrency is configurable
• Exponential backoff with jitter on failure
• Job state transitions are persisted with last error text
• Graceful shutdown drains in-flight work
• Unit tests cover backoff, max attempts, and state updates

Tasks

• Define job kinds and dispatch traits
• Implement polling loop with visibility timeout to prevent double work
• Implement exponential backoff with jitter and maximum attempts
• Add cooperative cancellation and shutdown signal handling
• Persist transitions: queued -> running -> succeeded or failed
• Add metric counters and timers via tracing events
• Write unit tests for scheduler, backoff and state transitions