RFC: Pre-seed BuildKit layer store to reduce build times

[hiroTamada]

Summary

• Proposes pre-seeding the builder VM rootfs with extracted base image layers to eliminate the ~10.6s decompression bottleneck on every code-change deployment
• Evaluates three approaches: pre-seed at build time, persistent volumes per tenant, shared read-only cache
• Recommends pre-seeding at build time as the simplest option with ~37% build time reduction

Problem

Every code-change deployment takes ~27s of build time. The single largest cost (~10.6s) is decompressing and extracting base image layers inside the ephemeral builder VM. This happens because:

1. Builder VMs start with an empty BuildKit content store
2. Any cache miss (code changed) requires BuildKit to reconstruct the base image filesystem
3. The registry stores compressed layers, but BuildKit needs uncompressed filesystem trees

Expected Impact

Scenario	Current	Proposed
Code change deploy	~27s build	~17s build (-37%)
Total deploy time	~50s	~40s

Open Questions

1. Should we pre-seed multiple base images or just nodejs?
2. Acceptable builder image size increase? (~100-150MB per base image)
3. Should warm-up be in CI/CD or manual when base images change?

🤖 Generated with Claude Code