Add process metrics sidecar with /info endpoint #277
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a separate HTTP server for process metrics and worker metadata, started by run.go after spawning SDK workers. This works for all language workers (Go/Python/Java/TS/.NET). The sidecar serves: - /metrics: Process metrics (CPU/memory) for the worker PID - /info: Worker metadata (sdk_version, build_id) The /info endpoint only returns fields that run-scenario doesn't already know. During metrics export, /info is fetched to populate additional fields in the parquet output.
THardy98
force-pushed
the
add-process-metrics-sidecar
branch
2 times, most recently
from
28bf7ba to
bd291f5
Compare
- Add language field to /info endpoint and MetricLine parquet export - Add --prom-export-process-job flag (default: omes-worker-process) to query process metrics from a separate Prometheus job than SDK metrics - Pass language from runner to StartProcessMetricsSidecar
THardy98
force-pushed
the
add-process-metrics-sidecar
branch
from
bd291f5 to
0ec4989
Compare
- Add --auth-header argument to .NET, TypeScript, Python, and Java workers - Fix TLS flag parsing in TypeScript and Python to accept string values - Add --build-id argument to non-Go workers
THardy98
force-pushed
the
add-process-metrics-sidecar
branch
from
0ec4989 to
94c9c9a
Compare
This flag allows specifying the SDK version/ref to report in metrics separately from the --version flag (used for build-time SDK selection). The flag is: - Used by the sidecar's /info endpoint as sdk_version - NOT passed through to the actual worker process - Falls back to --version value if not specified This enables reporting the actual git ref (e.g., "master") in metrics when building from a local SDK directory (--version ./repo).
THardy98
force-pushed
the
add-process-metrics-sidecar
branch
from
689aff8 to
237999d
Compare
The sdk_version from the sidecar's /info endpoint is now included in the exported parquet metrics file, making it available for analysis and dashboards.
THardy98
force-pushed
the
add-process-metrics-sidecar
branch
from
1193946 to
f6e7c49
Compare
Sushisource
reviewed
Jan 13, 2026
Member
Sushisource
left a comment
Sushisource
left a comment
There was a problem hiding this comment.
This mostly makes sense to me - an update to the README describing this architecture would be a good idea
cmd/clioptions/metrics.go
Outdated
| }) | ||
| }) | ||
|
|
||
| // Start server |
Member
There was a problem hiding this comment.
Boring comment (so are the other ones inside the func, really)
Comment on lines
+84
to
+87
| // MetricsVersionTag is the SDK version/ref to report in metrics. | ||
| // This is used by the sidecar's /info endpoint and is NOT passed to the worker. | ||
| // If empty, falls back to the --version flag value. | ||
| MetricsVersionTag string |
Member
There was a problem hiding this comment.
Why would we want this to be different from version?
Contributor
Author
There was a problem hiding this comment.
--version can be a path to a local repo (which is what we use in our cloud worker testing), we'd prefer a git commit hash or something that actually identifies the version
cmd/clioptions/metrics_test.go
Outdated
Comment on lines
57
to
58
| // Give time for shutdown | ||
| time.Sleep(100 * time.Millisecond) |
Member
There was a problem hiding this comment.
These should instead just use an eventually kind of thing on the subsequent assertion (also more boring comments)
Contributor
Author
There was a problem hiding this comment.
replaced with a util called eventually (assertion with a timeout)
Contributor
Author
|
updated README to include some info on the metrics sidecar |
Sushisource
approved these changes
Jan 14, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was changed
Add a separate HTTP server for process metrics and worker metadata, started when spawning SDK workers. This works for all language workers (Go/Python/Java/TS/.NET).
The sidecar serves:
The /info endpoint only returns fields that run-scenario doesn't already know. During metrics export, /info is fetched to populate additional fields in the parquet output.
Why?
We opt for a Prom server sidecar for process metrics because core-based SDKs have their existing Prom server started by core, making it difficult to:
Using a sidecar server to capture process metrics is simple.
We add an
/infoendpoint to this sidecar server so that the scenario runner can fetch additional worker metadata for metrics export. This is useful in scenarios where the scenario runner and the worker are running on different machines (as is the case in cloud worker testing).Added some basic integration tests