Skip to content

Conversation

@anmarchenko
Copy link
Member

Summary

  • Add --ci-node-workers setting (default: CPU count) to control how many parallel workers run within each CI node
  • When running in CI-node mode with multiple workers, tests assigned to the node are split among local workers and executed in parallel via errgroup
  • {{nodeIndex}} in worker-env is now a global worker index calculated as ciNode * ciNodeWorkers + localWorkerIndex, ensuring unique indices across all workers on all nodes
  • Fix edge case in parallelism calculation when maxParallelism is 0 or negative

Test plan

  • Added tests for splitTestFilesIntoGroups() helper (even/uneven splits, edge cases)
  • Added tests verifying global index calculation (e.g., ci-node=1 with 2 workers yields indices 2 and 3)
  • Added tests for single-worker and multi-worker CI-node modes
  • Added tests for ci_node_workers setting defaults and env/flag overrides
  • Added tests for maxParallelism <= 0 edge case
  • make test and make lint pass

@anmarchenko anmarchenko requested a review from a team as a code owner January 30, 2026 13:01
@anmarchenko anmarchenko changed the title Add local parallelism for CI nodes [SDTEST-2702] Add local parallelism for CI nodes Jan 30, 2026
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 477c53b619

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +80 to +83
// Global index = ciNode * ciNodeWorkers + localIndex
globalIndex := ciNode*ciNodeWorkers + localIndex
g.Go(func() error {
return runTestsWithGlobalIndex(ctx, framework, groupFiles, workerEnvMap, globalIndex)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Ensure global worker indices stay unique across CI nodes

The global index formula here (ciNode*ciNodeWorkers + localIndex) assumes every CI node uses the same ciNodeWorkers value, but runCINodeTests sources it from settings (default runtime.NumCPU) per node. In heterogeneous CI pools (different CPU counts), this yields overlapping indices (e.g., node0 with 8 workers produces 0–7, node1 with 4 workers produces 4–7), which defeats the stated uniqueness guarantee and can cause {{nodeIndex}}-based worker envs (DB names, ports) to collide across nodes unless users manually align ci-node-workers everywhere.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants