Add pants support for pytest by jasonwbarnett · Pull Request #323 · buildkite/test-engine-client

jasonwbarnett · 2025-05-30T22:33:28Z

Description

Adds experimental support for running pytest tests using pants via a new pytest-pants test runner in bktec. This enables teams using Pants to integrate with Buildkite Test Analytics while retaining core retry and mute capabilities.

Context

#315

Changes

Adds pytest-pants as a new supported runner type
Installs Pants CLI in the Dockerfile
Adds dedicated runner logic (pytest_pants.go) with validation and command construction
Routes pytest-pants test plans to the API as pytest for compatibility
Updates README feature matrix and docs
Adds a complete example with pants resolve+lockfile, requirements, and test cases

Testing

Added unit tests for pytest-pants in pytest_pants_test.go
Verified end-to-end with sample Pants project in testdata/pytest_pants
Confirmed retry and result collection work as expected
Validated in a private monorepo

jasonwbarnett · 2025-05-30T22:43:22Z

It works!

$ export BUILDKITE_TEST_ENGINE_TEST_RUNNER=pytest
$ export BUILDKITE_TEST_ENGINE_TEST_RUNNER_VARIANT=pants
$ export BUILDKITE_TEST_ENGINE_TEST_CMD="pants test --output=all src/python/devops/azure_devops/checks_test.py -- --json={{resultPath}}"
$ ~/git/test-engine-client/test-engine-client
+++ Buildkite Test Engine Client: bktec dev


______ ______ _____
___  /____  /___  /____________
__  __ \_  //_/  __/  _ \  ___/
_  /_/ /  ,<  / /_ /  __/ /__
/_.___//_/|_| \__/ \___/\___/

Info: Python package 'buildkite-test-collector' is required and will not be verified by bktec. Please ensure it is added to the pants resolve used by pytest.
Billing Error: Test Splitting is not enabled on your plan: please upgrade at https://buildkite.com/organizations/<redacted>/billing.
⚠️ Falling back to non-intelligent splitting. Your build may take longer than usual.
+++ Buildkite Test Engine Client: Running tests
pants test src/python/devops/azure_devops/checks_test.py -- --json=/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/bktec-pytest-1365635794/pytest-results.json

16:40:22.93 [INFO] Completed: Run Pytest - (environment:macos_arm64, src/python/devops/azure_devops/checks_test.py:tests) - succeeded.

✓ src/python/devops/azure_devops/checks_test.py:tests succeeded in 1.03s (ran in local environment `macos_arm64`).
+++ ========== Buildkite Test Engine Report  ==========
✅ All tests passed.

+---------+-----------+---+
| Passed  | first run | 6 |
+         +-----------+---+
|         | on retry  | 0 |
+---------+-----------+---+
| Muted   | passed    | 0 |
+         +-----------+---+
|         | failed    | 0 |
+---------+-----------+---+
| Failed  |           | 0 |
+---------+-----------+---+
| Skipped |           | 0 |
+---------+-----------+---+
|               TOTAL | 6 |
+---------+-----------+---+
===================================================

jasonwbarnett · 2025-05-31T11:43:52Z

Since opening this PR, I’ve realized there’s a key issue that needs to be accounted for:

test-engine-client assumes a single invocation of pytest, and therefore only a single {{resultPath}}.
This doesn’t work with Pants (unless using xdist), because Pants spawns a separate pytest process per test—or per group of tests when using [batch_compatibility_tag][batch-compatibility-tag]. Each of these processes writes to the same JSON file, which results in only one set of test results being preserved. The others are overwritten.

One potential solution would be to update [test-collector-python] to use a [filelock][filelock] and append or merge results into the JSON file in a thread/process-safe way. This would allow multiple processes to contribute to a single result file without clobbering each other, and still keep things compatible with test-engine-client.

nprizal · 2025-06-04T22:47:13Z

Hi @jasonwbarnett. Thank you for opening the PR. I might be missing some contexts here, but can you explain more about the support you want to add, or the use case you might have?

bktec is currently designed around test splitting, where multiple instances of the client run a subset of test files that it receives from the server as a Test Plan. However, looking at your changes, it seems like bktec won't do the splitting. Instead, pants will spawn multiple pytest processes, which is not what bktec is designed currently.

We do have other features, such as muting or retrying tests. If that's something you'd want to achieve, we could add the support, but but it might require a slightly different approach compared to other test runners with splitting support.

jasonwbarnett · 2025-06-05T01:47:34Z

@nprizal Exactly! We’re primarily interested in the muting and retry features. Test splitting isn’t feasible in our case due to Pants, which already has richer context about the test structure. It’s not that it couldn’t be done, it’s just not a priority for us right now. What we’re really eager for is the ability to automatically mute flaky tests and allow developers to manually mark tests as flaky so their results can be excluded.

nprizal · 2025-06-06T05:21:08Z

@jasonwbarnett Got it! After looking at your changes, it seems like the runner behaves slightly differently when using pants. The main thing that is the same is the results handling. Personally, I prefer to create a separate runner and reuse some of the shared functionality. This will make it easier if we want to implement a splitting feature for pants.

// pytest.go
type Pytest struct {}

func NewPytest(c Config) Pytest {
  ...
}

func (p Pytest) GetFiles() (string[], error) {
  ...
}

func (p Pytest) Run(...) error {
  ...
  err = runAndForwardSignal(cmd)
  ...
  err = HandlePytestResult(result, p.ResultPath)
  ...
}

func HandlePytestResult(result RunResult, path string) error { // this is the shared function
   tests, err := ParsePytestCollectorResult(p.ResultPath)
   ...
   for _, test := range tests {
     result.RecordTestResult(...)
   }
   ...
}

// pytest_pants.go
type PytestPants struct {}

func NewPytestPants(c Config) {
  ...
}

func (p PytestPants) GetFiles() (string[], error) {
  return []string{}, nil
}

func (p PytestPants) Run(...) error {
  ...
  err = runAndForwardSignal(cmd)
  ...
  err = HandlePytestResult(result, p.ResultPath)
  ...
}

We are also planning of extracting the results handling, including the parsing, separately from the runner in the future. So that the runner package is responsible for running the tests, while the result_handler is responsible for parsing and recording the results.

internal
  /runner
    /pytest.go
    /pytest_pants.go
  /result_handler
    /pytest.go
    /run_result.go
    /test_result.go

Since opening this PR, I’ve realized there’s a key issue that needs to be accounted for:

test-engine-client assumes a single invocation of pytest, and therefore only a single {{resultPath}}.
This doesn’t work with Pants (unless using xdist), because Pants spawns a separate pytest process per test—or per group of tests when using [batch_compatibility_tag][batch-compatibility-tag]. Each of these processes writes to the same JSON file, which results in only one set of test results being preserved. The others are overwritten.

One potential solution would be to update [test-collector-python] to use a [filelock][filelock] and append or merge results into the JSON file in a thread/process-safe way. This would allow multiple processes to contribute to a single result file without clobbering each other, and still keep things compatible with test-engine-client.

This sounds like a sensible approach. My main concern is that in the current bktec pytest runner implementation, we create a temporary directory to store the test result if the BUILDKITE_TEST_ENGINE_RESULT_PATH is not set. This directory will be destroyed when bktec terminated. We can make the BUILDKITE_TEST_ENGINE_RESULT_PATH mandatory for pants so we can use that file for many pytest processes.

jasonwbarnett · 2025-06-06T16:21:44Z

@nprizal I’ve updated the PR and implemented a distinct PytestPants runner, separate from the standard Pytest runner. It reuses the existing pytest results parsing logic.

This setup works, but note that the results are currently overwritten with each test run orchestrated by pants, so only one test result is retained. This will be resolved once buildkite/test-collector-python#55 is merged and published to PyPI.

I'm pretty sure CI should pass with the addition of pants into the docker image, but I'm not fully sure. Let me know if it needs any changes to pass.

jasonwbarnett · 2025-06-06T16:50:38Z

Tests are passing locally:

Running tool: /opt/homebrew/bin/go test -timeout 30s -run ^(TestPytestPantsRun|TestPytestPantsRun_RetryCommand|TestPytestPantsRun_TestFailed|TestPytestPantsRun_CommandFailed|TestPytestPantsGetFiles|TestPytestPantsGetExamples|TestPytestPantsCommandNameAndArgs_WithResultPath|TestPytestPantsCommandNameAndArgs_WithoutResultPath|TestPytestPantsCommandNameAndArgs_InvalidTestCommand)$ github.com/buildkite/test-engine-client/internal/runner

ok  	github.com/buildkite/test-engine-client/internal/runner	2.420s

jasonwbarnett · 2025-06-06T17:33:55Z

I just finished adding documentation to this PR. I'm a little confused what to label and name things because pants can technically run many different types of tests, i.e. python, javascript, kotlin, golang, scala, typescript, etc

This PR is strictly aiming to support Python which is arguably the most predominantly used language in pants.

I'm open to changing the naming of things as you see fit.

Cheers!

jasonwbarnett · 2025-06-06T20:11:35Z

I confirmed the --merge-json flag works as expected.

with --merge-json

$ export BUILDKITE_TEST_ENGINE_TEST_CMD="pants --filter-target-type=python_test test //src/python/devops/azure_devops:: -- --json={{resultPath}} --merge-json"
$ ~/git/test-engine-client/test-engine-client
+++ Buildkite Test Engine Client: bktec dev


______ ______ _____
___  /____  /___  /____________
__  __ \_  //_/  __/  _ \  ___/
_  /_/ /  ,<  / /_ /  __/ /__
/_.___//_/|_| \__/ \___/\___/

Info: Python package 'buildkite-test-collector' is required and will not be verified by bktec. Please ensure it is added to the pants resolve used by pytest.
2025/06/06 14:08:02 DEBUG: Fetching test plan
2025/06/06 14:08:02 DEBUG: Sending request GET https://api.buildkite.com/v2/analytics/organizations/org/suites/test-out-new-thing/test_plan?identifier=<redacted>/<redacted>&job_retry_count=0
2025/06/06 14:08:03 DEBUG: Response code 404
2025/06/06 14:08:03 DEBUG: No test plan found, creating a new plan
2025/06/06 14:08:03 DEBUG: Test plan created. Identifier: "<redacted>/<redacted>"
2025/06/06 14:08:03 DEBUG: My favourite ice cream is
+++ Buildkite Test Engine Client: Running tests
pants --filter-target-type=python_test test //src/python/devops/azure_devops:: -- --json=/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/bktec-pytest-189074465/pytest-results.json --merge-json

Skipping automatic ecr-login because PANTS_SKIP_AUTO_ECR_LOGIN is set to true
14:08:16.89 [INFO] Completed: Run Pytest - (environment:macos_arm64, src/python/devops/azure_devops/checks_test.py:tests) - succeeded.
14:08:16.90 [INFO] Completed: Run Pytest - (environment:macos_arm64, src/python/devops/azure_devops/pull_request_description_test.py:tests) - succeeded.
14:08:16.99 [INFO] Completed: Run Pytest - (environment:macos_arm64, src/python/devops/azure_devops/releases/notes/helpers_test.py:tests) - succeeded.

✓ src/python/devops/azure_devops/checks_test.py:tests succeeded in 1.47s (ran in local environment `macos_arm64`).
✓ src/python/devops/azure_devops/pull_request_description_test.py:tests succeeded in 1.47s (ran in local environment `macos_arm64`).
✓ src/python/devops/azure_devops/releases/notes/helpers_test.py:tests succeeded in 1.56s (ran in local environment `macos_arm64`).
+++ ========== Buildkite Test Engine Report  ==========
✅ All tests passed.

+---------+-----------+----+
| Passed  | first run | 18 |
+         +-----------+----+
|         | on retry  |  0 |
+---------+-----------+----+
| Muted   | passed    |  0 |
+         +-----------+----+
|         | failed    |  0 |
+---------+-----------+----+
| Failed  |           |  0 |
+---------+-----------+----+
| Skipped |           |  0 |
+---------+-----------+----+
|               TOTAL | 18 |
+---------+-----------+----+
===================================================

Without --merge-json

$ export BUILDKITE_TEST_ENGINE_TEST_CMD="pants --filter-target-type=python_test test //src/python/devops/azure_devops:: -- --json={{resultPath}}"
$ ~/git/test-engine-client/test-engine-client
+++ Buildkite Test Engine Client: bktec dev


______ ______ _____
___  /____  /___  /____________
__  __ \_  //_/  __/  _ \  ___/
_  /_/ /  ,<  / /_ /  __/ /__
/_.___//_/|_| \__/ \___/\___/

Info: Python package 'buildkite-test-collector' is required and will not be verified by bktec. Please ensure it is added to the pants resolve used by pytest.
2025/06/06 14:08:02 DEBUG: Fetching test plan
2025/06/06 14:08:02 DEBUG: Sending request GET https://api.buildkite.com/v2/analytics/organizations/org/suites/test-out-new-thing/test_plan?identifier=<redacted>/<redacted>&job_retry_count=0
2025/06/06 14:08:03 DEBUG: Response code 404
2025/06/06 14:08:03 DEBUG: No test plan found, creating a new plan
2025/06/06 14:08:03 DEBUG: Test plan created. Identifier: "<redacted>/<redacted>"
2025/06/06 14:08:03 DEBUG: My favourite ice cream is
+++ Buildkite Test Engine Client: Running tests
pants --filter-target-type=python_test test //src/python/devops/azure_devops:: -- --json=/var/folders/kw/qhf2k3js3nvgf7t8_1g46skm0000gq/T/bktec-pytest-189074465/pytest-results.json --merge-json

Skipping automatic ecr-login because PANTS_SKIP_AUTO_ECR_LOGIN is set to true
14:08:16.89 [INFO] Completed: Run Pytest - (environment:macos_arm64, src/python/devops/azure_devops/checks_test.py:tests) - succeeded.
14:08:16.90 [INFO] Completed: Run Pytest - (environment:macos_arm64, src/python/devops/azure_devops/pull_request_description_test.py:tests) - succeeded.
14:08:16.99 [INFO] Completed: Run Pytest - (environment:macos_arm64, src/python/devops/azure_devops/releases/notes/helpers_test.py:tests) - succeeded.

✓ src/python/devops/azure_devops/checks_test.py:tests succeeded in 1.47s (ran in local environment `macos_arm64`).
✓ src/python/devops/azure_devops/pull_request_description_test.py:tests succeeded in 1.47s (ran in local environment `macos_arm64`).
✓ src/python/devops/azure_devops/releases/notes/helpers_test.py:tests succeeded in 1.56s (ran in local environment `macos_arm64`).
+++ ========== Buildkite Test Engine Report  ==========
✅ All tests passed.

+---------+-----------+----+
| Passed  | first run | 6 |
+         +-----------+----+
|         | on retry  |  0 |
+---------+-----------+----+
| Muted   | passed    |  0 |
+         +-----------+----+
|         | failed    |  0 |
+---------+-----------+----+
| Failed  |           |  0 |
+---------+-----------+----+
| Skipped |           |  0 |
+---------+-----------+----+
|               TOTAL | 6 |
+---------+-----------+----+
===================================================

jasonwbarnett · 2025-06-06T20:25:55Z

I'm blocked until buildkite/test-collector-python#55 is merged and published on pypi.org. The tests and code depend on it now that I'm planning to require --merge-json argument.

nprizal

Overall looks good. I have comments around the naming and the test plan creation. I have also turned on the 3rd party contributor build, so next time you push a commit, it should trigger a CI build in the pipeline.

nprizal · 2025-06-09T04:23:14Z

main.go

-	debug.Println("Creating test plan")
-	testPlan, err := apiClient.CreateTestPlan(ctx, cfg.SuiteSlug, params)
+	var testPlan plan.TestPlan
+	if params.Runner != "pytestpants" {


We still need to create the test plan to get the list of muted tests from the server.

Ah, good catch — I meant to pull that out before opening the PR. I started looking into where the list of supported runners is coming from, and it seemed like it was hardcoded in the server-side API, since the request fails when using pytestpants and returns a list of valid runners. But now I’m second-guessing myself — it might actually be this client that’s doing the filtering, and I just failed to track it down properly.

When main.go is reverted to the main branch I get this:

Buildkite Test Engine Client: Couldn't fetch or create test plan: Invalid parameters: Runner must be one of "rspec", "jest", "cypress", "playwright", "gotest", "pytest", "cucumber"

It does appear to be that the server side API is validating the client and since this is new, it's throwing it out. Maybe I need to figure out how to send pytest to the call, even though it's coming from the pytest-pants runner?

Mind taking a quick look or pointing me in the right direction?

I found where it's building params for the request and just swapped it with pytest (source here). Let me know if I should do something else here.

internal/runner/detector.go

jasonwbarnett · 2025-06-10T01:41:42Z

@nprizal Can you unblock the build pipeline?

jasonwbarnett · 2025-06-10T01:45:11Z

@nprizal Thanks for the support. It looks like the tests are passing! 🎉 🎉

internal/runner/pytest_pants.go

jasonwbarnett · 2025-06-10T01:50:38Z

@nprizal I tested everything end to end in our monorepo. Things look good.

nprizal

This looks great 🚀! The changes are isolated, and won't affect the existing pytest runner and other common behavior. I have few non blocking comments.

nprizal · 2025-06-10T22:35:27Z

main.go

-		}, nil
+		}
+
+		if cfg.TestRunner == "pytest-pants" {


We can maybe add a comment to explain why this is necessary?

docs/pytest-pants.md

jasonwbarnett · 2025-06-11T00:35:46Z

This looks great 🚀! The changes are isolated, and won't affect the existing pytest runner and other common behavior. I have few non blocking comments.

Thanks! I'll get to this in an hour.

jasonwbarnett · 2025-06-11T01:29:16Z

@nprizal I've updated the code with the two requested changes. Should I squash all commits or will you do that on merge?

nprizal

@jasonwbarnett Thanks for adding the comment! Could you please squash the commits? Once you’ve done that, I’ll merge the PR and create a release candidate so your team can begin testing it.

jasonwbarnett · 2025-06-11T01:46:34Z

@nprizal squashed and pushed.

nprizal · 2025-06-11T02:40:25Z

@jasonwbarnett I have created a release candidate that includes this PR
https://github.com/buildkite/test-engine-client/releases/tag/v1.6.0-rc.1

jasonwbarnett · 2025-06-11T02:43:26Z

I just realized I failed to update the PR description before this was merged. It's wildly inaccurate. Is there anyway to fix it?

nprizal · 2025-06-11T02:53:15Z

@jasonwbarnett I think you can still edit the PR description.

jasonwbarnett · 2025-06-11T21:03:26Z

@nprizal Great. The PR description is updated.

jasonwbarnett requested a review from a team as a code owner May 30, 2025 22:33

jasonwbarnett force-pushed the feat/add-pants-support branch 3 times, most recently from ea44b1b to bfeef2e Compare May 30, 2025 22:40

jasonwbarnett force-pushed the feat/add-pants-support branch 6 times, most recently from 693db65 to 2e09de0 Compare May 30, 2025 23:03

jasonwbarnett mentioned this pull request May 31, 2025

add --merge-json cli flag to support bktec and pants buildkite/test-collector-python#55

Merged

jasonwbarnett force-pushed the feat/add-pants-support branch 2 times, most recently from 48a75d7 to df291c5 Compare June 6, 2025 16:20

jasonwbarnett force-pushed the feat/add-pants-support branch 2 times, most recently from 0e203d1 to e4eab4e Compare June 6, 2025 16:26

jasonwbarnett changed the title ~~add pants support to pytest runner~~ feat: add pants support for pytest Jun 6, 2025

jasonwbarnett force-pushed the feat/add-pants-support branch 5 times, most recently from 0cdde1b to 54b2420 Compare June 6, 2025 16:49

jasonwbarnett force-pushed the feat/add-pants-support branch 2 times, most recently from 98e5300 to 35dc597 Compare June 6, 2025 17:24

jasonwbarnett force-pushed the feat/add-pants-support branch 2 times, most recently from 539a2b6 to 0ff4bd8 Compare June 6, 2025 17:28

jasonwbarnett force-pushed the feat/add-pants-support branch from 0ff4bd8 to 7365401 Compare June 6, 2025 17:37

nprizal requested changes Jun 9, 2025

View reviewed changes

jasonwbarnett force-pushed the feat/add-pants-support branch from ff33821 to d8d858c Compare June 9, 2025 20:43

jasonwbarnett requested a review from nprizal June 9, 2025 20:53

jasonwbarnett force-pushed the feat/add-pants-support branch 2 times, most recently from cdfa8c3 to db467c9 Compare June 10, 2025 01:38

jasonwbarnett commented Jun 10, 2025

View reviewed changes

internal/runner/pytest_pants.go Outdated Show resolved Hide resolved

nprizal approved these changes Jun 10, 2025

View reviewed changes

jasonwbarnett force-pushed the feat/add-pants-support branch from c042e79 to bf5ade5 Compare June 11, 2025 01:28

nprizal approved these changes Jun 11, 2025

View reviewed changes

feat: add pants support for pytest

b785e85

jasonwbarnett force-pushed the feat/add-pants-support branch from bf5ade5 to b785e85 Compare June 11, 2025 01:46

nprizal changed the title ~~feat: add pants support for pytest~~ Add pants support for pytest Jun 11, 2025

nprizal merged commit 9271e18 into buildkite:main Jun 11, 2025
1 check passed

jasonwbarnett deleted the feat/add-pants-support branch June 11, 2025 02:40

Conversation

jasonwbarnett commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Context

Changes

Testing

Uh oh!

jasonwbarnett commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonwbarnett commented May 31, 2025

Uh oh!

nprizal commented Jun 4, 2025

Uh oh!

jasonwbarnett commented Jun 5, 2025

Uh oh!

nprizal commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonwbarnett commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonwbarnett commented Jun 6, 2025

Uh oh!

jasonwbarnett commented Jun 6, 2025

Uh oh!

jasonwbarnett commented Jun 6, 2025

with --merge-json

Without --merge-json

Uh oh!

jasonwbarnett commented Jun 6, 2025

Uh oh!

nprizal left a comment

Choose a reason for hiding this comment

Uh oh!

nprizal Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

jasonwbarnett Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasonwbarnett Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jasonwbarnett commented Jun 10, 2025

Uh oh!

jasonwbarnett commented Jun 10, 2025

Uh oh!

Uh oh!

jasonwbarnett commented Jun 10, 2025

Uh oh!

nprizal left a comment

Choose a reason for hiding this comment

Uh oh!

nprizal Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jasonwbarnett commented Jun 11, 2025

Uh oh!

jasonwbarnett commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nprizal left a comment

Choose a reason for hiding this comment

Uh oh!

jasonwbarnett commented Jun 11, 2025

Uh oh!

Uh oh!

nprizal commented Jun 11, 2025

Uh oh!

jasonwbarnett commented Jun 11, 2025

Uh oh!

nprizal commented Jun 11, 2025

Uh oh!

jasonwbarnett commented Jun 11, 2025

jasonwbarnett commented May 30, 2025 •

edited

Loading

jasonwbarnett commented May 30, 2025 •

edited

Loading

nprizal commented Jun 6, 2025 •

edited

Loading

jasonwbarnett commented Jun 6, 2025 •

edited

Loading

jasonwbarnett Jun 9, 2025 •

edited

Loading

jasonwbarnett Jun 9, 2025 •

edited

Loading

jasonwbarnett commented Jun 11, 2025 •

edited

Loading