Passing ENVIRONMENT doesn't work during discovery

[miklelappo]

It seems that ENVIRONMENT settings passed into pytest_discover_tests only make difference during tests execution.
During discovery they seem to have no effect and though tests parametrized via env are discovered in a form 4/213 Test #4: XXX_e2e.test_add[token].

I didn't find an easy workaround for this.

[miklelappo]

I believe the problem is in this part of the code:

        # Set environment variables for collecting tests.
        set(ENV{${LIBRARY_ENV_NAME}} "${LIBRARY_PATH}")
        set(ENV{PYTHONPATH} "${PYTHON_PATH}")
        set(ENV{PYTHONWARNINGS} "ignore")

        # Collect tests.
        execute_process(
            COMMAND "${PYTEST_EXECUTABLE}"
                --collect-only -q
                --rootdir=${WORKING_DIRECTORY} ${DISCOVERY_EXTRA_ARGS} .
            OUTPUT_VARIABLE _output_lines
            ERROR_VARIABLE _output_lines
            OUTPUT_STRIP_TRAILING_WHITESPACE
            WORKING_DIRECTORY ${WORKING_DIRECTORY}
        )

[miklelappo]

And additionally, that add_custom_command doesn't support:

-D "ENVIRONMENT=${_ENVIRONMENT}"

[buddly27]

This behavior is intentional—changing the environment during discovery can interfere with the process, which needs to remain predictable to ensure the output can be reliably parsed. We recently introduced DISCOVERY_EXTRA_ARGS to address a similar situation (#54), and while we could consider adding a DISCOVERY_ENVIRONMENT, that feels a bit excessive for now.

During discovery they seem to have no effect and though tests parametrized via env are discovered in a form 4/213 Test #4: XXX_e2e.test_add[token].

I'm not entirely sure I follow this use case. Are you dynamically generating test cases based on environment variables? For instance:

cases = ["a", "b"] if os.getenv("TEST_CASE") == "1" else ["x", "y"]

@pytest.mark.parametrize("val", cases)
def test_param(val):
    assert val

If so, would you consider switching to a custom command-line argument instead? For example:

import pytest

def pytest_addoption(parser):
    parser.addoption("--test-case", action="store", default="0")

@pytest.fixture
def test_case(request):
    return request.config.getoption("--test-case")

@pytest.mark.parametrize("val", lambda test_case: ["a", "b"] if test_case == "1" else ["x", "y"])
def test_param(val):
    assert val

You could then pass the argument like this in your CMake configuration:

pytest_discover_tests(
    ...
    DISCOVERY_EXTRA_ARGS "--test-case"
)

Let me know if it works for you!

[miklelappo]

My use case is to pass an ENV variable from CMAKE to specify a directory, where test cases are located. (input/output). These are getting into parametrize then.

But I think the way you described above might be even cleaner... Let me try this out.

[miklelappo]

This worked like a charm, thanks!

[buddly27]

Great to hear it’s working!