Skip to content

Skip a UVM-Heavy Test Under Compute Sanitizer #1400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
leofang merged 1 commit into from
Dec 18, 2025
Merged

Skip a UVM-Heavy Test Under Compute Sanitizer #1400

leofang merged 1 commit into from
Dec 18, 2025

Conversation

@Andy-Jost
Copy link
Contributor

@Andy-Jost Andy-Jost commented Dec 17, 2025 •
edited
Loading

Summary

Adjust CI to improve debuggability of long-running tests.

Changes

  • Add --durations=0 to CI pytest invocations so logs report per-test timings.
  • Skip cuda_core/tests/test_helpers.py::test_patterngen_seeds when running under compute-sanitizer (detected via cuda_python_test_helpers.under_compute_sanitizer()), since this test can become extremely slow under sanitizer/UVM-heavy behavior.

Rationale

test_patterngen_seeds performs many syncs and host/device-visible memory comparisons; under compute-sanitizer this can balloon to very long runtimes in some CI environments. Skipping it under sanitizer keeps the CI signal while avoiding hour-scale jobs.

Test plan

  • CI

@Andy-Jost Andy-Jost self-assigned this Dec 17, 2025
@copy-pr-bot
Copy link
Contributor

copy-pr-bot bot commented Dec 17, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Andy-Jost Andy-Jost changed the title Skip test_latchkernel Skip tests in preparation for release Dec 17, 2025
@Andy-Jost Andy-Jost marked this pull request as draft December 17, 2025 20:02
@Andy-Jost
Copy link
Contributor Author

Andy-Jost commented Dec 17, 2025

/ok to test ec93c12

@github-actions

This comment has been minimized.

Sign in to view
@leofang leofang added this to the cuda.core beta 10 milestone Dec 17, 2025
@leofang leofang added bug Something isn't working P0 High priority - Must do! test Improvements or additions to tests cuda.core Everything related to the cuda.core module labels Dec 17, 2025
@Andy-Jost Andy-Jost force-pushed the disable-tests-for-release branch 2 times, most recently from ce6b1b5 to e3183ab Compare December 17, 2025 21:58
@Andy-Jost
Copy link
Contributor Author

Andy-Jost commented Dec 17, 2025

/ok to test e3183ab

@Andy-Jost
Copy link
Contributor Author

Andy-Jost commented Dec 17, 2025

/ok to test cb86bd1

@Andy-Jost
Copy link
Contributor Author

Andy-Jost commented Dec 17, 2025

/ok to test 8823fcc

@Andy-Jost
CI: add pytest durations and skip patterngen seeds under sanitizer
cbe5b2d 
- Add --durations=0 to CI pytest invocations to report per-test timings.
- Add cuda_python_test_helpers.under_compute_sanitizer() and use it to skip
  the UVM-heavy PatternGen seed test under compute-sanitizer.
@Andy-Jost Andy-Jost force-pushed the disable-tests-for-release branch from 8823fcc to cbe5b2d Compare December 17, 2025 23:42
@Andy-Jost
Copy link
Contributor Author

Andy-Jost commented Dec 17, 2025

/ok to test cbe5b2d

Andy-Jost
Andy-Jost commented Dec 17, 2025
View reviewed changes
ci/tools/run-tests
"LD:${CUDA_PATHFINDER_TEST_LOAD_NVIDIA_DYNAMIC_LIB_STRICTNESS} " \
"FH:${CUDA_PATHFINDER_TEST_FIND_NVIDIA_HEADERS_STRICTNESS}"
pytest -ra -s -v tests/ |& tee /tmp/pathfinder_test_log.txt
pytest -ra -s -v --durations=0 tests/ |& tee /tmp/pathfinder_test_log.txt
Copy link
Contributor Author

@Andy-Jost Andy-Jost Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems reasonable to report test times so that these issues are easier to debug in the future.

@Andy-Jost Andy-Jost requested a review from leofang December 17, 2025 23:45
@Andy-Jost Andy-Jost changed the title Skip tests in preparation for release Skip a UVM-Heavy Test Under Compute Sanitizer Dec 17, 2025
@Andy-Jost Andy-Jost marked this pull request as ready for review December 17, 2025 23:46
@Andy-Jost
Copy link
Contributor Author

Andy-Jost commented Dec 18, 2025

CI confirms the fix. This test was taking >20min:

CI / Test linux-64 / py3.12, 12.9.1, local, l4 (push)Successful in 6m

@Andy-Jost
Copy link
Contributor Author

Andy-Jost commented Dec 18, 2025

Longest durations now reported in the test logs:

============================== slowest durations ===============================
7.42s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPool::test_main[3]
6.67s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingRegistry::test_main[3]
6.64s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingIPCDescriptors::test_main[3]
6.63s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingRegistry::test_main[1]
6.44s call     tests/memory_ipc/test_send_buffers.py::TestIpcReexport::test_main[DeviceMR]
6.38s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPool::test_main[1]
5.60s call     tests/memory_ipc/test_memory_ipc.py::TestIPCSharedAllocationHandleAndBufferObjects::test_main[DeviceMR]
5.57s call     tests/memory_ipc/test_memory_ipc.py::TestIPCSharedAllocationHandleAndBufferDescriptors::test_main[DeviceMR]
4.50s call     tests/memory_ipc/test_memory_ipc.py::TestIPCMempoolMultiple::test_main[DeviceMR]
4.43s call     tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[1]
4.40s call     tests/memory_ipc/test_workerpool.py::TestIpcWorkerPoolUsingIPCDescriptors::test_main[1]
4.39s call     tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main[DeviceMR]
3.42s call     tests/memory_ipc/test_event_ipc.py::TestEventIpc::test_main[DeviceMR]
3.36s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-True-True]
3.36s call     tests/memory_ipc/test_errors.py::TestImportWrongMR::test_main[DeviceMR]
3.36s call     tests/memory_ipc/test_send_buffers.py::TestIpcSendBuffers::test_main[3]
3.35s call     tests/memory_ipc/test_errors.py::TestDanglingBuffer::test_main[DeviceMR]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-False-True]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-True-False]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-False-True]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-True-False]
3.35s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[False-False-False]
3.34s call     tests/memory_ipc/test_serialize.py::TestObjectSerializationWithMR::test_main[DeviceMR]
3.34s call     tests/memory_ipc/test_serialize.py::TestObjectPassing::test_main[DeviceMR]
3.33s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-False-False]
3.32s call     tests/memory_ipc/test_event_ipc.py::TestIpcEventProperties::test_main[True-True-True]
3.32s call     tests/memory_ipc/test_memory_ipc.py::TestIpcMempool::test_main[DeviceMR]
3.31s call     tests/memory_ipc/test_errors.py::TestImportBuffer::test_main[DeviceMR]
2.65s call     tests/memory_ipc/test_errors.py::TestAllocFromImportedMr::test_main[DeviceMR]
2.39s call     tests/example_tests/test_basic_examples.py::TestExamples::test_example[/__w/cuda-python/cuda-python/cuda_core/tests/example_tests/../../examples/vector_add.py]
2.35s call     tests/example_tests/test_basic_examples.py::TestExamples::test_example[/__w/cuda-python/cuda-python/cuda_core/tests/example_tests/../../examples/saxpy.py]
2.25s call     tests/example_tests/test_basic_examples.py::TestExamples::test_example[/__w/cuda-python/cuda-python/cuda_core/tests/example_tests/../../examples/memory_ops.py]
1.65s call     tests/test_helpers.py::test_latchkernel

@leofang leofang enabled auto-merge (squash) December 18, 2025 00:11
@leofang leofang merged commit 083315c into NVIDIA:main Dec 18, 2025
80 checks passed
@github-actions
Copy link

github-actions bot commented Dec 18, 2025

Doc Preview CI
Preview removed because the pull request was closed or merged.

@Andy-Jost Andy-Jost deleted the disable-tests-for-release branch December 18, 2025 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leofang leofang leofang approved these changes

Assignees

@Andy-Jost Andy-Jost

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do! test Improvements or additions to tests

Projects

None yet

Milestone

cuda.core beta 10

Development

Successfully merging this pull request may close these issues.

2 participants

@Andy-Jost @leofang