Skip to content

Split sublibrary CI to only test on dependency changes#3142

Merged
ChrisRackauckas merged 6 commits intoSciML:masterfrom
ChrisRackauckas-Claude:sublibrary-ci-change-detection
Mar 17, 2026
Merged

Split sublibrary CI to only test on dependency changes#3142
ChrisRackauckas merged 6 commits intoSciML:masterfrom
ChrisRackauckas-Claude:sublibrary-ci-change-detection

Conversation

@ChrisRackauckas-Claude
Copy link
Contributor

Summary

  • Splits sublibrary tests out of CI.yml into a new SublibraryCI.yml workflow that only runs tests for sublibraries affected by changed files
  • Adds .github/scripts/compute_affected_sublibraries.py which reads the dependency graph from all lib/*/Project.toml files (both [deps] and [extras]), computes the transitive reverse-dependency closure, and maps changed file paths to affected sublibraries
  • Global tests always run: Interface, Integrators, AlgConvergence, Regression, Downstream, ModelingToolkit, QA, AD — unchanged in CI.yml

How it works

  1. detect-changes job runs git diff and pipes changed files into the Python script
  2. Script parses all lib/*/Project.toml to build the internal dependency graph
  3. For each changed lib/<pkg>/... file, the script includes that package plus all transitive reverse dependencies
  4. The affected list becomes a dynamic GitHub Actions matrix for both functional tests (4 Julia versions) and QA tests (Julia 1 only)

Example behavior

Changed file Sublibraries tested
lib/OrdinaryDiffEqCore/src/... All 36 sublibraries
lib/OrdinaryDiffEqTsit5/src/... Tsit5 + Default + Nordsieck
lib/OrdinaryDiffEqFeagin/src/... Only Feagin
test/interface/... or src/... No sublibrary tests (global CI still runs)

Benefits

  • Massive CI savings on targeted changes — e.g., a change to OrdinaryDiffEqFeagin only runs 5 jobs (4 versions + QA) instead of ~170
  • Self-maintaining — new sublibraries or dependency changes are automatically picked up from Project.toml
  • Eliminates ~100 QA exclude entries — QA is now a separate job that only runs on Julia 1

Test plan

  • Verify detect-changes job correctly identifies affected sublibraries in CI logs
  • Verify sublibrary tests still pass for affected packages
  • Verify global tests in CI.yml are unaffected
  • Verify that changes only to test/ or src/ (non-lib) don't trigger sublibrary tests

🤖 Generated with Claude Code

ChrisRackauckas and others added 3 commits March 17, 2026 08:45
Move sublibrary functional and QA tests from CI.yml into a new
SublibraryCI.yml workflow that only runs tests for sublibraries
affected by the changed files. A Python script reads the dependency
graph from lib/*/Project.toml, computes transitive reverse dependencies,
and outputs the affected sublibraries as a dynamic matrix.

Global tests (Interface, Integrators, Regression, etc.) always run
in CI.yml regardless of which files changed.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace compute_affected_sublibraries.py with .jl equivalent using
stdlib TOML. Add OrdinaryDiffEqCore README to trigger all sublibrary
tests on this PR as a verification.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sublibraries can now declare custom test groups (beyond FUNCTIONAL/QA)
by adding a test/test_groups.toml file:

  [FUNCTIONAL]
  versions = ["lts", "1.11", "1", "pre"]

  [QA]
  versions = ["1"]

  [GPU]
  versions = ["1"]

The detection script reads these to build the full matrix of
(group, version) pairs. Default remains FUNCTIONAL + QA when
no test_groups.toml exists.

Also updates runtests.jl to generically detect any _{SUFFIX} group
by checking against actual lib/ directories, replacing the hardcoded
_QA suffix check. This means GROUP="OrdinaryDiffEqFoo_GPU" will
correctly resolve to base_group="OrdinaryDiffEqFoo" with
ODEDIFFEQ_TEST_GROUP="GPU".

The workflow is simplified to a single test job consuming the full
matrix include list from the detection script.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude
Copy link
Contributor Author

Updated to:

  1. Julia script instead of Python — uses stdlib TOML parser, no external deps
  2. Sublibraries can declare custom test groups via test/test_groups.toml (e.g., add a GPU group with its own Julia version requirements)
  3. Generic group suffix detection in runtests.jl — replaces the hardcoded _QA check with a lookup against actual lib/ directories, so GROUP=OrdinaryDiffEqFoo_GPU correctly resolves
  4. Single unified test job — the script outputs the full {group, version} matrix, eliminating the separate test/qa jobs
  5. OrdinaryDiffEqCore README change included to trigger all 36 sublibrary tests for verification

@oscardssmith
Copy link
Member

This is a very nice change. Good idea.

@ChrisRackauckas
Copy link
Member

This was needed to complete the StochasticDiffEq and DelayDiffEq stuff haha.

ChrisRackauckas and others added 3 commits March 17, 2026 09:19
Rename the default test group from "FUNCTIONAL" to "Core" across all
36 sublibrary runtests.jl files, the top-level runtests.jl, the
detection script, and DowngradeSublibraries.yml.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The while loop inside @time's local scope caused idx to be treated as
a new local variable, triggering UndefVarError. Replace with a for loop
over character indices to avoid the reassignment scoping issue.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GitHub Actions exclude doesn't work with include-only matrices and
silently prevents job creation. Move the BDF pre-release exclusion
into the Julia script's EXCLUDES set instead.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ChrisRackauckas ChrisRackauckas merged commit 509f956 into SciML:master Mar 17, 2026
271 of 276 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants