Add automated notebook testing with Papermill#602
Add automated notebook testing with Papermill#602drbenvincent wants to merge 29 commits intomainfrom
Conversation
Introduces a GitHub Actions workflow to run and validate Jupyter notebooks in CI using a new runner script. Adds scripts to mock PyMC sampling for faster execution, updates test dependencies to include papermill, and documents the notebook runner usage. Also updates the interrogate badge to reflect new coverage.
There was a problem hiding this comment.
Pull request overview
This PR introduces automated testing for Jupyter notebooks in CI using Papermill. The implementation includes a runner script that mocks PyMC's MCMC sampling with faster prior predictive sampling to validate notebooks execute without errors.
Key changes:
- New notebook runner script with filtering capabilities for different notebook types
- Mock PyMC sampling implementation that replaces expensive MCMC with prior predictive sampling (10 draws)
- GitHub Actions workflow that runs notebooks in parallel across three categories (PyMC, sklearn, and other notebooks)
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/run_notebooks/runner.py | Main script for executing notebooks with Papermill, includes filtering and logging |
| scripts/run_notebooks/injected.py | Mock implementation of pm.sample that uses prior predictive sampling |
| scripts/run_notebooks/README.md | Documentation for the notebook runner usage and CI integration |
| .github/workflows/test_notebook.yml | GitHub Actions workflow for parallel notebook testing |
| pyproject.toml | Adds papermill to test dependencies |
| docs/source/_static/interrogate_badge.svg | Updates documentation coverage badge from 96.3% to 96.0% |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #602 +/- ##
=======================================
Coverage 94.35% 94.35%
=======================================
Files 44 44
Lines 7517 7517
Branches 456 456
=======================================
Hits 7093 7093
Misses 262 262
Partials 162 162 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Updated the mock for pm.sample() to use 100 draws instead of 50 for prior predictive sampling, as reflected in both the injected script and documentation. This change aims to provide more robust validation during notebook execution.
Updated the mock for pm.sample to use 500 draws instead of 100 to ensure compatibility with notebook code that iterates over posterior samples, such as plot_ate which defaults to 500 draws. Adjusted documentation and injected.py accordingly.
Introduces skip_notebooks.yml to specify notebooks incompatible with prior predictive sampling mock. Updates runner.py to filter out these notebooks and reduces MIN_DRAWS from 500 to 100 for faster execution.
Replaces import of LinearRegression from causalpy.skl_models with sklearn's LinearRegression and removes execution count from the first code cell.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Replaces hardcoded sample size with dynamic calculation based on the length of 'uncertainty' to prevent errors when fewer than 500 samples are available. Also resets execution count to null for the affected notebook cell.
Expanded the skip_notebooks.yml file to include iv_pymc.ipynb, which requires JAX not available in the CI test environment. Updated comments to clarify reasons for skipping each notebook.
Installs Graphviz as a system dependency in the test_notebook GitHub Actions workflow to support notebooks or tests that require it.
|
passing! @NathanielF See |
|
bugbot review |
PR SummaryIntroduces CI to validate docs notebooks execute without errors.
Written by Cursor Bugbot for commit c15a229. This will update automatically on new commits. Configure here. |
|
Papermill has good support for the trick where we change sampling behaviour to sample from the prior for ultra fast evaluation |
Add nbclient widget output guards, clear notebook outputs before injection, and support optional parallel execution to reduce notebook flakiness in CI.
Use the requested draw count when provided and fall back to the minimum to avoid indexing errors in notebooks that iterate over many draws.
Summary
Test Plan
📚 Documentation preview 📚: https://causalpy--602.org.readthedocs.build/en/602/