perf: Cache redundant work in transform phase#40
Open
danceratopz wants to merge 14 commits intoSamWilsn:masterfrom
Open
perf: Cache redundant work in transform phase#40danceratopz wants to merge 14 commits intoSamWilsn:masterfrom
danceratopz wants to merge 14 commits intoSamWilsn:masterfrom
Conversation
- Use explicit patterns instead of recursive ** globs. - Setuptools doesn't reliably include files with ** in sdist builds.
- Add dedicated test extras with pytest and pytest-cov.
- Add test extras to both test and type environments so pytest and pytest-cov are available alongside lint deps.
- Configure pytest to run with coverage enabled. - Set minimum coverage threshold to 80%. - Exclude common non-testable patterns from coverage.
- Add tests for CLI, settings, context, and document modules. - Add tests for plugin loader and transform pipeline. - Add tests for HTML, mistletoe, and verbatim plugins. - Add end-to-end HTML rendering pipeline tests. - Add tests for Python CST parsing and node types. - Add tests for references, search, and resources plugins. - Add integration tests for end-to-end workflows. - Add behavior-level pipeline contract tests. - Achieve 90% code coverage.
The test_enter_returning_tag_pushes_and_traverses test was directly assigning a fake renderer into visitor.renderers, which with the shared _LOADED_RENDERERS cache now persists across all subsequent HTMLVisitor instances. This caused test_definition_to_html_output to use the fake ListNode renderer instead of the real one, producing empty HTML output. Wrap the fake renderer injection in patch.dict() so it is automatically cleaned up after the test completes.
Author
|
Ah nice, I didn't see that fc0d5eb had already landed, will rebase on main tomorrow 🥱 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requires #37
Summary
Add module-level and instance-level caching to seven hot paths identified via profiling. Each optimization eliminates redundant work that was repeated per-document or per-node during the build and transform phases.
What changed
entry_points()in HTMLVisitor — Module-level_get_html_entry_points()replaces per-instanceentry_points(group="docc.plugins.html")calls.entry_points()in Loader — Module-level_get_plugin_entry_points()replaces per-instanceentry_points(group="docc.plugins")calls.dataclasses.fields()in PythonNode — Class-level_fields_cacheClassVar replaces per-callfields(self)inchildrenandreplace_child._BoundsVisitorresults in VerbatimVisitor — Instance-level_bounds_cacheeliminates double tree traversal inenter()/exit().TextSource.line()— Lazy_lines_cachestores file content on first access, eliminating re-reads perline()call._get_jinja_env()(HTML) and_get_listing_env()(listing) replace per-renderEnvironment(...)construction._LOADED_RENDERERSdict replaces per-instanceself.renderers = {}, soEntryPoint.load()runs at most once per node type.Why
The transform phase dominates runtime (~87% of wall time). Profiling showed that
entry_points(),dataclasses.fields(), Jinja2Environmentconstruction,_BoundsVisitortraversal, file I/O inTextSource.line(), andEntryPoint.load()were all called thousands of times with identical arguments across documents. Caching these at the appropriate scope (module, class, or instance) eliminates the redundant work.Test coverage
Each optimization includes three categories of tests:
unittest.mock.patchto verify the expensive call happens only once when multiple consumers access the cache.Benchmark
Command:
Individual optimizations
Combined result
Per-phase breakdown (combined)