From ea7b66f5718fbc0fb0679f6ebda3f530b4adca4f Mon Sep 17 00:00:00 2001 From: David Seddon Date: Fri, 10 Jan 2025 14:57:18 +0000 Subject: [PATCH 1/2] Add docs for benchmarking --- CONTRIBUTING.rst | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 0d9811e7..dc934f70 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -88,3 +88,28 @@ To run a subset of tests:: To run all the test environments in *parallel* (you need to ``pip install detox``):: detox + + +Benchmarking +============ + +Codspeed +-------- + +A few benchmarks are run automatically on pull requests, using `Codspeed `. +Once the benchmarks have completed, a report will be included as a comment on the pull request. + +Codspeed also shows flame graphs which can help track down why a change might have impacted performance. + +Local benchmarking +------------------ + +It's also possible to run local benchmarks, which can be helpful if you want to quickly compare performance +across different versions of the code. + +To benchmark a particular version of the code, run ``tox -ebenchmark``. This command creates a report that will be +stored in a local file (ignored by Git). + +You can then see how your latest benchmark compares with earlier ones, by running +``pytest-benchmark compare --group-by=func --sort=name --columns=mean``. This will display a list of all the benchmarks +you've run locally, ordered from earlier to later. From a13039479946a8225a62b2ef1cb1ed1148b9d156 Mon Sep 17 00:00:00 2001 From: David Seddon Date: Fri, 10 Jan 2025 15:06:50 +0000 Subject: [PATCH 2/2] Add profiling docs --- CONTRIBUTING.rst | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index dc934f70..5ecedfc5 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -96,7 +96,7 @@ Benchmarking Codspeed -------- -A few benchmarks are run automatically on pull requests, using `Codspeed `. +A few benchmarks are run automatically on pull requests, using `Codspeed `_. Once the benchmarks have completed, a report will be included as a comment on the pull request. Codspeed also shows flame graphs which can help track down why a change might have impacted performance. @@ -110,6 +110,31 @@ across different versions of the code. To benchmark a particular version of the code, run ``tox -ebenchmark``. This command creates a report that will be stored in a local file (ignored by Git). -You can then see how your latest benchmark compares with earlier ones, by running -``pytest-benchmark compare --group-by=func --sort=name --columns=mean``. This will display a list of all the benchmarks -you've run locally, ordered from earlier to later. +You can then see how your latest benchmark compares with earlier ones, by running: + +``pytest-benchmark compare --group-by=func --sort=name --columns=mean`` + +This will display a list of all the benchmarks you've run locally, ordered from earlier to later. + +Profiling +========= + +Codspeed +-------- + +The easiest way to profile code is to look at the Codspeed flamegraph, automatically generated during benchmarking +(see above). + +Profiling Rust code locally +--------------------------- + +Rust integration tests can be profiled using `Cargo Flamegraph `_ +(which will need to be installed first, e.g. using ``cargo install flamegraph``). + +Navigate to the ``rust`` directory in this package. + +Run cargo flamegraph on the relevant test. E.g. to profile ``rust/tests/large.rs``, run: + +``sudo cargo flamegraph --root --test large`` + +This will create a file called ``flamegraph.svg``, which you can open to view the flamegraph. \ No newline at end of file