Skip to content

ci: Use codspeed for benchmarking#3071

Merged
larseggert merged 33 commits intomozilla:mainfrom
larseggert:ci-codspeed
Nov 11, 2025
Merged

ci: Use codspeed for benchmarking#3071
larseggert merged 33 commits intomozilla:mainfrom
larseggert:ci-codspeed

Conversation

@larseggert
Copy link
Collaborator

@larseggert larseggert commented Oct 24, 2025 •
edited
Loading

Basic codspeed benchmarking framework. Uploads to https://codspeed.io/mozilla/neqo successfully.

At the moment, runs in parallel to the existing benchmarks. With more work (in separate PRs), might make some of them superfluous, and possibly obviate the need for https://bencher.dev.

Items to discuss:

@codecov
Copy link

codecov bot commented Oct 24, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.37%. Comparing base (12c5f48) to head (4f9e5e6).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #3071      +/-   ##
==========================================
- Coverage   93.43%   93.37%   -0.06%     
==========================================
  Files         124      124              
  Lines       36234    36234              
  Branches    36234    36234              
==========================================
- Hits        33854    33835      -19     
- Misses       1531     1552      +21     
+ Partials      849      847       -2
Components Coverage Δ
neqo-common 97.32% <ø> (ø)
neqo-crypto 83.25% <ø> (-0.48%) ⬇️
neqo-http3 93.29% <ø> (ø)
neqo-qpack 94.18% <ø> (ø)
neqo-transport 94.47% <ø> (-0.06%) ⬇️
neqo-udp 79.42% <ø> (+0.47%) ⬆️
mtu 85.76% <ø> (ø)

@github-actions
Copy link
Contributor

github-actions bot commented Oct 24, 2025 •
edited
Loading

🐰 Bencher Report

Branchci-codspeed
TestbedOn-prem

🚨 1 Alert

BenchmarkMeasure
Units		ViewBenchmark Result
(Result Δ%)		Upper Boundary
(Limit %)
coalesce_acked_from_zero 1+1 entriesLatency
nanoseconds (ns)		📈 plot
🚷 threshold
🚨 alert (🔔)		90.13 ns
(+1.43%)Baseline: 88.86 ns
90.12 ns
(100.00%)

Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client📈 view plot
🚷 view threshold		205,490,000.00 ns
(-0.60%)Baseline: 206,723,224.85 ns
216,497,546.13 ns
(94.92%)
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client📈 view plot
🚷 view threshold		200,240,000.00 ns
(-0.33%)Baseline: 200,908,313.61 ns
211,271,433.69 ns
(94.78%)
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client📈 view plot
🚷 view threshold		38,529,000.00 ns
(+22.26%)Baseline: 31,514,127.22 ns
42,633,418.24 ns
(90.37%)
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client📈 view plot
🚷 view threshold		288,440,000.00 ns
(-1.03%)Baseline: 291,439,349.11 ns
303,755,822.80 ns
(94.96%)
1-streams/each-1000-bytes/simulated-time📈 view plot
🚷 view threshold		119,090,000.00 ns
(+0.36%)Baseline: 118,667,100.59 ns
120,722,042.51 ns
(98.65%)
1-streams/each-1000-bytes/wallclock-time📈 view plot
🚷 view threshold		593,410.00 ns
(+0.10%)Baseline: 592,819.11 ns
615,719.37 ns
(96.38%)
1000-streams/each-1-bytes/simulated-time📈 view plot
🚷 view threshold		2,332,800,000.00 ns
(-83.00%)Baseline: 13,722,073,372.78 ns
22,625,261,311.11 ns
(10.31%)
1000-streams/each-1-bytes/wallclock-time📈 view plot
🚷 view threshold		12,777,000.00 ns
(-7.85%)Baseline: 13,865,532.54 ns
15,059,249.83 ns
(84.84%)
1000-streams/each-1000-bytes/simulated-time📈 view plot
🚷 view threshold		16,390,000,000.00 ns
(-12.39%)Baseline: 18,707,168,639.05 ns
20,525,616,911.04 ns
(79.85%)
1000-streams/each-1000-bytes/wallclock-time📈 view plot
🚷 view threshold		50,547,000.00 ns
(-0.25%)Baseline: 50,675,346.15 ns
57,142,600.18 ns
(88.46%)
RxStreamOrderer::inbound_frame()📈 view plot
🚷 view threshold		110,140,000.00 ns
(+0.41%)Baseline: 109,690,857.99 ns
111,598,484.14 ns
(98.69%)
coalesce_acked_from_zero 1+1 entries📈 view plot
🚷 view threshold
🚨 view alert (🔔)		90.13 ns
(+1.43%)Baseline: 88.86 ns
90.12 ns
(100.00%)

coalesce_acked_from_zero 10+1 entries📈 view plot
🚷 view threshold		106.60 ns
(+0.49%)Baseline: 106.08 ns
107.23 ns
(99.41%)
coalesce_acked_from_zero 1000+1 entries📈 view plot
🚷 view threshold		93.02 ns
(+3.04%)Baseline: 90.28 ns
94.86 ns
(98.06%)
coalesce_acked_from_zero 3+1 entries📈 view plot
🚷 view threshold		107.13 ns
(+0.50%)Baseline: 106.60 ns
107.69 ns
(99.48%)
decode 1048576 bytes, mask 3f📈 view plot
🚷 view threshold		1,757,900.00 ns
(+7.71%)Baseline: 1,632,085.21 ns
1,802,391.92 ns
(97.53%)
decode 1048576 bytes, mask 7f📈 view plot
🚷 view threshold		5,040,900.00 ns
(-0.51%)Baseline: 5,066,831.36 ns
5,113,485.38 ns
(98.58%)
decode 1048576 bytes, mask ff📈 view plot
🚷 view threshold		3,002,600.00 ns
(-0.90%)Baseline: 3,029,882.84 ns
3,053,037.90 ns
(98.35%)
decode 4096 bytes, mask 3f📈 view plot
🚷 view threshold		6,222.20 ns
(-15.69%)Baseline: 7,379.86 ns
10,405.12 ns
(59.80%)
decode 4096 bytes, mask 7f📈 view plot
🚷 view threshold		19,641.00 ns
(-0.83%)Baseline: 19,806.20 ns
20,472.37 ns
(95.94%)
decode 4096 bytes, mask ff📈 view plot
🚷 view threshold		11,376.00 ns
(+0.09%)Baseline: 11,365.57 ns
12,534.14 ns
(90.76%)
sent::Packets::take_ranges📈 view plot
🚷 view threshold		4,448.90 ns
(-5.80%)Baseline: 4,722.71 ns
4,961.98 ns
(89.66%)
transfer/pacing-false/same-seed/simulated-time/run📈 view plot
🚷 view threshold		25,234,000,000.00 ns
(-0.71%)Baseline: 25,413,872,023.81 ns
26,036,411,011.75 ns
(96.92%)
transfer/pacing-false/same-seed/wallclock-time/run📈 view plot
🚷 view threshold		24,678,000.00 ns
(-3.99%)Baseline: 25,703,785.71 ns
27,102,712.33 ns
(91.05%)
transfer/pacing-false/varying-seeds/simulated-time/run📈 view plot
🚷 view threshold		25,205,000,000.00 ns
(+0.12%)Baseline: 25,175,038,690.48 ns
25,224,153,410.51 ns
(99.92%)
transfer/pacing-false/varying-seeds/wallclock-time/run📈 view plot
🚷 view threshold		24,981,000.00 ns
(-3.10%)Baseline: 25,780,541.67 ns
27,427,465.87 ns
(91.08%)
transfer/pacing-true/same-seed/simulated-time/run📈 view plot
🚷 view threshold		25,301,000,000.00 ns
(-1.10%)Baseline: 25,583,244,047.62 ns
25,878,219,628.17 ns
(97.77%)
transfer/pacing-true/same-seed/wallclock-time/run📈 view plot
🚷 view threshold		26,320,000.00 ns
(-2.68%)Baseline: 27,045,258.93 ns
28,614,184.46 ns
(91.98%)
transfer/pacing-true/varying-seeds/simulated-time/run📈 view plot
🚷 view threshold		25,006,000,000.00 ns
(+0.04%)Baseline: 24,995,136,904.76 ns
25,043,665,250.92 ns
(99.85%)
transfer/pacing-true/varying-seeds/wallclock-time/run📈 view plot
🚷 view threshold		25,487,000.00 ns
(-3.02%)Baseline: 26,280,863.10 ns
28,011,872.68 ns
(90.99%)
🐰 View full continuous benchmarking report in Bencher

@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2025 •
edited
Loading

🐰 Bencher Report

Branchci-codspeed
TestbedOn-prem

🚨 1 Alert

IterationBenchmarkMeasure
Units		ViewBenchmark Result
(Result Δ%)		Upper Boundary
(Limit %)
2neqo vs. google (cubic, paced)Latency
milliseconds (ms)		📈 plot
🚷 threshold
🚨 alert (🔔)		778.36 ms
(+2.49%)Baseline: 759.48 ms
774.21 ms
(100.54%)

Click to view all benchmark results
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
google vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		280.84 ms
(+0.92%)Baseline: 278.28 ms
282.62 ms
(99.37%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
msquic vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		226.40 ms
(+14.02%)Baseline: 198.57 ms
235.79 ms
(96.01%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. google (cubic, paced)📈 view plot
🚷 view threshold
🚨 view alert (🔔)		778.36 ms
(+2.49%)Baseline: 759.48 ms
774.21 ms
(100.54%)

BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. msquic (cubic, paced)📈 view plot
🚷 view threshold		157.69 ms
(-0.07%)Baseline: 157.79 ms
160.65 ms
(98.15%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (cubic)📈 view plot
🚷 view threshold		94.61 ms
(+3.43%)Baseline: 91.47 ms
96.79 ms
(97.75%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		95.19 ms
(+2.52%)Baseline: 92.85 ms
98.08 ms
(97.05%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (reno)📈 view plot
🚷 view threshold		94.25 ms
(+3.02%)Baseline: 91.49 ms
96.67 ms
(97.49%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (reno, paced)📈 view plot
🚷 view threshold		94.62 ms
(+2.05%)Baseline: 92.72 ms
97.71 ms
(96.84%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. quiche (cubic, paced)📈 view plot
🚷 view threshold		194.17 ms
(+0.27%)Baseline: 193.64 ms
196.99 ms
(98.57%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. s2n (cubic, paced)📈 view plot
🚷 view threshold		222.71 ms
(+0.72%)Baseline: 221.12 ms
224.09 ms
(99.38%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
quiche vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		154.58 ms
(+1.03%)Baseline: 153.01 ms
158.22 ms
(97.70%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
s2n vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		172.58 ms
(-0.72%)Baseline: 173.83 ms
178.01 ms
(96.95%)
🐰 View full continuous benchmarking report in Bencher

@codspeed-hq
Copy link

codspeed-hq bot commented Oct 29, 2025 •
edited
Loading

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 23 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

@larseggert larseggert marked this pull request as ready for review October 29, 2025 11:30
Copilot AI review requested due to automatic review settings October 29, 2025 11:30
@larseggert larseggert requested a review from mxinden as a code owner October 29, 2025 11:30
Copilot AI reviewed Oct 29, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@larseggert larseggert requested a review from Copilot October 29, 2025 12:37
Copilot AI reviewed Oct 29, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings October 29, 2025 13:59
Copilot AI reviewed Oct 29, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

@larseggert
Update .github/workflows/codspeed.yml
8638d28 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Lars Eggert <lars@eggert.org>
Copilot AI review requested due to automatic review settings October 29, 2025 14:20
Copilot AI reviewed Oct 29, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

@larseggert
Update .github/workflows/codspeed.yml
d97578d 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Lars Eggert <lars@eggert.org>
Copilot AI reviewed Oct 30, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

@larseggert
Update .github/workflows/codspeed.yml
b0a57e7 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Lars Eggert <lars@eggert.org>
Copilot AI review requested due to automatic review settings October 30, 2025 09:09
Copilot AI reviewed Oct 30, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

@larseggert
Copy link
Collaborator Author

larseggert commented Oct 30, 2025

@mxinden would appreciate a review here – this needs to be merged before the dashboard becomes useful.

mxinden
mxinden reviewed Oct 31, 2025
View reviewed changes
Copy link
Member

@mxinden mxinden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In favor to experiment with codspeed.

Are all the Cargo.lock changes needed?

@larseggert
Copy link
Collaborator Author

larseggert commented Nov 5, 2025

@mxinden are we OK with the dependencies this seems to need?

@larseggert
Copy link
Collaborator Author

larseggert commented Nov 6, 2025

I submitted CodSpeedHQ/codspeed-rust#148 to reduce the dependencies pulled in by Codspeed.

@larseggert larseggert requested review from Copilot and mxinden November 7, 2025 21:03
Copilot AI reviewed Nov 7, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.

@larseggert
Merge branch 'main' into ci-codspeed
7fa5d75 
Signed-off-by: Lars Eggert <lars@eggert.org>
@larseggert
Merge
46ace57
Copilot AI review requested due to automatic review settings November 7, 2025 21:08
Copilot AI reviewed Nov 7, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

mxinden
mxinden approved these changes Nov 11, 2025
View reviewed changes
@larseggert
Remove FIXME
7aeece4
@larseggert
Merge branch 'main' into ci-codspeed
c084f2b 
Signed-off-by: Lars Eggert <lars@eggert.org>
Copilot AI review requested due to automatic review settings November 11, 2025 13:38
Copilot AI reviewed Nov 11, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

@larseggert larseggert enabled auto-merge November 11, 2025 13:51
@larseggert larseggert added this pull request to the merge queue Nov 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2025

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to 12c5f48.

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Merged via the queue into mozilla:main with commit b9c32c7 Nov 11, 2025
106 of 107 checks passed
@larseggert larseggert deleted the ci-codspeed branch November 11, 2025 14:14
@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2025

Benchmark results

Performance differences relative to b5196e7.

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: Change within noise threshold.
       time:   [199.81 ms 200.24 ms 200.71 ms]
       thrpt:  [498.24 MiB/s 499.40 MiB/s 500.48 MiB/s]
change:
       time:   [-0.6122% -0.3666% -0.1011%] (p = 0.00 < 0.05)
       thrpt:  [+0.1013% +0.3680% +0.6160%]

Found 2 outliers among 100 measurements (2.00%)

1 (1.00%) high mild

1 (1.00%) high severe

1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.
       time:   [286.72 ms 288.44 ms 290.22 ms]
       thrpt:  [34.457 Kelem/s 34.669 Kelem/s 34.877 Kelem/s]
change:
       time:   [-0.2701% +0.5925% +1.4508%] (p = 0.18 > 0.05)
       thrpt:  [-1.4301% -0.5891% +0.2708%]

Found 2 outliers among 100 measurements (2.00%)

2 (2.00%) high mild

1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
       time:   [38.384 ms 38.529 ms 38.688 ms]
       thrpt:  [25.848   B/s 25.955   B/s 26.053   B/s]
change:
       time:   [-0.5197% -0.0131% +0.5375%] (p = 0.96 > 0.05)
       thrpt:  [-0.5347% +0.0131% +0.5224%]

Found 5 outliers among 100 measurements (5.00%)

2 (2.00%) high mild

3 (3.00%) high severe

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: 💔 Performance has regressed.
       time:   [205.19 ms 205.49 ms 205.82 ms]
       thrpt:  [485.85 MiB/s 486.65 MiB/s 487.36 MiB/s]
change:
       time:   [+1.4566% +1.6588% +1.8753%] (p = 0.00 < 0.05)
       thrpt:  [-1.8408% -1.6317% -1.4357%]

Found 6 outliers among 100 measurements (6.00%)

3 (3.00%) high mild

3 (3.00%) high severe

decode 4096 bytes, mask ff: 💚 Performance has improved.
       time:   [11.325 µs 11.376 µs 11.438 µs]
       change: [-4.3214% -2.8037% -1.2963%] (p = 0.00 < 0.05)

Found 15 outliers among 100 measurements (15.00%)

2 (2.00%) low severe

2 (2.00%) low mild

1 (1.00%) high mild

10 (10.00%) high severe

decode 1048576 bytes, mask ff: No change in performance detected.
       time:   [2.9930 ms 3.0026 ms 3.0138 ms]
       change: [-0.8940% -0.4034% +0.0455%] (p = 0.10 > 0.05)

Found 9 outliers among 100 measurements (9.00%)

9 (9.00%) high severe

decode 4096 bytes, mask 7f: 💚 Performance has improved.
       time:   [19.580 µs 19.641 µs 19.715 µs]
       change: [-2.9726% -2.1841% -1.5326%] (p = 0.00 < 0.05)

Found 14 outliers among 100 measurements (14.00%)

1 (1.00%) low mild

13 (13.00%) high severe

decode 1048576 bytes, mask 7f: Change within noise threshold.
       time:   [5.0295 ms 5.0409 ms 5.0539 ms]
       change: [-1.5239% -1.1869% -0.8286%] (p = 0.00 < 0.05)

Found 13 outliers among 100 measurements (13.00%)

13 (13.00%) high severe

decode 4096 bytes, mask 3f: 💚 Performance has improved.
       time:   [6.2002 µs 6.2222 µs 6.2513 µs]
       change: [-25.315% -24.730% -24.081%] (p = 0.00 < 0.05)

Found 10 outliers among 100 measurements (10.00%)

2 (2.00%) low mild

1 (1.00%) high mild

7 (7.00%) high severe

decode 1048576 bytes, mask 3f: Change within noise threshold.
       time:   [1.7577 ms 1.7579 ms 1.7580 ms]
       change: [-2.1691% -1.5603% -0.9571%] (p = 0.00 < 0.05)

Found 4 outliers among 100 measurements (4.00%)

1 (1.00%) low mild

3 (3.00%) high mild

1-streams/each-1000-bytes/wallclock-time: Change within noise threshold.
       time:   [589.78 µs 593.41 µs 597.20 µs]
       change: [+0.5340% +1.2473% +2.0455%] (p = 0.00 < 0.05)
1-streams/each-1000-bytes/simulated-time: No change in performance detected.
       time:   [118.86 ms 119.09 ms 119.31 ms]
       thrpt:  [8.1852 KiB/s 8.2004 KiB/s 8.2158 KiB/s]
change:
       time:   [-0.3195% -0.0347% +0.2355%] (p = 0.80 > 0.05)
       thrpt:  [-0.2349% +0.0347% +0.3205%]

Found 1 outliers among 100 measurements (1.00%)

1 (1.00%) low mild

1000-streams/each-1-bytes/wallclock-time: Change within noise threshold.
       time:   [12.740 ms 12.777 ms 12.814 ms]
       change: [-1.1180% -0.7125% -0.2903%] (p = 0.00 < 0.05)
1000-streams/each-1-bytes/simulated-time: No change in performance detected.
       time:   [2.3286 s 2.3328 s 2.3371 s]
       thrpt:  [427.89   B/s 428.66   B/s 429.43   B/s]
change:
       time:   [-0.2490% -0.0034% +0.2381%] (p = 0.98 > 0.05)
       thrpt:  [-0.2376% +0.0034% +0.2496%]

Found 1 outliers among 100 measurements (1.00%)

1 (1.00%) high mild

1000-streams/each-1000-bytes/wallclock-time: 💚 Performance has improved.
       time:   [50.440 ms 50.547 ms 50.658 ms]
       change: [-2.6972% -2.3874% -2.0998%] (p = 0.00 < 0.05)

Found 1 outliers among 100 measurements (1.00%)

1 (1.00%) high mild

1000-streams/each-1000-bytes/simulated-time

time:   [16.119 s 16.390 s 16.657 s]

thrpt:  [58.629 KiB/s 59.583 KiB/s 60.586 KiB/s]

change:

time:   [-2.4660% -0.2345% +1.8573%] (p = 0.84 > 0.05)

thrpt:  [-1.8234% +0.2350% +2.5283%]

No change in performance detected.

Found 2 outliers among 100 measurements (2.00%)

2 (2.00%) low mild

coalesce_acked_from_zero 1+1 entries: No change in performance detected.
       time:   [89.765 ns 90.126 ns 90.479 ns]
       change: [-2.5236% -0.7825% +0.2880%] (p = 0.40 > 0.05)

Found 11 outliers among 100 measurements (11.00%)

10 (10.00%) high mild

1 (1.00%) high severe

coalesce_acked_from_zero 3+1 entries: No change in performance detected.
       time:   [106.75 ns 107.13 ns 107.54 ns]
       change: [-1.9386% -0.8004% -0.0135%] (p = 0.10 > 0.05)

Found 15 outliers among 100 measurements (15.00%)

2 (2.00%) low mild

3 (3.00%) high mild

10 (10.00%) high severe

coalesce_acked_from_zero 10+1 entries: No change in performance detected.
       time:   [106.33 ns 106.60 ns 106.97 ns]
       change: [-1.4213% -0.5587% +0.2660%] (p = 0.19 > 0.05)

Found 15 outliers among 100 measurements (15.00%)

4 (4.00%) low mild

5 (5.00%) high mild

6 (6.00%) high severe

coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
       time:   [92.935 ns 93.024 ns 93.128 ns]
       change: [-0.2789% +0.3644% +0.9785%] (p = 0.26 > 0.05)

Found 13 outliers among 100 measurements (13.00%)

5 (5.00%) high mild

8 (8.00%) high severe

RxStreamOrderer::inbound_frame(): Change within noise threshold.
       time:   [110.04 ms 110.14 ms 110.25 ms]
       change: [+0.2676% +0.4063% +0.5442%] (p = 0.00 < 0.05)

Found 4 outliers among 100 measurements (4.00%)

2 (2.00%) low mild

1 (1.00%) high mild

1 (1.00%) high severe

sent::Packets::take_ranges: :green_heart: Performance has improved.
       time:   [4.3733 µs 4.4489 µs 4.5127 µs]
       change: [-6.2126% -3.7066% -1.0306%] (p = 0.01 < 0.05)

Found 1 outliers among 100 measurements (1.00%)

1 (1.00%) high mild

transfer/pacing-false/varying-seeds/wallclock-time/run: Change within noise threshold.
       time:   [24.933 ms 24.981 ms 25.037 ms]
       change: [-3.1851% -2.9267% -2.6552%] (p = 0.00 < 0.05)

Found 2 outliers among 100 measurements (2.00%)

1 (1.00%) high mild

1 (1.00%) high severe

transfer/pacing-false/varying-seeds/simulated-time/run: No change in performance detected.
       time:   [25.169 s 25.205 s 25.241 s]
       thrpt:  [162.27 KiB/s 162.51 KiB/s 162.74 KiB/s]
change:
       time:   [-0.0800% +0.1181% +0.3189%] (p = 0.24 > 0.05)
       thrpt:  [-0.3179% -0.1180% +0.0800%]
transfer/pacing-true/varying-seeds/wallclock-time/run: Change within noise threshold.
       time:   [25.429 ms 25.487 ms 25.546 ms]
       change: [-2.9971% -2.6357% -2.2850%] (p = 0.00 < 0.05)

Found 1 outliers among 100 measurements (1.00%)

1 (1.00%) high mild

transfer/pacing-true/varying-seeds/simulated-time/run: No change in performance detected.
       time:   [24.961 s 25.006 s 25.052 s]
       thrpt:  [163.50 KiB/s 163.80 KiB/s 164.10 KiB/s]
change:
       time:   [-0.1640% +0.0455% +0.2865%] (p = 0.70 > 0.05)
       thrpt:  [-0.2857% -0.0454% +0.1642%]

Found 1 outliers among 100 measurements (1.00%)

1 (1.00%) high mild

transfer/pacing-false/same-seed/wallclock-time/run: Change within noise threshold.
       time:   [24.662 ms 24.678 ms 24.694 ms]
       change: [-2.7208% -2.5672% -2.4392%] (p = 0.00 < 0.05)

Found 1 outliers among 100 measurements (1.00%)

1 (1.00%) high mild

transfer/pacing-false/same-seed/simulated-time/run: No change in performance detected.
       time:   [25.234 s 25.234 s 25.234 s]
       thrpt:  [162.32 KiB/s 162.32 KiB/s 162.32 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000%]
transfer/pacing-true/same-seed/wallclock-time/run: 💚 Performance has improved.
       time:   [26.303 ms 26.320 ms 26.337 ms]
       change: [-3.8496% -3.6800% -3.5440%] (p = 0.00 < 0.05)

Found 2 outliers among 100 measurements (2.00%)

2 (2.00%) high mild

transfer/pacing-true/same-seed/simulated-time/run: No change in performance detected.
       time:   [25.301 s 25.301 s 25.301 s]
       thrpt:  [161.89 KiB/s 161.89 KiB/s 161.89 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000%]

Download data for profiler.firefox.com or download performance comparison data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@mxinden mxinden mxinden approved these changes

@KershawChang KershawChang Awaiting requested review from KershawChang KershawChang is a code owner

@martinthomson martinthomson Awaiting requested review from martinthomson martinthomson is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@larseggert @mxinden