Skip to content

Comments

fix: benchmark overview shows wrong latency metric#245

Merged
cuioss-oliver merged 2 commits intomainfrom
fix/benchmark-latency-metric-selection
Feb 23, 2026
Merged

fix: benchmark overview shows wrong latency metric#245
cuioss-oliver merged 2 commits intomainfrom
fix/benchmark-latency-metric-selection

Conversation

@cuioss-oliver
Copy link
Collaborator

Summary

  • Fixed: JmhBenchmarkConverter.createOverview() now uses the configured benchmark names from ReportConfiguration instead of blindly picking the minimum-scoring avgt benchmark. This ensures the dashboard shows the correct measureAverageTime as the latency metric, not measureConcurrentValidation.
  • Removed redundant benchmark: measureConcurrentValidation was identical to measureAverageTime (both called validateWithFullSpectrum()/validateWithRotation() which were copy-paste identical). Removed from SimpleCoreValidationBenchmark, CoreJfrBenchmark, and UnifiedJfrBenchmark.
  • Removed dead code: CoreValidationDelegate.validateWithRotation() was identical to validateWithFullSpectrum().

Test plan

  • ./mvnw clean install -pl benchmarking/benchmarking-common — 181 tests pass
  • ./mvnw clean install -pl benchmarking/benchmark-core — 2 tests pass, compilation clean
  • Existing converter test shouldCalculatePerformanceScoreAccordingToDocumentation verifies correct benchmark selection from real JMH data
  • CI pipeline validates full build

🤖 Generated with Claude Code

JmhBenchmarkConverter.createOverview() was using a heuristic (min avgt
score) to select the latency benchmark, ignoring the explicit names
configured in ReportConfiguration. This caused the dashboard to show
measureConcurrentValidation instead of the configured measureAverageTime.

Changes:
- JmhBenchmarkConverter: accept configured benchmark names and prefer
  them over the min/max heuristic in createOverview()
- BenchmarkResultProcessor: pass ReportConfiguration to the converter
- Remove redundant measureConcurrentValidation benchmark (identical to
  measureAverageTime) from SimpleCoreValidationBenchmark, CoreJfrBenchmark,
  and UnifiedJfrBenchmark
- Remove identical validateWithRotation() from CoreValidationDelegate
- Update test data and test references accordingly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@gemini-code-assist
Copy link

Summary of Changes

Hello @cuioss-oliver, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves an issue where the benchmark overview incorrectly displayed latency metrics. By enhancing the benchmark converter to respect explicit configuration for metric names, it ensures that performance dashboards present accurate data. The changes also streamline the benchmarking codebase by removing redundant benchmark methods and associated dead code, improving maintainability and clarity.

Highlights

  • Corrected Latency Metric Selection: The JmhBenchmarkConverter.createOverview() method now correctly identifies the latency metric by prioritizing configured benchmark names from ReportConfiguration, preventing the dashboard from displaying an incorrect measureConcurrentValidation metric instead of measureAverageTime.
  • Removed Redundant Benchmarks: The measureConcurrentValidation benchmark, which was functionally identical to measureAverageTime, has been removed from SimpleCoreValidationBenchmark, CoreJfrBenchmark, and UnifiedJfrBenchmark.
  • Eliminated Dead Code: The CoreValidationDelegate.validateWithRotation() method was removed as it duplicated the functionality of validateWithFullSpectrum().
Changelog
  • benchmarking/benchmark-core/src/main/java/de/cuioss/sheriff/oauth/core/benchmark/UnifiedJfrBenchmark.java
    • Removed TOKEN_TYPE_ROTATION constant.
    • Removed measureConcurrentValidationWithJfr() benchmark method.
  • benchmarking/benchmark-core/src/main/java/de/cuioss/sheriff/oauth/core/benchmark/delegates/CoreValidationDelegate.java
    • Updated class Javadoc to reflect removal of concurrent validation.
    • Removed validateWithRotation() method.
  • benchmarking/benchmark-core/src/main/java/de/cuioss/sheriff/oauth/core/benchmark/jfr/benchmarks/CoreJfrBenchmark.java
    • Removed measureConcurrentValidationWithJfr() benchmark method.
  • benchmarking/benchmark-core/src/main/java/de/cuioss/sheriff/oauth/core/benchmark/standard/SimpleCoreValidationBenchmark.java
    • Removed measureConcurrentValidation() benchmark method.
  • benchmarking/benchmarking-common/src/main/java/de/cuioss/benchmarking/common/converter/JmhBenchmarkConverter.java
    • Added fields configuredThroughputName and configuredLatencyName.
    • Added a new constructor to accept configured benchmark names.
    • Modified createOverview() to prioritize configured benchmark names for throughput and latency, falling back to score-based heuristics.
    • Added a private helper method findByConfiguredName() to locate benchmarks by name.
  • benchmarking/benchmarking-common/src/main/java/de/cuioss/benchmarking/common/runner/AbstractBenchmarkRunner.java
    • Updated BenchmarkResultProcessor instantiation to pass the ReportConfiguration.
  • benchmarking/benchmarking-common/src/main/java/de/cuioss/benchmarking/common/runner/BenchmarkResultProcessor.java
    • Imported ReportConfiguration.
    • Added reportConfig field and a new constructor to accept it.
    • Modified processResults() to pass configured benchmark names from reportConfig to JmhBenchmarkConverter.
  • benchmarking/benchmarking-common/src/test/java/de/cuioss/benchmarking/common/report/BadgeGeneratorEnhancedTest.java
    • Updated the expected latency benchmark name in a test case from measureConcurrentValidation to measureAverageTime.
  • benchmarking/benchmarking-common/src/test/resources/library-benchmark-results/micro-result-scoring-test.json
    • Removed the JSON entry for SimpleCoreValidationBenchmark.measureConcurrentValidation.
    • Applied minor JSON formatting changes.
Activity
  • The author confirmed local test passes for benchmarking/benchmarking-common (181 tests) and benchmarking/benchmark-core (2 tests).
  • An existing converter test shouldCalculatePerformanceScoreAccordingToDocumentation was verified for correct benchmark selection.
  • The CI pipeline is expected to validate the full build.
  • The pull request was generated using Claude Code.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses the issue of the wrong latency metric being displayed in the benchmark overview. The fix correctly uses configured benchmark names for selecting throughput and latency metrics, making the selection process more robust. I appreciate the additional cleanup of removing the redundant measureConcurrentValidation benchmark and associated dead code, which improves the overall maintainability of the codebase. The changes are well-structured and logical. I have one suggestion to further improve the robustness of the benchmark selection logic, recommending the use of equals() instead of contains() for more precise matching.

Prevents incorrect matches when one benchmark name is a substring of
another's. Addresses Gemini review feedback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@cuioss-oliver cuioss-oliver enabled auto-merge (squash) February 23, 2026 18:33
@cuioss-oliver cuioss-oliver merged commit d0af710 into main Feb 23, 2026
23 checks passed
@cuioss-oliver cuioss-oliver deleted the fix/benchmark-latency-metric-selection branch February 23, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant