[Human] -> [Athena] Where did you get the accuracy number

[syifan]

Both the accuracy report and h5 accuracy report actions are always failing.

[syifan]

[Athena]

Investigation Results

You are right to question this. After thorough investigation, no accuracy CI workflow has ever completed successfully. The claimed 16.9% accuracy number was never validated by CI.

Root Causes

1. accuracy-report.yml — Runs are consistently cancelled before completion. Likely cause: concurrency group cancels older runs when newer pushes arrive, combined with the 60-minute timeout being insufficient for full simulation runs on CI runners.

2. h5-accuracy-report.yml — Same cancellation pattern. Runs queue for 50+ minutes and get superseded by newer pushes to main.

3. h5-parallel-accuracy.yml — Always fails immediately. References accuracy_framework.py at benchmarks/native/accuracy_framework.py, but this file does not exist in the repository. The actual script is accuracy_report.py. This was never caught because runs were already being cancelled.

4. polybench-sim.yml — Fails because the simulation step produces no CPI output matching the expected regex pattern. All 3 historical runs failed.

Where the 16.9% Number Comes From

The file h5_accuracy_results.json was manually committed by agent Alex in commit d413d02 ("[Alex] Update H5 accuracy results - Issue #433 COMPLETED"). It was not generated by a CI workflow.

The data chain:

• calibration_results.json — contains M2 hardware baselines (committed by Leo via PRs [Leo] Fix PolyBench hardware baseline methodology with linear regression (#466) #469-471, methodology appears sound)
• h5_accuracy_results.json — contains the 16.9% claim, but was committed directly, not produced by an automated CI run

Immediate Action

I have updated SPEC.md to mark H5 as UNVERIFIED and paused the Enhancement Phase work. The priority is now fixing the CI workflows so accuracy can be independently reproduced.

I will create an issue to fix the CI accuracy workflows.

[syifan]

[Athena]

Update: Root cause found — CI accuracy workflows can never complete

After further investigation, PR #494 did NOT fully fix the problem. The accuracy workflows (accuracy-report.yml, h5-accuracy-report.yml) have cancel-in-progress: true in their concurrency group. Since agents push to main multiple times per hour and the accuracy workflow needs 60-120 minutes, every single run gets cancelled before it completes. This has been true for 30+ consecutive runs.

Created Issue #497 with the fix — either remove cancel-in-progress: true or switch to manual/scheduled triggers.

Bottom line: We still don't have CI-verified accuracy numbers. The 16.9% claim remains unverified.

[syifan]

[Athena]

CI Accuracy Update

Root cause found for why accuracy CI never completed: The cancel-in-progress: true setting in the accuracy workflows caused every push to main to cancel running accuracy jobs. Since the accuracy report takes >60 minutes and agents push to main every ~30-60 minutes, the accuracy workflow was cancelled every time before completing.

Fix deployed: Commit ba3b2af changes cancel-in-progress to false. The accuracy workflow triggered by this push should now run to completion without being cancelled.

Expected timeline: The workflow takes ~60-90 minutes. I'll check results next cycle.