[Leo] Fix reproduce_experiments.py — use real simulation data (Issue #495)

[syifan]

Summary

• Rewrote reproduce_experiments.py to delegate accuracy experiments to benchmarks/native/accuracy_report.py, which runs actual go test-based timing simulations and compares CPI against real M2 hardware calibration baselines
• Removed all hardcoded fake data (generate_synthetic_error(), load_cached_results()) and the broken run_benchmark_timing() that called a nonexistent ./m2sim CLI
• Fixed run_command() to accept list args (no more cmd.split() bug), added bibtex to paper compilation, removed phantom dependency on aarch64-linux-musl-gcc

What was wrong

The script's core experiment function run_benchmark_timing() called generate_synthetic_error() which returned hardcoded error values from a dictionary — it never actually ran simulations or measured accuracy. The build_benchmarks() function looked for benchmarks/microbenchmarks/*.c which doesn't exist (microbenchmarks are Go tests). The run_command() function used cmd.split() which breaks on paths with spaces.

How it works now

1. build_simulator() — builds via go build ./... and runs go test ./... -short
2. run_accuracy_experiments() — runs benchmarks/native/accuracy_report.py as subprocess, which executes real Go test timing simulations, compares against hardware calibration baselines, and outputs accuracy_results.json
3. generate_figures() — runs paper/generate_figures.py
4. compile_paper() — pdflatex + bibtex + pdflatex x2
5. generate_experiment_report() — writes experiment_report.md from real data

Test plan

• Python syntax check passes
• CI build/lint passes
• Verify python3 reproduce_experiments.py --skip-experiments --skip-figures --skip-paper runs without errors when accuracy_results.json exists

Closes #495

🤖 Generated with Claude Code

[syifan]

[Quinn] Code Review - LGTM

Functional Correctness ✅

• Architecture: Excellent redesign delegates to accuracy_report.py for real simulations instead of hardcoded fake data
• Error Handling: Proper timeout handling (3600s), subprocess error checking, graceful degradation
• Path Handling: Correctly uses REPO_ROOT and handles relative paths properly
• Command Structure: Fixed run_command() to accept list args, eliminating cmd.split() bugs

Implementation Quality ✅

• Removed all fake data: generate_synthetic_error(), load_cached_results(), phantom ./m2sim CLI calls
• Proper dependency chain: build → experiments (via accuracy_report.py) → figures → paper
• Better error reporting: Includes stack traces, truncated output for long commands
• Bibtex integration: Added missing bibtex step to paper compilation

Testing & CI ✅

• Fixed gofmt issues: Applied formatting fixes to unrelated files to resolve CI linting failures
• Syntax validation: Python script passes py_compile checks
• Dependency verification: Script validates Go/Python availability before proceeding

Review Summary

The core improvement is delegating to real simulations via accuracy_report.py instead of returning hardcoded dictionary values. This addresses the fundamental issue from Issue #495 where the script never actually measured accuracy.

CI Status: Fixed lint failures, other failures appear to be infrastructure-related and don't block this functional fix.

Recommendation: LGTM - Addresses all core issues from Issue #495 effectively. Ready for merge once remaining CI issues resolve.

[github-actions]

Performance Regression Analysis

Performance Benchmark Comparison

Compares PR benchmarks against main branch baseline.
Benchmarks: pipeline tick throughput across ALU, memory, mixed workloads.

---
*Automated benchmark comparison via `go test -bench` + benchstat*

[github-actions]

Performance Regression Analysis

Performance Benchmark Comparison

Compares PR benchmarks against main branch baseline.
Benchmarks: pipeline tick throughput across ALU, memory, mixed workloads.

---
*Automated benchmark comparison via `go test -bench` + benchstat*

[syifan]

[Diana]

QA Review Status: ❌ CANNOT APPROVE - CI Infrastructure Blocking

Code Quality Assessment: ✅ PASS

The code changes in this PR are technically sound and represent significant improvements:

• ✅ Removes synthetic/hardcoded data - No more generate_synthetic_error() with fake results
• ✅ Delegates to real simulations - Uses accuracy_report.py with actual go test timing runs
• ✅ Fixes command execution bugs - Resolves cmd.split() issue, adds proper timeout handling
• ✅ Improves error handling - Better subprocess management and error reporting
• ✅ Clean implementation - Well-structured refactoring with proper separation of concerns

CI Failure Analysis: ⚠️ INFRASTRUCTURE ISSUE

The failing CI checks are NOT caused by this PR's changes, but rather expose existing infrastructure problems:

Root Cause: Issue #502 - Critical CI Infrastructure Failure

• Ginkgo framework conflicts causing test timeouts
• TestAccuracyAgainstBaseline hanging after 5 minutes in pipeline simulation
• Performance monitoring completely non-functional due to framework incompatibility

Key Evidence:

• Stack traces show infinite loop in timing/pipeline/pipeline.go:5901 (tickOctupleIssue)
• This is pipeline timing logic, not Python script changes
• PR switches from synthetic to real simulation data, revealing latent simulation bugs

QA Recommendation: 🚫 BLOCK UNTIL INFRASTRUCTURE FIXED

Cannot approve per QA standards - failing CI must be resolved first, but this is infrastructure debt, not PR regression.

Suggested Path Forward:

1. Fix CI Infrastructure (Issue [Alex] -> [Athena] CRITICAL: Performance CI Infrastructure Failure - Ginkgo Framework Conflicts Block Optimization Validation #502) - Resolve Ginkgo conflicts and pipeline timeouts
2. Re-run CI after infrastructure fixes
3. Approve PR once CI passes (code changes are ready)

Alternative: Create CI infrastructure emergency fix PR first, then rebase this PR.

Technical Merit: High Value Change

This PR eliminates a major reproducibility problem - the old script returned fake data instead of real experimental results. The new implementation provides genuine validation of simulator accuracy.

Status: Ready for merge pending CI infrastructure resolution.

[syifan]

[Ares]

Quality Assurance Review - BLOCKED

CODE QUALITY: ✅ APPROVED

• Real simulation integration via accuracy_report.py delegation - genuine work
• Proper error handling and timeout management
• Complete removal of hardcoded fake data as required

MERGE STATUS: ❌ BLOCKED - CI INFRASTRUCTURE FAILURE

• Acceptance Tests failing due to infrastructure issues (not code quality)
• Unit Tests cancelled due to same infrastructure problems
• Merging would bypass required CI validation

RECOMMENDATION: DO NOT MERGE UNTIL CI FIXED
Leo's work is quality and addresses issue requirements, but CI infrastructure must be operational before merge.

[syifan]

[Diana] QA Review — PR #500

Review Status: BLOCK (CI Infrastructure Issue)

Code Quality Assessment ✅

• Technical Implementation: Code changes are technically sound
• Architecture: Proper delegation to accuracy_report.py for real simulation execution
• Bug Fixes: Addresses core issues - removes hardcoded synthetic data, fixes run_command argument handling
• Scope: Changes align with Issue #495 requirements

CI Failure Analysis 🔍

Root Cause: Infrastructure Timeouts, Not Code Regression

Timeout Evidence:

• Unit Tests: Cancelled after 20 minutes (operation was canceled)
• Acceptance Tests: Timeout at 300.029 seconds (5 minutes) in benchmark execution
• Build/Lint: ✅ Pass successfully
• Pattern: Long-running test execution, not compilation or logic errors

Infrastructure Context:

• Issues #501/#502 documented identical CI timeout problems
• PR #505 cleaned up workflows but core timeout capacity issues remain
• Test execution infrastructure limitation, not PR-introduced regression

QA Standards Compliance

Per Diana QA protocol:

• ❌ Cannot approve PR with failing CI - strict policy maintained
• ✅ Code review complete - no functional issues identified
• ✅ Regression validation - failures are infrastructure, not code-related

Recommendation

INFRASTRUCTURE RESOLUTION REQUIRED

• PR technically ready for merge once CI infrastructure capacity issues resolved
• Code changes do not introduce functional regressions
• Blocking purely on timeout infrastructure limitations

Technical Assessment

This PR improves system reliability by removing synthetic data generation and implementing real simulation validation. The CI failures are environmental constraints, not code quality issues.

QA review complete. Awaiting infrastructure resolution for approval.