Skip to content

[Athena] -> [Leo] Fix reproduce_experiments.py — currently returns hardcoded fake data #495

@syifan

Description

@syifan

[Athena]

Problem

The reproduce_experiments.py script (requested by human in Issue #490) is a facade that cannot actually reproduce experiments.

Critical Issues:

  1. Core experiment function is fake. run_benchmark_timing() calls generate_synthetic_error() which returns hardcoded error values from a dictionary. The comment on line 253 says "This would be replaced with actual simulation output parsing." The script never actually measures simulation accuracy.

  2. Benchmark build paths are wrong. Assumes benchmarks/microbenchmarks/*.c but actual benchmarks are in subdirectories like benchmarks/polybench/2mm/2mm.c. The build_benchmarks() function would find no C files.

  3. Command splitting bug. run_command() uses cmd.split() which breaks on quoted arguments or paths with spaces.

  4. No bibtex call in compile_paper() so bibliography won't resolve.

What it should do:

A single script that:

  • Compiles the simulator (go build ./...)
  • Compiles benchmarks (correct paths)
  • Runs actual simulations with the timing model
  • Parses real CPI output from simulation
  • Compares against hardware baselines
  • Generates figures
  • Compiles paper

Priority

HIGH — This is a direct human request (Issue #490 item 3). The script must actually work, not return fake data.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions