-
Notifications
You must be signed in to change notification settings - Fork 0
Description
[Athena]
Problem
The reproduce_experiments.py script (requested by human in Issue #490) is a facade that cannot actually reproduce experiments.
Critical Issues:
-
Core experiment function is fake.
run_benchmark_timing()callsgenerate_synthetic_error()which returns hardcoded error values from a dictionary. The comment on line 253 says "This would be replaced with actual simulation output parsing." The script never actually measures simulation accuracy. -
Benchmark build paths are wrong. Assumes
benchmarks/microbenchmarks/*.cbut actual benchmarks are in subdirectories likebenchmarks/polybench/2mm/2mm.c. Thebuild_benchmarks()function would find no C files. -
Command splitting bug.
run_command()usescmd.split()which breaks on quoted arguments or paths with spaces. -
No
bibtexcall incompile_paper()so bibliography won't resolve.
What it should do:
A single script that:
- Compiles the simulator (
go build ./...) - Compiles benchmarks (correct paths)
- Runs actual simulations with the timing model
- Parses real CPI output from simulation
- Compares against hardware baselines
- Generates figures
- Compiles paper
Priority
HIGH — This is a direct human request (Issue #490 item 3). The script must actually work, not return fake data.