Code and Scripts for reproduction

[zjy-ucas]

I am attempting to reproduce the paper's results using the provided YAML configuration file, but I have encountered some issues, resulting in a discrepancy in the final scores. I achieved a 77.2% using the Opus and 74% with Gemini-3-Pro-Preview. Here are my settings:

API: OpenRouter.
Software: mini-swe-agent v1.14.2 and the YAML configuration file provided in the repository.
Hardware: I am running the Docker environment on a high-performance CPU server (384 CPU cores, 2.2T RAM).
I launch 16 Docker containers concurrently for each run and run experiment twice for both temperature 0 and temperature 1.

For some of the instances that were not solved, I manually simulated the release's trajectory in my local Docker environment, and the returned environment states were consistent.

I think there might be subtle differences in the implementation.Could you please share the exact mini-swe-agent code and testing scripts you used for your official results? This would let me compare directly and figure out what I'm missing.

[brutalsavage]

Hi @zjy-ucas,

I also used mini-swe-agent v1.14.2 (091e8f9da2aca6f1dd53c8a0f5080262957ada89) for all our experiments, we did not make any implementation changes to mini-swe-agent in our experiments and the provided YAML file is the one used in our experiments

I manually simulated the release's trajectory in my local Docker environment, and the returned environment states were consistent.

I'm not sure what you mean by this

Due to the non-determinstic nature of LLMs, its expected that we can get different results across different runs