Requesting scores for GPT 5.1 and GPT 5.2 (Possibility of crowd-sourcing evaluations for new models?)

[apoorvagnihotri]

Hello, I found this work to be one of the firsts which could actually test LLMs on their SWE performance independently of the SWE agent being used.

I am curious if there is any way I could contribute to the evaluation numbers for newer models coming out every now and then. I would probably be not able to run the whole benchmark but it would be super cool in case I could contribute with some issues so that the evaluations can be democratized and whenever there's a new model coming out, people can contribute to the evaluations.

[brutalsavage]

Hi, we actually also ran GPT-5.2 recently, you can see all the models we have tested on live-SWE-agent on our leaderboard: https://live-swe-agent.github.io/

I could contribute to the evaluation numbers for newer models

Right now we don't have a mechanism for people to submit scores for other models, but yeah this is something we can definitely look into in the future. Feel free to try live-SWE-agent on some models and let us know the results.