Something I'm kind of curious about is the degree to which eval performance might be due to parts of the SWE-bench dataset getting into the latest LLM models.
A while back someone on Twitter seemed to confirm that Claude-3.5 was aware of the Github issues inside the dataset by mentioning them, but I couldn't find the original post.
30% performance on the full SWE-bench benchmark is quite the leap, but just how "real" of an achievement is this? Anecdotal reports mention that GPT-4o is marginally better than GPT-4 turbo at best, and yet agents leveraging the LLM did perform better.
What would happen if SWE-bench was updated, top to bottom, with completely new Github issues? Would all these agents just completely shit the bed?