Hacker News Reader

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

do you have some time to chat? janak@withspecific.com

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

would love to chat and learn more about your set up! here's my email - janak@withspecific.com

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

we manually vet all codebases and companies

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

yes!

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

that's fair - for long horizon engineering tasks would speed still matter?

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

this is a good question. what would make you reject an otherwise working PR on design grounds?

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app store.

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

i appreciate the feedback, the benchmark is primarily long horizon real world engineering tasks on big private codebases.

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

we're going to open source some of our tasks and model trajectories as well

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

all the codebases were written pre-2023, so pre when AI got good at coding

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

would love to learn why?

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

will do! happy to chat more on janak@withspecific.com as well

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

we've done our best to use the native provider's harness. all models were run on 'high' reasoning. this is still v1 and tons of room for improvement - really appreciate your feedback!

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

makes sense

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

the tasks on the benchmar are long horizon swe tasks - where GLM does surprisingly well

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

here's where all the models messed up!

its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com

we also have the setup in the blog. the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks.

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

nope, these were private codebases

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

hey this seems really interesting - what prompted you to test multiple agents on your codebase?

janaksunilon Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

i'm janak, cofounder of Specific Labs (YC F25) and one of the authors of Real-SWE.

if i can help answer any questions please feel free to email me at janak@withspecific.com, happy to send over my phone number as well :)