bfogelman

we’ve been using it internally (on sculptor) and the speed ups are crazy — we can now have our agents run tests all the time and iterate quickly! Excited other people can now give it a spin

We’re running both vet and codex on all PRs to do code review and have found they compliment each other well. Vet often catches issues that codex does not!

ooh this is a great idea -- thanks for sharing!

See this comment for some differences: https://news.ycombinator.com/item?id=45428185

nope but vibekit looks interesting -- will take a look

hmm not ideal -- will try and take a look and see whats going wrong

haha honestly a little bit ya. One key thing we've learned from working on this is that lowering the barrier to working in parallel is key. Making it easy to merge, context switching, etc are all important as you try to parallelize things. I'm pretty excited about "pairing mode" for this reason as it mirrors an agents branch locally so you can make your own edits quickly and test changes.

We've also shipped "suggestions" under beta (think CI pipelines for your parallel agents) which might feel a little different. The idea is to use LLMs and your regular coding tools (pytest, pyre, ...) to verify that the code produced by the agents is actually correct.

in the works! we want it to be possible to always have the best models and agents available

lffgggg excited to see where you take lingo log :)

Hopefully in the next couple of days! You can join the discord and we'll post an announcement when its ready https://discord.gg/GvK8MsCVgk

right now we're using docker -- we're planning to support modal (https://modal.com/) for remote sandboxes and a "local" mode that might use something like worktrees

Member of the team here, happy to answer questions. Took a lot of ups, downs and work to get here but excited to finally get this out. Even more excited to share other features we've been cooking behind the scenes. Give it a try and let us know what you think, we're hungry for feedback.

One thing I’d be curious to see is how well this translates to things outside of HumanEval! How does it compare to using ChatGPT for example.

Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the problems themselves aren’t really indicative of actual software engineering problems. Also, we’ve been able to get around 85% pass@1 on GPT-4 internally as of a couple weeks ago. It’s hard to say if they’ve contaminated the models with RLHF though. It still is exciting how close we’re getting with open source models but we’ve still got a decent amount of work to go!

If you’re interested there’s a new RL benchmark that was built using Godot (disclaimer I helped make it!)

https://github.com/Avalon-Benchmark/avalon