locallylinear

I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we trust benchmarks on tasks we basically have no idea what they do (not for lack of transparency) or how they translate to work I’m actually doing

And several companies to do just this exist (eg Superconductor, Fullbeam) - does anyone use them and find them valuable? And curious if you’ve noticed an improved change in your agents work as a result, and if over time you see yourself switching models ? Or is it more of a one time realization that model family X is best for your code.

This is interesting but also think about how we as humans memorize things in code

I don’t think “oh it’s lines 10-50 in file A that has this code I’m looking for”. It’s much looser, something like:

- maybe I can’t fully remember file A’s name but I can fuzzy search for it based on what I do remember

- I search for a function or variable name that I remember, again kind of fuzzy searching. Or if I can’t find it,

- I start quickly scanning the whole file for relevant variable/function names

Obviously storing exact references will be more efficient at lookup time but to your great point, the environment is changing too much, especially now. Having just a) a starting point on where and what to search and b) an understanding the code base will and should change , feel like some of the principles a memory system should assume. Not that it’s mutually exclusive with what you said - just not sure the juice is worth the squeeze to make the thing memory is pointing to portable.

Idk, I just ask my agent what the status of the PR is in terms of failing checks, outstanding comments, etc.

But this seems fine

If anything I think the place to invest a better UX is on reviewing code, even “my” (agent’s) own code before I ask for a review. In a world where we’re moving away from IDEs, and using terminal apps or desktop apps to code, I often don’t even see the code until it’s on GitHub. But I still cling on to some semblance of needing to review, in particular what’s important.

And yes, me one year ago would be shocked at what I just said.

The most glaring downside with practically using decision models is if you need the model to not only decide something but give a reason, for every case, you end up not saving money or time, but just adding an extra step.

Give me a decision model that can do both, more expensive than today’s decision models but still cheaper than full on frontier model, and it’d open up so many more use cases, IMO

> Calibration matters

I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a calibration data set for the now cheaper LLM judge actually running as our judge

Is this standard ? Or am I just too lazy to manually label enough data to calibrate ?

The ROI on this is massive if the bounties would actually be accepted (they won’t, right?)

It begs the interesting question on the future of bug bounty programs: when it’s so easy to find and solve issues there’s less incentive to offer a bounty, which in turn leads to less incentive for humans/agents to find and fix security issues..?

As someone not plugged in on the current times with respect to bug bounty programs, curious for peoples take on this / is this already the reality ?

Very cool, great work and awesome story of how you all preserved to get to this point.

Are you thinking at all about security certifications to more easily be approved with, or sell subscriptions to, enterprise companies?

I’d love a pair but realistically I couldn’t use them with work unless it was an approved vendor; we use Wispr Flow though so I imagine being on a similar plane to them would suffice.