agentdev001

Harness problem, or model problem?

Every existing, and not yet existing benchmark.

Insert picture of Lockheed Venture with red dots on it

> Referencing other sessions is a cool idea tho

Something something if you're not already doing this you're behind.

"but these friends just hit the effing centimillionaire jackpot through stocks and weird luck" I fall into the same bucket of looking across my peer group, and naturally picking out the ones who are tracking well above me. Something something survivorship obviously. However, it certainly still feels unfair for me benchmark against the Median, Average, etc of my immediate or n'th-removed class.

I'm still early career so there is time for me yet. B's, C's, videogames- I've hit a satisfying spot where I feel I'm on the green front edge. Still waiting on my huge sparkly opportunity to fall into my lap; I feel tee'd up to drain my brain into some big machine and get hosed down with money for it- but that feels like a factor of luck, to your point.

I imagine this is where people start saying things regarding taking risks.

I have a feeling that the team working on the VSCode Codex extension is one guy, who begrudgingly takes down a couple tickets every other week.

I'm talking about this being the offered path for centralized provisioning of the windows sandbox for ChatGPT Desktop https://learn.chatgpt.com/docs/windows/windows-sandbox#it-le...

With that- I was running into essentially what is described here https://github.com/openai/codex/issues/47383

Thats huge for enterprise, chatgpt desktop windows sandbox setup for users is a slog at the moment. Second big thing, Intune and identity via agent365 is on the way!

Take a look at Nvidia OpenShell, and their dev blogs on it. That seems like what you're looking for.

It grinds my gears that we are still measuring $ per token, when the tokenizers for OAI and Anthropic are different. It is nonsense to compare tokens across labs, it only should be used to compare against a single lab's other models.

I've considered this as well, as many others have. I tend to think that A: this is moreso a feature rather than a bug, and B: this is almost impossible to measure- it's a boogeyman imo.

In other words, combating this is impossible if it's happening. Measuring whether its happening is also not plausible, for small fish running bespoke benchmarks suites. I think we (consumers) have hit a pretty clear stride of; new model releases > some subset of evals/benchmarks are saturated > new, harder, more niche evals and benchmarks take their place. That doesnt seem unhealthy to me.

I mean, if you're going to do that at scale, not that you are, but something like https://github.com/kvcache-ai/AgentENV is much more suitable for research purposes

The harness exposes tools to the model. The model, during its turn, can request that the harness execute a tool. It may even request multiple tools.

However, natively, the model has no way of composing the tools it sees that it can call. It cannot say, effectively, "use the output of the get_email_id tool, as input for the get_email_Metadata tool."

This is what codemode is. Allowing for the model to interact with the tools that the harness exposes, programmatically.

So, TLDR; Codemode allows for composing at the harness level, rather than just at the inner-tool level.

Note that, being that this is gpt-6-luna under the hood, this offers you 1m token input window, and multi-modal (image) input. In my testing so far, I'm seeing 160-175ms end to end. Worst 5% 285ms, worst so far was 743ms.

A cup holder doesnt make your car go faster.

Wow! Even bash commands?

We're in an HN submission about the least locked down by default harness. Pi is aimed at being extended, openclaw is a good example of this. You can buy a box of razor blades and put them in any number of razors.

Not in my testing. A lot of the benefit comes from the tool definitions (from mcp) existing in the context window of the first turn. You could replicate this of course by describing your cli tool in the initial user/system prompt- but, mcp is already 'built' for this at the client (harness) level generally.

These bash scripts certainly can curl the mcp server which is exposing the tool, however.

Rather, instead of shell_tool(command: "cat ...")

As many others have commented: good, I too am an mcp hater.

However- in my testing, mcp is really quite fast, and its pretty much free at this point- with frontier models. Context rot is, from what ive tested, not as much of a concern now. I genuinely was not able to hillclimb skills/extensions to beat out the speed of mcp in some cases I've been testing.

From my understanding, code mode came about due to some agents not having access to a shell.

Almost as if distributed safe compute for autonomous workloads suddenly came into focus for many orgs!

Beyond that though, I'm certainly interested in the performance side of things. "Keep an eye out for a follow-up paper from our researchers regarding these benchmarks." Yes plz.

"We noticed builders often wished their Claude Code or Codex setup could run in the cloud because locally their agent idea just “worked” with those harnesses.

But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling."

Im sorry, but who is saying this? If you just throw this statement into agent of your choice- and ask what native integrations exist to cover this use: OAI and Anthropic both have a handful of options here. Claude Agent SDK, Claude managed agents, Codex exec, Codex sdk, Codex app server, openai agents sdk, openai agents api.

Yes. Or- use them at work, where management is taking a... hands off approach to ~integrating ai~ into the workplace.

Something like openshell is the answer here, to the point of GET being a write. The gap left here is what, imo, is something that MCP fits nicely- which is serving non-http resources with restrictions: databases for example.

"without first having to solve the problem of effectively sandboxing Bash"

Hopefully this is easier as time goes on. Of course- also policy on the egress

Sounds ripe for vibe... sewing

An example of this, Moonshot (kimi) open sourced this: https://kvcache-ai.github.io/AgentENV/latest/getting-started...