agentdev001

I thought that the name of this undo implementation was Muhammad Aqeel. Its PgVolvra, which is arguably a worse name- phonetically.

Cost v Performance pareto on Runebench, scoring the same as Opus 5.5 for 1/3 of the cost.

What is your implication here, in the case of 6.1 Terra?

I'll wait for the runebench results!

On AA intelligence index, this fills in the cost per task frontier nicely between 6 Luna and 6.1 sol. So when 6.1 Luna inevitably releases...

The OAI model tango seems to not have an end in sight, for better or worse. Give me more API toys!

I guess 5x/95% is where we're landing with the 6.1 family, that's nice.

Yes, that's my understanding- though I realize that OpenAI has made changes to what you can do with caching over the API recently, so you may be able to do some fancy stuff to get around this.

So like- I considered this idea of using a fast decision model to prune before compaction, but I scrapped it because A; im not bottlenecked by speed, and B; cache invalidation dark magic

From my understanding, if we have turns [A, B, C, D, E] and we decide that turn C is useless context, then we lose our cache for the [D, E] segment. (plz correct me if im wrong, I havent tested this)

Personally, I pay no mind to session state nowadays. I exclusively use provider native hosted compaction, which if you believe the providers, is the way to go. For context management- I spend most of my effort on the things which dump output back to the model.

If anyone has any good reads on benchmarking OpenAI and Anthropic hosted/native compaction against custom stuff, I would love to read!

Nvidia Openshell is absolutely awesome, great bundle of features in one spot. Im happy to see Nvidia seems to have accelerated on support here, I was kinda worried that as competing projects have popped up in the last 6mo that openshell would get google'd

Max and ultra are slow and overengineer, based on evidence from evaluations.

Artificial analysis' compilation of search api costs per 1000 queries, for comparison:

https://artificialanalysis.ai/agents/search-api?search-api-c...

I wonder what best practices here look like at this point. In my world, we've moved past skills for something like this, having porting helper/validator/etc things be harness level is measurably better than what skills look like right now. The open plugin spec is certainly an improvement (namely in packaging and distribution)- but, really you'd be leaving perf on the table if you were not to tune your harness. I wonder if something like dsh, pi, or other extensible (coding) harnesses have something like a concept of "harness as an extension".

Or my manager... or my stakeholders. Damn you powers that be!

https://youtu.be/Wb_vD3XZYOA The minidoc on eBPF is worth a watch if OP/OC excites :)

I recognize that, generally, the circles I'm a part of tend to pay the most attention to Coding-focused evals and benchmarks as frontier models improve. Biggest jumps I'm seeing with Astra and 6/6.1 Sol, and their Anthropic counterparts to a smaller degree, seems to be in spatial reasoning and playing games.

Looks like Argon wasn't covered here, but I'm getting the sense that Argon made similar leaps as Astra did. I would certainly be curious how Argon would have done here- it did quite well on RuneBench (https://maxbittker.github.io/runebench/).

"He went to crypto. I went to the El." Crypto ain't got the birds. Good move, OP.

Some chollima, its always a chollima.

You seem to care, so I wanna point out a semantic nit; "such a tool exists" could be interpreted as a harness provided tool, rather than a command line tool. (Edit, on a re-read its pretty obvious you mean cli tool, but keeping this statment here as im sure it'll open someone's eyes to the difference)

I've gotta give hax a go, as a pi user who came on-board due to the no frills direction pi takes. I say takes rather than took, because regardless of the (heated) discussion surrounding mcp and code mode support, I cannot fight the numbers- augmenting the harness provided tools is a notable boost to execution round trip performance.

It's allowed, but you get bumped to per token pricing.

Harness problem, or model problem?

Every existing, and not yet existing benchmark.

Insert picture of Lockheed Venture with red dots on it

> Referencing other sessions is a cool idea tho

Something something if you're not already doing this you're behind.

"but these friends just hit the effing centimillionaire jackpot through stocks and weird luck" I fall into the same bucket of looking across my peer group, and naturally picking out the ones who are tracking well above me. Something something survivorship obviously. However, it certainly still feels unfair for me benchmark against the Median, Average, etc of my immediate or n'th-removed class.

I'm still early career so there is time for me yet. B's, C's, videogames- I've hit a satisfying spot where I feel I'm on the green front edge. Still waiting on my huge sparkly opportunity to fall into my lap; I feel tee'd up to drain my brain into some big machine and get hosed down with money for it- but that feels like a factor of luck, to your point.

I imagine this is where people start saying things regarding taking risks.

I have a feeling that the team working on the VSCode Codex extension is one guy, who begrudgingly takes down a couple tickets every other week.

I'm talking about this being the offered path for centralized provisioning of the windows sandbox for ChatGPT Desktop https://learn.chatgpt.com/docs/windows/windows-sandbox#it-le...

With that- I was running into essentially what is described here https://github.com/openai/codex/issues/47383

Thats huge for enterprise, chatgpt desktop windows sandbox setup for users is a slog at the moment. Second big thing, Intune and identity via agent365 is on the way!

Take a look at Nvidia OpenShell, and their dev blogs on it. That seems like what you're looking for.

It grinds my gears that we are still measuring $ per token, when the tokenizers for OAI and Anthropic are different. It is nonsense to compare tokens across labs, it only should be used to compare against a single lab's other models.

I've considered this as well, as many others have. I tend to think that A: this is moreso a feature rather than a bug, and B: this is almost impossible to measure- it's a boogeyman imo.

In other words, combating this is impossible if it's happening. Measuring whether its happening is also not plausible, for small fish running bespoke benchmarks suites. I think we (consumers) have hit a pretty clear stride of; new model releases > some subset of evals/benchmarks are saturated > new, harder, more niche evals and benchmarks take their place. That doesnt seem unhealthy to me.

I mean, if you're going to do that at scale, not that you are, but something like https://github.com/kvcache-ai/AgentENV is much more suitable for research purposes