CBLT

Yeah, I also think decision models for pruning context isn't worthwhile. You're paying to load it into context (of your decision model), in order to save on loading the same data into context (of your generation model). The only way the economics makes sense is if you need a much smarter model for the compaction, but can't pay for it to load all of that context (because it's uncached or something). That sounds like a niche but it really doesn't exist. Cheap models are good enough, and smart models are cheap enough.

Personally, I use a harness extension that auto-compacts my context just before the cache expires. Combined with the default compact-when-full, this ends up being pretty much always what I want. It's one less thing to think about.

Please don't change "cybersecurity" to "cybercrime" from one sentence to the next as a rhetorical flourish. It comes off as disingenuous and makes it harder to get your real point across.

Damn, no results when I grep for 0x5f3759df in the source[0].

[0] https://github.com/terrapapagalli1516/quake-srp

Correct. Updated the message to brittlest. Thank you.

Sessions just broke all the time for me. The one time I dug into it, it ended up being a known issue where if codex returned over 10k characters for a turn it breaks the session. Because docker agent does not use protocols like ACP, instead trying to parse Codex's internal state in a brittle manner.

I tried not to do anything too crazy, but even use using an API key in the docker agent natively was getting me sessions that would hang and couldn't be recovered.

I tried really hard to make this work for me a couple of weeks ago but it was the brittlest harness out of any that I've used. I'd come back when it's more mature.

Sorry, private corp repos. Thanks for the offer!

We saw similar (slightly smaller effect, ~30% less instead of the 50% less you saw) improvement from trimming the number of turns. I'm not at all confident we've found the best way to do this here. We do need the thinking tokens to be spent in order to get the less obvious findings, but we couldn't get them from just increasing thinking effort. We went with forcing the model to use a tool that parallelizes the calls, and has an output floor so there's no turns with very little content read. Again, not at all confident this is the best way to do this, just what we've thrown together that works.

The actual biggest improvement is reducing the number of review rounds a PR has to go through. We were having too many rounds of drip-feeding new findings, because the AI reviewer is a subjective grader. Even if it found the same finding before, the next time it runs it might score it wildly differently. And it tends to want to score its findings across the whole grading curve, because it thinks that's more correct-looking. Our answer is to use previous rounds scores as anchors for the next round. Since we can't just use the same context and still have good performance, we needed to condense the score report into a small json we send forward.

I just reduced our AI-in-CI bill this month (while keeping or improving our KPIs), but now we have new and exciting ways to spend tokens right around the corner!

Replacement. Getting all customer requirements and migrating them to a superior service implementation.

Deleting files is easy. Decommissioning production services is not. It's too early to say for me, we've only decommissioned one vibe-coded service at my Day Job and it wasn't any worse than decommissioning any other legacy service has been. But it became legacy much faster - a couple of months.

> Some of us cared and contributed

Under a CLA.

Yeah I also thought it was strange their pareto frontier didn't include cost.

I buried the lede in my suggestion. I was saying that a future would give enough time to build or expand the fab in order to fill it. This means the company can fill an arbitrarily large number of orders - both consumers and big players - because it will just build the capacity to meet it.

I'd come at this from a different angle: we still want this to be market system, so we need to make this priced into the market. How can we price this in?

I would try to solve this by making the market structure reflect the underlying difficulty: we have to decide what capacity to produce years in advance, to construct the memory fabs. So this should be a futures market, and a capacity crunch would affect short-term-futures, but leave full term futures at the same price. Because the companies supplying the memory can just construct more capacity to fill those futures at the same cost regardless of the AI demand.

The readme seems to say the answer is yes? You'll lose the disk-based cache, but it's not inherently unsafe because the only coordination is object storage.

If you don't have any kind of agent instructions saying "don't copy code into prose", you'll inevitably get text that reflects some previous state of the system instead of its current state.

I've only seen it in bigcorp cross-service typedefs, or in startup's code that re-implements the checks in every service.

There's a good amount of literature about this (check the other comments), but you can vastly simplify this into two things you need to do:

1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.

2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.

Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.

Can you recommend any model that doesn't do this?

I also read that comment as an adversarial situation at work.

It used to be that when someone else at your company was asking for something that wasn't a priority, you would erect bureaucratic roadblocks to protect your time. Now, the new normal is to just forward their questions to AI and sling the slop back over to them.

I was bothered by the writing and fought through it, and carefully went through everything in the article... and you didn't miss anything. It's a tweet expanded to article length.

If you can actually time the market, sure. I cannot, so I don't pull money out of my stock indexes; I just send a larger fraction of my new investments into bonds.

I've been telling people the same thing. The tech industry has been cargo-culting "scaling the organization" where people are rewarded for org size. That zeitgeist has ended, and we've not even overcorrected much the other way.

> the CPUID example in the article [...] forgot to bind ECX as an input.

I'm not really familiar with this stuff, but the example uses what it calls a "pin" (which in their docs is a type of "binding") on ECX before calling CPUID.

The Google Drive link is to a simple tarball, not a git artifact.

Non-sequitor? They're not providing a (sha-1) hash, they're providing source code to integration partners using their business channels, not public git providers. Those business channels include contracts etc to "secure their supply chain".

You and I aren't in those business channels, and we're not being given anything with a hash. There's simply no hash to collide with?

It doesn't break supply chain security for anybody with power to change the situation.

Interestingly, this was tackled in this blog post[0] a month ago. They claim that plan files aren't token-efficient, because after reading the plan the workhorse model then reads all the relevant files anyways.

[0] https://news.ycombinator.com/item?id=48916512

That's a measure of the costs GitHub incurs to operate, not of its value as a platform.