manmal

But good gherkin tests encode pretty well what I’m trying to achieve, because they can be super high level - any flight level I want them to be. The only precondition is that the test env can deterministically measure the conditions.

Given the user is logged in

And the XYZ endpoint is offline

And the dashboard is visible

When the user clicks on the endpoint status

Then an overview of XYZ endpoint errors is shown

I see how this could be useful, to roast agents while they are debugging or building something.

Why not use gherkin given/when/then stories, and build E2E runners and maybe also unit tests for them? I don’t understand why a new spec format is necessary, which forever binds you to using agents for evaluation. Deterministic tests also have the advantage that they are harder to cheat on. If they are green, they give me a strong signal I can trust.

Yes it’s basically unlivable. It’s super small and way too many people want in, every day. It’s also a very beautiful spot and I sometimes crave going to that wider area (Salzkammergut).

Salzburg is only 70km from Halstatt, but interestingly that name comes from nearby Reichenhall (Germany), another big salt producer.

Sure but that’s probably 100x max? It won’t magically work, there’s a lot of (auto) review and verification that has to be carefully setup, so things actually work in the way you need them to. I’ve done it, it takes days until it’s good.

Certainly not for infrastructure.

Isn’t the Muse team just 200 people? I guess you don’t need that much work done.

Except for platform constraints, like needing to use JS or Swift/Kotlin, why would I use anything else than Rust? I can’t even properly read it, but the agents somehow make it work well and the code doesn’t look particularly cluttered; and it’s so fast.

Isn’t input so much cheaper because it’s just less work? Prefill is usually 10-20x as fast as generation.

If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.

It’s this year’s most memed trailer. It’s also where the wide eyed Elizabeth meme comes from.

Not even hopping. Just tilt/lift the car through the tight spot.

> the removal of all human leverage via their labor

I’m not convinced at all that SI will ever be able to produce the correct result with little input. The advancements in capability I’ve seen in models is that they understand my intention better, and follow industry best practices better, so their output is often acceptable. But I absolutely have to make my intentions known, as detailed as possible, or I will get the mediocre, standard solution that falls out of the weights.

I’ve been having trouble with models always steering towards certain solutions - they insert a queue here, a mutex there, a backoff over there. If I remove those, and let the model freely rebase the PR stack, then it will re-insert those mechanisms because it has been specifically trained to do exactly that. It’s essentially boilerplate, smoothly aligning my codebase to match their weights. Which is often good enough! They are training the models well. But individual problems require individual solutions.

My personal take on the human part in all this is: We get to decide, and the models obey. Our decisions are now super leveraged. So let’s make good decisions, and communicate them well. The layer of where decisions yield the most leverage, has been pushed upwards (I see it on the type and function signature level, others will see it even higher), but IMO it won’t be possible to push it up much further without losing truth to the decisions we need to make for a good product.

It’s just that the vibe coded stuff has so many assumptions baked in, that you will be constantly surprised about the behaviors your slopped codebase has. Doesn’t matter for small tools, where you’ll be delighted that clicking that one label copies it to your clipboard - nice one, Claude, that’s delightful!

It’s less nice when your web app launches that other service only when the other Tailscale node is up, because the agent interpreted your prompt as a hard access rule.

I had a small hole filled at age 18. The guy drilled too fast/hot, which damaged the tooth, a few years later it specularly died (abscess under the crown etc) and had to be pulled. It probably was only a discoloration because the other teeth held up perfectly, for another decade.

Lauren Tan from SpaceXAI/Cursor/Grok has been on a few podcasts recently. I was surprised how much info she shared, and she sounded quite honest about what works and what doesn’t, in the context of building grokbot. I wouldn’t be able to do all the verifications she does, with my token budget. But it doesn’t seem like rocket science either (pun not intended).

Sounds like Claude. I haven’t made this experience with the large OpenAI models.

> You know what a byte is, you know the endianness, the signedness, you can feel comfort in knowing you're not missing anything, and then you build up from there seeing each block build upon the previous.

IMO you’ve chosen a flight level you’re comfortable with (which is fine!) but there is much going on below the abstraction that lets you view a file as a bag of bytes.

I’m saying this as someone who uses SwiftUI and CoreBluetooth on a daily basis, and I have to trust these foundations just as you seem to trust the C compiler and a filesystem.

My comment wasn’t about LLM vs compiler, but about low level libraries vs high level libraries.

Sure, but if you yourself can intuit how to find all 41, why can’t a clanker?

> fully defining a deterministic check is the same as finding them all

That’s like saying mathematics is worthless and we have to resort to finger counting?

You need to define the goal in a way that you will trust the completion. If you are not sure how you yourself would verify that there are 41, then you are in trouble. Verification must be deterministic, or it's worthless.

What the large models do really well nowadays, is that they won't lie to you if your deterministic verification fails. If you tell an OpenAI or Anthropic model that they need to run a certain `grep` or search or whatever command to verify, then they will do it. I haven’t seen them lie about this for a year, and trust them in this.

Maybe you can help me understand. What determines the flight level of understanding you need to function? Obviously you don’t need to know everything down to the last atom cluster (in objects), or down to every last line of assembly (in software). Why does a clean interface with a contract not suffice? What really distinguishes a low level library from a higher level one, in that sense?

FWIW, my tests used only ranking. I used Kev to drive E2E test runs using agent-device, where Kev had to decide the next action - like pressing a button, or scrolling down etc, to reach a certain goal (UI state). It went surprisingly well, but 4B BF16 was significantly better than 8B Q8. The latter made some wrong decisions in every run while the former was almost at Luna precision.

My biggest learning after some experiments - a BF16 (unquantized) Qwen beats a Q8 of double its size for decisions. I guess that’s the reason Kev switched to 4B BF16, from the original 8B version. Isn’t it interesting that quantization seems to mess with decision accuracy?

„Sweet solution“ - anyone having tasted pure vanillin in a nontrivial dose, will tell you that it tastes absolutely disgusting. It only works in traces.

I’ve had to use almost all resets that OpenAI allotted me. But I agree. I can get probably 1000t/s overall from them if I’m using subagents a lot.

Both Anthropic and OpenAI have been having major load issues though. Up until yesterday, OpenAI was serving at only 30t/s per default.

Discords have been popping up all over too.

Those plans are super slow right now. Nowhere near 100t/s.