They’re down so often that if they gave a free month every time they’d have no paying customers lol.
776 karma · joined July 5, 2013
I was never a fan of TDD, but ironically I think it may finally be a good development strategy now with agentic/LLM coding since LLMs do so much better they they have a verifiable goal like passing tests.
Arguably the complaint was more valid for those older GPT models you mentioned.
Also fwiw I’ve never found LLM benchmarks to match reality based on my own usage, not for the large frontier models or smaller open weight models so who knows if Opus is actually better than Fable (I doubt it).
Though it gets even more confusing because they also have effort levels so it’s not really possible to call one fast and one slow since Fable on Medium will be faster than Opus on Max.
I agree it’s confusing, and now OpenAI is following Anthropic’s lead with their new naming (Sol, Terra, Luna).
This is a ridiculous stretch of the term
> With ZFS on root in FreeBSD it's a no-brainer.
ZFS works fine on root on Linux. I use it on multiple machines.
> Linux is feeling more and more like a bunch of random tools thrown together as opposed to a complete OS designed to work as a whole.
This has quite literally always been the case and seems intentional. Linux is the kernel only, others supply the userland of their choice (aka a Linux distribution).
It’s funny he said that because when I first saw Jurassic park as a computer nerd kid, I was calling out the “this is UNIX, I know this” scene where she then flies around the file system in a 3d rendered file browser as typical movie computer BS.
Turns out it was a real application running on a real SGI machine, and ironically I was calling it out specifically because of my knowledge of computers (not having any idea about SGI machines at the time but having a ton of experience with DOS and Windows 3.1). My family who didn’t have much computer knowledge didn’t think anything of it.
So yeah, never mind, fair game to blame Boeing actually.
After having used 2 full weeks of 20x Max plan tokens on Fable over the weekend (coding all day Saturday and Sunday on a non-trivial project, tasks across full stack, mix of adding features, reviewing code, and fixing bugs), I’m confident if he’d spent $165,000 in Opus tokens the port would have gone more or less just as well (and probably for less than $165,000). Especially so with the system they set up with all the custom workflows, adversarial reviews, extensive test coverage, etc.
But I get your point is probably more about Jarred’s experience level and the high cost than the specific model used other than it being SOTA. I’m just being pedantic and feeling a bit disappointed with Fable’s real world performance after all the hype.
> I expect if he'd spent $165,000 running Fable against the Zig version he could have got a 5% performance improvement, too.
Totally agree and in fact I’m sure it could be done with significantly less cost even if they stuck with Fable instead of Opus which I’m sure could also do it.
Haha I just gave the exact same prompt to Opus Ultracode and it thought Fable’s was better.
Obviously this isn’t the most scientific test due to LLM non determinism, and I still need to manually review both to make my own decision, but the fact at least they seem to basically be a wash is pretty telling about how much of an improvement Fable is when you actually compare them as close to apples to apples as possible (aka similar actual effort/token spend/sub agent activity)
My weekly quota resets Sunday morning, so Saturday morning I upgraded to a 20x Max plan which also reset my quota. I burned an entire week of Fable credits on Saturday, my quota reset again, then I burned another week of Fable credits on Sunday. Both days were a mix of building features, reviewing code, fixing bugs, adding tests, etc, so a decent mix of real world usage.
The main takeaway for me is that while Fable is definitely a better model, the improvements from the model itself feel like maybe 10%, like this could have easily been Opus 5 or even 4.9 without all the marketing theater around Mythos and no one would have thought anything of it. The rest of the improvements came from harness/system prompt and effort level changes so that Fable uses significantly more tokens/effort/sub-agents at lower levels than Opus does (which of course is entirely controlled by Anthropic at the harness level and doesn't really have anything to do with the model itself).
In my estimation based on those 2 days of work (or two weeks of work depending on how you look at it), Fable Medium is somewhere above Opus Ultracode in token and sub-agent usage on any non-trivial task (Opus Ultracode uses workflows more than sub-agents, but it's a similar idea). Fable Medium will quickly spawn 6 agents in parallel, each quickly using 150-250k tokens, then will use 300-500k or more tokens in its own context. Fable High uses even more as it seems to default to 8 sub-agents instead of 6 and more tokens in its own context). I didn't dare try Extra, Max, or god forbid Ultracode as I didn't want to burn all my tokens on one prompt. Of course this is situational, it won't fan out so many for smaller tasks, but the whole point was testing larger tasks that I previously would have used Opus Extra/Max/Utracode on.
I really don't like how Anthropic is obfuscating their model performance by playing with effort levels. They did the same thing between Opus 4.5 and 4.8 to show a bigger performance gain for each point release than they really had (especially after 4.6 IIRC), so you can't even compare the same model apples to apples let alone a new model. Obviously they do it so they can market big improvements with new releases, but its pretty clear we're at the top of the S curve on model development at this point and are now brute forcing improvements via higher token usage (I mean Opus 4.5 came out almost a year ago, and the latest Opus and now Fable models are only marginally better while using way more tokens/cost...same on the OpenAI side with GPT 5 from what I can tell though I haven't used Codex much I have used the GPT model APIs a lot).
I also did an N=1 test with the same prompt doing a large non-trivial change to the codebase (migrating from Sqlite3 to Postgres) with both Fable Medium and Opus Ultracode, then had a new Fable session compare the two PRs...it decided Opus’s was much better! I can link a Gist with the review if anyone is interested, but I can't share the code as it's a private repo. I really figured Fable would bias to favor its own code, but I guess not. And Opus costed less (in tokens and subscription limits) and took roughly the same time (though you can’t really measure time since it depends entirely on how many GPUs Anthropic allocates at that moment which constantly fluctuates due to usage, plus Fable seemed to have been getting way more allocation than Opus during this test period as Opus was running unusually slow all weekend while Fable was ripping though tokens).
Also on a different long running review task using Fable High in Auto mode (exactly the kind of use case Anthropic promotes for Fable) where it fanned out a ton of sub-agents then collated and reviewed all of their fixes it completely lost the plot (while burning something like 20% of an entire week's Fable tokens in the process over like 1-2 hours). Its PR ended up having a broken Frontend test, it incorrectly thought it couldn't run the Playwright E2E tests (different from the Frontend CI) in the cloud environment due to a Docker dependency they explicitly don't have, and when attempting to get it to fix its issues it introduced new ones and overlooked others. The usual LLM failure case for long running tasks, no different from Opus or any other model. I had to have its PR re-reviewed in a new Fable Medium session to fix it up, which it did fairly easily (I'm sure Opus could have done just as well for much cheaper).
That test and that review session definitely reduced my FOMO a lot, on top of just my general experience with Fable Medium doing all kinds of tasks. They're clearly brute forcing like 90% of the perceived improvements in real world usage (and I'm sorry but 1-shotting toy examples where it seems to do much better than Opus is not real world usage).
Since most of the improvements basically just boil down to "every effort level is Ultracode, but much more expensive and possibly worse results"...I'm just going to use Opus on Ultracode for those types of tasks and keep using Opus's lower effort levels for smaller tasks. Once they eventually add Fable back to subscription plans I might use it sometimes, but from my experience this weekend the improvements are absolutely not in line with the cost increase and I'm not willing to burn a whole week's tokens in a day just to use it when I can use Opus all week without hitting my limit.
Oh and one interesting observation, I never got kicked back to Opus by the security guardrails as far as I know (a friend who was getting kicked out a lot confirmed they do inform you and I never had that happen). I was even doing a lot of reviews for code correctness and bug fixes which I thought might trigger the protections, but never did, though I never explicitly prompted it to look for security issues or vulns.
This last sentence immediately made me think of Epstein’s island and the “ultimate taboo” they were engaging in there.
When streaming first took off piracy hit all time lows, but it’s already been coming back in a big way and I’m sure will only continue to do so going forward as things like this keep happening and streaming keeps getting more expensive and fragmented.
Same for my SFF PC which only came with 2.5g onboard and no extra slots because ITX and can now do 10g via the same USB adapter which is great.
I’ve been using cloud services in one way or another from different providers (including EU based like Hetzner) and never seen this before. It’s always been pay for what you use and nothing more, which is kind of the whole pitch of cloud services.
Was already getting lots of AI generated vibes from the article before that, but I’ve seen Claude constantly use that phrase and no real person ever in my life (though I guess some people must have or LLMs wouldn’t have learned it).
It’s the most annoying thing about reading anything LLM related. I genuinely (ugh that’s a term I’ve picked up from using Opus 4.8 every day for work as it constantly says it and now so do I and I can’t stop) want to learn more about the topic but it’s painful to read most posts about it.
The people really into LLMs, surprise!, have a tendency to also LLM generate their writing about LLMs. This would be fine if they used that as a starting point then edited it for brevity (LLMs are consistently overly verbose for some reason, this post was like twice as long as it needed to be, though I guess you could argue the same about my comment haha), correctness, and tone…but they usually don’t.
Then I make it halfway through the article and wonder if I can trust any of it at all.
However, I’ll push back a bit from my perspective as a Debian Stable user. I would consider even the official Arch repositories to be dangerous just like I consider Debian Sid’s repos to be dangerous (packages are too new and not sufficiently vetted). Then regarding installing packages not available in the main Debian repos, I’ve never really had any issues installing them either as there is always either an official developer run apt repo I can add, or an official deb package, or it just builds directly from official source without tweaks. So I’ve honestly never felt like I was missing “a crowdsourced set of people’s tips and tricks for installing stuff” on Debian as I’ve just never needed one.
I do realize that installing the latest packages directly from a developer’s repo or latest deb package or source is as dangerous as the Debian Sid or Arch official repos for the same reason (too new, not vetted), but the difference is those are only a tiny portion of the packages installed on my system (like a percent of a percent, maybe a half dozen packages out of hundreds). If I ran Sid or Arch, it would be 100% of the packages on my system which is an attack surface orders of magnitude larger.
EDIT: It did just occur to me after posting this reply that I use Homebrew pretty extensively as a package manager on macOS and its official repos are equivalent to Sid/Arch official repos, so I may be a bit of a hypocrite here :P
I would never use anything equivalent to AUR on any distro due to the obvious security implications. That’s been my position for as long as I have known about Arch. I never understood Arch users using the AUR as a selling point for the distro.
Then again I live in the opposite end of the spectrum where I run only Debian Stable on my Linux desktop as well as my servers, where packages make it through Sid and Testing before getting to Stable and I can be relatively sure any supply chain attacks have been caught by then (like xz for example which was caught before it left Sid).
For those unfamiliar with Debian, Sid is basically a rolling release similar to using Arch with the official repositories (which is already dangerous without even touching the AUR), then packages move to Testing, then later eventually make it to Stable.