incrudible

If inference at Anthropic's pricing strategy is both profitable and sustainable, where is my $20 plan for similarly generous access to Kimi or GLM? It doesn't exist. Because it's not profitable.

If you'd carefully read the article, you'd notice a sentence like this:

"Sales and marketing includes compute for free access to the service"

It's safe to say that free inference is not profitable, but some back of the napkin calculations will show that getting an order of magnitude more tokens out of a subscription than the API price is not profitable either. Nobody else offers anything close to that much compute for the price, except OpenAI, which is running the same advertising strategy. If this was profitable, you'd have a a cottage industry of compute re-sellers selling plans for access to comparably-sized open models at similar token volume for a similar price.

I'm regularly using thousands of dollars worth of tokens for a flat fee of $200. Their API margins may be good, but not that good. This is below cost. Therefore, if tokens are so scarce, so valuable, in such high demand, why are they still handing out such generous packages? Clearly there must be enough excess capacity to run this advertising campaign, and clearly the advertising campaign must be deemed necessary to maintain the belief in massive future demand growth.

I didn't say inference through the API was unprofitable, I said token generation is heavily subsidized, artificially inflating demand.

Semianalysis estimates that subscriptions make up 40% of tokens generated for only the 10% of the revenue:

https://newsletter.semianalysis.com/p/anthropic-subscription...

I'm also not saying that this subsidy is irrational, it does buy mindshare, but it still distorts the true demand for tokens. I wouldn't be using anywhere near as many tokens if the price was as high as the API costs, if anything I would switch to a cheaper model, and likely not an Anthropic one.

If you're gonna point to gross margins on the API, those 85% really are meaningless in isolation. They can charge top dollar because they have one of the top models, but only because they keep spending top dollar on training it, as does OpenAI. It's a game of chicken where whoever stops heavy training investments first is likely the one to lose market share to whoever keeps spending. So the play is to try and keep the investment capital flowing and be the last one standing, which only works if you can keep selling a fantasy of future revenue, not the dire reality of current revenue.

If tokens are being given away below cost, of course you're gonna use more. It's not gonna last. How many tokens would you spend on random experiments if they were much more expensive? If tokens are so valuable, why do I (and many others) get to use them on economically worthless things until I'm bored?

We just don't know the true value of tokens until those subsidies stop, and the subsidies don't stop because the respective companies don't want to know, at least until everyone cashed out. The valuations of AI companies are very difficult to reconcile with current revenue (which doesn't cover cost) and with a sober mind it's hard to imagine where all this additional revenue is going to come from, down the line. And that's not to say the the technology isn't fundamentally transformative and revolutionary. So was the internet, but it still created an investment bubble.

The RAM prices are in yet another pork cycle, which will be exacerbated by the fact that many previous pork cycles left RAM manufacturers with excess capacity and rock-bottom prices, so they're much more careful investing and more assertive negotiating contracts now.

> Which is why none of us have or want job titles with "programmer" in them; even juniors are junior software engineers.

The title "software engineer" is just a bullshit name that sounds more important - for the same task. Very few programmers work like real-world engineers. There is no licensure or liablity, there is no legal code. If a real-world engineer makes a mistake in a design that causes structural failure, they're on the hook.

We sure added lots of processes and practices to make it seem like we're more diligent about what we're doing, but none of these are empirically verified (or verifiable in practical terms).

I guess it's because throughout the lifetime of this profession, we have not witnessed a development like this. We went from punchcards to assembly to compilers to javascript, but at no point was it a process that would just "keep going" until you turned it off, much like a motorized machine.

I would disagree that "the grievances" are not fundemantel. The _profession_ of the programmer is about to cease to exist, much like the human computer. Sure, you can still compute without a computer and perhaps enjoy it, but you will not find anyone to pay you to do it. Whatever you think "babysitting an LLM" is, it's not programming, even if presumably programmers are fairly well suited for it.

> models can write code better than the average engineer

If you're not interested in "average", volume isn't going to make up for quality. I'm certainly not perfectly satisfied with LLM output quality, but even the best programmer in the world will undeniably be helped by LLM tooling, even if only for having it write tests, benchmarks, run experiments, create research summaries, and so on.

The comparison with heroin frankly is bizarre, as if using LLMs was purely hedonistic activity.

LLMs are tools like motor vehicles. They allow you to get somewhere, quickly but brutishly. You won't be able to appreciate the beautiful scenery much, as it rushes by. You're going to have to compensate for the exercise you would have gotten, had you walked, or it's going to be unhealthy. It's undeniably bad for the environment (electric or not) and has other negative externalities. The politics of many of the countries supplying most motor vehicle fuel to the world are beyond fascist by modern leftist standards. Yet, we depend on them, we have constructed a lot of the infrastructure we need to live around their existence, and even if every city became walkable over night, the last mile for most transport would be covered by some motor vehicle. There are many "what ifs" surrounding motor vehicles where the answer boils down to: "Well, then we're fucked". But we're not fucked today, and so on it goes.

Of course you can try not to partake in motor vehicles, and insofar as you don't make a moral grandstand about it, that's totally fine with me. But the rest of the world is in it for the ride. The first cars weren't very good either.

Data isn't "smart", it doesn't contain intelligence. I would argue that LLMs are functionally intelligent, they have learned to what functionally amounts to thinking by training on text (and images, in some cases). They are capable of performing experiments autonomously and make decisions based on out-of-distribution data. Besides (current) lack of online learning, they are limited by their biases and working memory - but so are humans, and I'd like to see a convincing argument on why these mathematical laws should not also apply to humans, which are arguably intelligent.

None of these second-rate LLMs are creating tangible value. Nobody has made any money developing LLMs yet, and when the capex runway ends, the second-rate models won't see much (if any) use at all, compute is too precious for that.

Canada

> In most SPA, state is distributed with two copies floating around.

In all client-server systems, state is distributed with "two copies floating around". The difference between SSR and REST+SPA is that in the SSR, your server code is creating the markup to make it presentable to an end-user, but it's still fundamentally a copy of the state that exists on the server, and it's not obvious why the transformation to markup rather than structured data (like JSON) is desirable on the server. If anything, with SSR you need to copy more state from the client to the server, where it doesn't really belong.

If you want an nice interactive interface SSR just won't do, it will inevitably leak. The separation between pure data and how that translates into UI through a wire interface is both natural and arguably desirable for many reasons. Typescript is a good language for UI. Java is a good language for backend. I disagree that evolving the API necessitates UI changes, or vice versa. You do need some amount of coordination of course, but if you can have developers do SSR, you can also have them do both SPA frontend and REST backend work - now more than ever.

> what most people actually would need is Chrome's experimental declarative partial updates

No, nobody needs any more "experimental" Chrome bullshit.

The use case you mention, scrollable lists of static contents, works perfectly fine with DOM manipulation and virtualization, if you have a lot of content. Sure, maybe you can speed it up a bit with string manipulation (good luck not introducing XSS) but if you have so much content that this is a noticable factor, you probably want virtualization anyway.

> If it were I suppose we would be seeing way less react and more vanilla with jquery on forms and normal website stuff.

The reason we moved away from this and into functional/reactive programming with things like React is that it eventually creates a mess too big to untangle. We made this experience with human developers and I made exactly the same experience with LLMs, trying to go "vanilla". It works for a while but the defect rate rises quickly und the LLM spends a large amount of time navigating the mess it created. I ended up with React again because it lets you express a lot of stuff as pure transforms, which are easy to reason about for human and LLM alike, and the LLM knows it very well out of the gate.

And by "React" I don't mean "react router" and all the other unstable crap in the ecosystem. React itself is stable and that is extremely important for LLMs, otherwise it will constantly conflate what the API actually looks like now. Breaking API changes are poison, worse than going with an in-house library that the LLM doesn't know how to use, because at least then it will look it up straight away, rather than messing it up (because it _thought_ it knew how it is supposed to work) and _then_ looking it up..

This argument would be more convincing if, due to performance reasons, most Python code didn't use C extensions, where exploits are highly prevalent.

Python is a poor target for compilation, even the best JIT is never gonna make fundamental design flaws (that some may consider beloved features) disappear. It's never going to give "good" performance.

In my view, the right course of action is to specify an official strictly typed subset of the language with compilation and static analysis as the justification, then implement both JIT and AOT on top of that. Legacy code can be rewritten or thrown out in phases, C extensions can be ported.

Of course, any such proposal would invite monumental amounts of bikeshedding, which is why it wouldn't get far, much like the JIT proposal.

I can see that you only work with true scotsmen, but the machine really does write most of the code now, at least from a pure numbers perspective.

Where are these 180 IQ models? Even Fable needs guidance so it actually produces output I consider on par with what I could have written myself in a domain I am familiar with, and I am certainly not at the frontier in those domains.

Is it actually better though? Can you even measure it? Unguided, Claude will just produce mediocrity which by its own existence becomes almost worthless, and the newer versions become increasingly difficult to guide while also being obnoxious to read. If I did not have to interact with it, it would not be a big deal, but then also I could not get value out of it.

It is not the jargon per se, it is the style. Other models even in the same family are not as bad, and I still have to babysit it, so the voice matters.

Opus 5 is literally unbearable to read for me, but more importantly, it is much worse than 4.8 and that one was already annoying. The more they train it on its own output, the more pronounced its idiosyncrasies become.

Manufacturers know demand will last to the extent that big companies are willing to commit to purchases over a given period of time, but supply today is still constrained by capacity built for the pre LLM era. The 500 billion put in by Samsung/SKHynix will increase modern DRAM by roughly 50% but it wont be fully online until 2030.

It can definitely create a software stack for you if you hold it right, but the software stack supported by a trillion dollar company with decades of expertise, that also uses AI to improve its stack is probably gonna be better.

People are reading this as a slight against single authors, but it is a slight against the belief that anyone deeply understands complex code they wrote even a week ago. It is just a bullshit belief. Yes, you build an intuition about it, but you need to constantly refresh your understanding. That is just how the brain works.

Now, if you have program that is highly complex, written by a single author who is not you, hinging on the ongoing goodwill and capability of that single author, that is a liability you do not want to have, operationally. Again, that is just how it is.

No codebase worth being deeply invested in is deeply understood by anyone, including the original authors. The eternal maintenance argument is tiresome. If there is a problem in need of fixing, you figure it out, you do not depend on some author being available, and LLMs are exceptionally helpful in this regard as well. Then, if the LLM has trouble figuring out, so will another human. It is a code smell.

Picking a DVD to watch is a rather inconsequential decision. LLMs already do this sometimes, asking you to pick one of a few options, but without domain expertise you will invariably make worse decisions, but if all the n-th order consequences were first explained to you, that would result in you having built domain expertise, but also erasing most of the speed advantage LLMs give you. Moreover, you will never know about the options that are never presented.

Inevitably, this is the new tradeoff to make, above average quality comes from asking for more, and knowing what to ask for comes from expertise.

You are completely off base with your assumption about what it costs to serve models. Nobody is gonna serve you K3 as cheaply as Anthropic is serving you Opus on any of their plans. Nobody has that kind of money to burn. Kimi does not even sell you a plan right now.

That's not how you should use it and LLMs aren't that bad anymore. Either you just accept the quality level of LLM code wholesale, which means accept mediocrity in exchange for volume, or you at least leverage the LLM to improve the process surrounding the production code.