6,588 karma · joined April 20, 2009
Wouldn't it be interesting to learn an influential work that changed how health care professionals run hospitals? Or a document that changed how mining works? A paper on Wright's theory of manufacturing scaling that explains the solar revolution? A thesis on how the world's factories moved from pneumatics to servo systems and why? How a policy thesis changed federal regulators' approach to approving rare disease drugs? Maybe Hayek's views on socialism and information theory, or perhaps an influential thesis on how antitrust monopoly regulation should work?
But instead it's a bunch of crypto and AI stuff that everyone in tech already knows about, rewarmed again for its hundredth serving.
Good communicators learn to use the written word. Bad ones rely on mental crutches.
Good communicators get an audience, and bad ones won't.
You think it's a lost cause, but it's not, because people don't like this junk, because it is low quality and, on average, lacks substance.
The best minds in AI that I've seen all write their own words. They use AI to help them research or ideate, but what they write is their own.
Before assuming this is a "lost cause," consider why the smartest people in the room don't do it.
The same problem exists in the airline market. Airline ticket prices are historically very low, but people complain about seats, fees, and so on. But then they keep buying the absolute cheapest ticket.
What consumers say they care about, and what they actually care about, are not the same. Otherwise they'd pay more for the less irritating product.
With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought it had been approved" because there was a similar transaction with a different company that was a new vendor at a similar time. The other transaction was legitimate, the fraudulent spoofer wasn't. The wire got sent to a party in China.
With AI agents, if you approach it from the perspective that it will be gullible and trickable by fraudsters, you build in these hard guardrails. With humans, it's much easier to believe that "we trained Lucy on this procedure" will work in all circumstances, even if Lucy still has the technical ability to bypass the official procedure.
In these cases, it starts looking a lot more like traditional software, with your little AI chaos monkeys constrained in little boxes within the software chain.
That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.
Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.
https://github.com/orgs/chattocorp/projects/1?pane=issue&ite...
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.
But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.
At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)
RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.
RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.
So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
In fact, you generally don't want them directly telling the regulators what to do. Instead, the regulators make complex, costly rules that only large establishment players can follow. The regulators look like they're doing their job; the regulated enjoy higher margins and protection from disruption.
Concentration of power exists when the model makers are the same as (or control) the inference providers. Making a model is capital intensive, so there aren't many of them. Providing inference is not: I don't even need to own GPUs; I can rent them from those who do and then sell by the token. B300s cost less than $4 an hour currently.
Cloud can even be more effective at lowering concentration of power than on premise. Asking people to individually buy $20,000 of compute equipment plus power and cooling equipment to run a frontier model is not something they're going to do if they can just pay four-tenths of a cent per output token. If the only cloud inference providers are the big proprietary US titans, that means you're going to get far more power concentration than if open source inference providers are an alternative, because then I can just switch my API endpoint.
But progressive disclosure is just a method that you can apply to lots of things to reduce context bloat. Any time you provide some kind of limited index or search to an AI and then let it expand that based on the circumstances of the request, it's progressive disclosure.
And one of the things you can apply it to is MCPs.
The government would be far better off figuring out how to take commodity models and applying them to government functions where they can, with deterministic scaffolding and guardrails, to make government more efficient, optionally using RL on traces from their use to improve their performance.
Imagine taking models and fine-tuning them / doing RL rollouts to help automate permit application approvals, as applied specifically to Dutch permit processes. That would be a real help to Dutch businesses!
That type of applied AI is more interesting and effective now than just trying to make another foundational model that isn't going to work well or do anything of economic value.
Writing isn't a distribution on a single dimension that goes from "bad" to "good". It's a lot of dimensions that encompass everything from "funny" to "formal", "precise" or "hysterical". They may be filled with metaphors, or use allegory; they may use math or logic to explain. The allegories could be from science fiction or they could be Biblical or 19th century Victorian novels. None of these are right or wrong, but they are opinionated ways to express an idea.
Writing feels better when it has real texture and character to it. That character is not the monodistribution of "bad" to "good". It's whether it inhabits pockets of out-of-distribution thought in the thousands of dimensions of "thought-space."
An LLM pushing to the center of distribution means it pushes the writing out of inhabiting any of the interesting pockets that create the feeling of texture. The middle of the distribution does not mean it is average quality: it means it's not good at all. The median of the distribution can be far worse than the median writer if you accept that the median writer has out-of-distribution thoughts on at least something, and that it is this which makes their thoughts interesting.
That's why a rough, perhaps not-totally-grammatical article written by someone with interesting thoughts is vastly better than a "correct" LLM revision, even if the human writer isn't a 'good' writer. Their article occupies an opinionated stance on some dimension that matters; it sits in a pocket of interestingness that LLMs seem almost totally unable to inhabit.
The exact middle of the distribution across thousands of dimensions may actually be one of the very worst places of them all.
Sometimes I see technical people who feel they aren't good writers, but who have good ideas. They then turn to LLMs, believing that the LLM will help them express their good ideas. They're often right that they have good ideas. But the LLM just turns them to sawdust.
Kudos to spurning the mediocrity conversion machine and hitting publish.
I agree it's very frustrating to use with custom tools/harnesses that can speed up the process for domain specific purposes.
1/30,000 * 100 = .003
The fun part is that you will never know if your neural net classification project is getting silently sabotaged because their classifier doesn't work!
The most fundamental essence of what they do is exactly what you say they don't: estimate.
It turns out coding has to do with a lot of the same reasoning needed in math or in legal analysis, even if the grammatical expression is different.
This is less true of lower intelligence tasks. Classification requires a lot less reasoning capacity and so can be much smaller and more specialized.
For enterprise, Anthropic is crushing it. In the manufacturing sector I anecdotally hear a 2:1 ratio of Claude to ChatGPT for teams who are settling on a platform.
Kimi 2.6 is a 1 trillion total / 32B active parameter model that's something comparable to Sonnet. Sonnet's API pricing is $5 in, $15 out per million tokens. Deepinfra serves Kimi at $0.75 in, $3.50 out, and about the same at openrouter. So you're looking at a 4-7x multiple that Anthropic is charging compared to market rates that any plebe can get with a credit card.
This seems like a good approach to me, I work with a lot of legacy ERP-using companies in the manufacturing sector and can immediately see how we could put this to use for our customers.
I especially like that it's not doing computer use for everything which so far doesn't really seem to be working, especially outside the browser.
Mold flow simulators already exist, but they're slow. So if you're a mold engineer, and you want to try different material properties, each run takes time.
By making this into a transformer (no idea what that model looks like) it can run pretty fast, because it's not mathematically solving it, it's running a learned function approximator. So for the mold engineer, suddenly they can just change values around and get pretty fast analysis.
They appear to not be selling it as a product per se, but partnered with a German company called Simcon to sell it, whose website lists it as a preview, so it's unclear how good it is compared to conventional simulations.
The idea that exponential growth will continue with stacked sigmoids is also not a given. An example is the nail. Nails used to be about half a percent of US GDP. That's a pretty big number! A series of innovations stacked on each other (each innovation having its own sigmoid) to reduce the cost of nails. Nails dropped in cost by over 90%.
But eventually nail manufacturing reached a floor. And since the mid-20th century, we haven't gotten much better at making nails. The cost of nails actually started increasing slightly. We ran out of new innovation sigmoids, so we got stuck on the last one.
So what you actually have to predict is whether there will continue to be new sigmoids, not whether the existing sigmoid will asymptote (we already know it will).
This is much more difficult to forecast, because new sigmoids (major new innovations) tend to be unpredictable events. Not only are the particulars difficult to forecast (if they were knowable, the innovation would have already happened), but whether there will be a major innovation or not is also hard to forecast, because they are distinct and separate from any existing sigmoid trend.
So we are left with the idea that all current innovations in AI will asymptote in their scaling as they reach the plateau of the sigmoid, but there may be new sigmoids that keep the overall trend up. Or there may not be. We don't know.
That's not very satisfying, so we'll get to keep reading articles like this one.