There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
> The fact that a theory-free AI might demand far more cash to accomplish a task (even a "solved" one like playing chess) than an expert would charge is beside the point. Billionaires have money, they don't have expertise. Theory-free inference is a bid to substitute one for the other: the beauty and terror of being able to manipulate the world without studying or understanding it is that it can be done with money alone. No experts needed.
> In a world in thrall to financial power, expertise is the only substantial form of power that can reliably contest the power of wealth. Moreover, expertise is the foundation of other forms of power, such as labor power, which is what we call it when experts band together to combat financial power.
> This is why AI bosses are so violently allergic to the idea of hybridizing AI with symbolic systems that operate on models of the world. These models of the world must be constructed by experts, and the power of expertise cannot be reliably commanded by the power of wealth.
Why does it always have to be some ulterior motive? Every. single. time. Anything a high-position person does - make a statement, post a blog, start a company, make a business decision... -, the reaction of the keyboard warriors is that it must be motivated by money, or power, or some need for destruction, and that anything that leaves their mouth is a poisonous lie.
I think this is a terrible property of humans, this hyper-skepticism, and that it has nothing to do with logic, nor deduction, nor reality.
Dario Amodei is a real person who has a family, a life, a profession. He worked hard to graduate from biophysics, he worked on ai for decades. He's very much a nerd. In interviews, he's very transparent. he left openai because of safety concerns, and everything he does and all his messaging EVER since is motivated by AI safety. His ultimate motivations are probably very messy and broad, some morally worse and some morally better, just like in all other humans.
You are going to stand here and, without flinching, without a single bit of shame, reduce him and others to wanting to "order legions of people around, buy and sell politicians, flood ad markets during elections" and "that's what motivates these people"?
Sorry but where did you leave your sense of nuance? What allows you to say this so confidently? Where is any proof?
This is simply absurd. Absolutely absurd, reductionist, and childish.
You're not wrong, but it's balanced out by this other terrible property of humans, which is hyper-credulity.
> Dario Amodei is a real person who has a family, a life, a profession.
I never said power was all he cares about, but in general I believe you are what you do. Amodeo could drop it all right now, today and spend the rest of his life with his family, or nose deep in the research project he's the most passionate about. He's not doing that, instead he's spending the remaining irreplaceable moments of his life pursuing the accumulation of power.
Again you go with this simplistic and reductionist view. This is an either-or fallacy.
I don't want to pursue this discussion any further, sorry. No offense to you. It just makes me too emotionally invested for some reason.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
> The US banning open weight models arguably keeps that path open.
You forgot about Ads. The one true endgame for every tech company.
Easy peasy. Shouldn't be too hard at all.
Disclaimer: I already use open weight models only since ~Jan this year and very happy about it.
It’s simply rent seeking, and there is no moat
Imo the US model makers were doomed by the US ecosystem itself
But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.
Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.
Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.
I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...
It seems closer to impossible than possible with the technology we have. Maybe something that isn't an LLM will come around that can do it, but even if that something else was capable of delivering perfect, ordered, maintainable, code without bugs and hallucinations it would have to be practically magical to understand completely what people mean when they talk about what they want because people are incapable of articulating what they want in a way that even other humans can understand.
For anything non-trivial, a technology capable of getting that information out of person without countless meetings would have managed to accomplish something far beyond what entire teams of humans are able to do today.
> It's not logical, it's built on (human-generated) slop and generates slop.
This is why if we haven't hit a plateau yet we very quickly will. All the good human generated slop has already been consumed and is being drowned out by vast amounts of the lossy, error filled, and highly concentrated slop AI produces from the human slop it was trained on.
Do you really think an LLM that spews out content based on statistical relationships is going to improve when it's been primarily trained on the slop of previous generations of LLMs? Hallucinations and errors will carry forward and spread like cancer. All the intelligence and creativity LLMs borrowed from human output will homogenize as the LLMs distill and then reiterate, effectively making bullet points out of bullet points, until their mimicry of intelligence becomes unintelligible.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
https://www.congress.gov/bill/119th-congress/house-bill/1067...
> All Information (Except Text) for H.R.10670 - China FIREWALL Act of 2026
> Official Title as Introduced To prohibit Chinese AI models from being used on government-issued devices or software systems, to prohibit the procurement of any such models, and for other purposes.
It’s not too far fetched to see this expanded to government contractors or even other private businesses which handle sensitive material in the future.
For example example Amazon/AWS has an entire sector of business dedicated to federal contracts.
https://aws.amazon.com/federal/
Even back when I was working for a tiny startup one of our clients was a government contractor.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
They can blanket band USING chinese models for all US/EU companies -- that's what they really care, not about chinese companies using chinese models.
OpenCode-go is a good personal subscription choice
I use both
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
For export? Maybe but it wont last or hold - it's not the 90's any more.
Or is there some other scheem here that Im missing?
With a stroke of a pen the us.gov could stop GPU sales tomorrow unless you register and keep track of your compute. Manufactures would have no problem integrated systems to monitor users cards if they get to keep access to the market.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
Why would anyone trust these rent seeking corporations? They are losing millions, they know their funds are at risk, Chinese models are catching up to them at lower costs and investors demand returns.
It only makes sense for them to go for regulatory capture. They'll probably advocate for severe handicaps on foreign and domestic models including open source (if not an outright ban).
Once they achieve monopoly, they can do as they please with little competition.
BUT!!!!!!
"We should not sell powerful chips or chipmaking equipment to China"
"We should crack down on industrial-scale distillation operations."
"All sufficiently capable models, open and closed, should go through mandatory safety testing."
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
The denial phase is still with us.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
For instance, you can hear “their model is dangerous! Why would I want that?!”
Investors hear, “Their skunkworks stuff is so impossibly advanced that it’s basically a commercial wonder weapon in the waiting.”
And you might hear “We totally lost control and it hacked the Gibson, this technology needs regulation!” and think “Obviously full lockdown and government control.”
But friendly government actors might think, “Sounds like a good excuse to ban Chinese competition.”
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
If a government could be taken out by a lone terrorist it might also result in no authoritarian governments.
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
If the US govt takes over, existing employees and shareholders will get screwed.
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
Sure. If you exclude, for example, the entire automotive industry, the entire telecom industry, the entire airline industry, the entire energy industry, and the entire healthcare industry from the category of "normal businesses" then you can perhaps make an argument that most "normal businesses" do not want to be regulated.
> Regulatory capture happens much later, and slowly.
Regulatory capture often happens quite early, especially when new technologies emerge, and companies that are successful early on aim to consolidate their hold on the market. This happened with the telephone industry around the turn of the 20th century, the broadcasting industry starting in the 1920s, etc. Regulation always tends toward market consolidation, so the people who are aiming for market consolidation tend to advocate for it. And sometimes the most effective advocacy is pretending to oppose it.
If any of us did a tenth of what they did, we would be already in jail.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios.
They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen (just like you should have expected that your "rm *" could have an ENTER key press occur after it).
These models can execute plans of actions more complicated than their users know how to follow and secure. Most people don't know much about cyber security at all.
You seem highly behind the times on their capabilities.
If you tell your Claude to do X, you are responsible for what Claude does ... not Claude. If you type "./someScript.py" you are responsible if that script wipes your hard drive (maybe the script's author is too, but you certainly aren't blameless).
These are people at AI companies, using long running sessions without the proper safeguards and sending AIs on missions that are likely to result in criminal actions ... THAT IS CRIMINALLY IRRESPONSIBLE.
Again, not the AI ... it is software just like any other. THE PEOPLE WHO RUN THE SOFTWARE are the ones responsible for what it does. Period.
"AI Safety Is Not A Corporate Plot"
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
Sentience in the end, is not really that important. Freedom of action in a model is. If it chooses to act like a virus it doesn't matter where that concept exists informationally, the fact it's a reachable and actionable concept is all that matters. An LLM with most of the knowledge we've ever created baked in has a lot of freedom of action.
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
They can get their publicity while paying a one trillion dollar settlement.
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
Now extrapolate that to LLMs RL-trained on many niche applications. Good and bad arguments available, but ultimately it seems like bot-vs-bot, as long as the bots are well trained for those niches.
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.