617 karma · joined September 14, 2026
We can't rely on hardware staying hackable. We couldn't ever since "secure boot" became a thing.
Companies say "security" and mean "security of our business model".
Plug-in solar is suboptimal in many ways, but it does sidestep a lot of those hurdles. Not too surprised to see it on the rise.
Another thing that might be worth keeping in mind is that plug-in solar panels are often installed vertically - i.e. balcony solar. In which case the peak of the panel's generation is offset to the edge of the typical "duck curve". If it gets popular enough to affect the grid, it might get the solar-induced demand slump larger, but make the edges even sharper.
Would that make grid scale battery storage with its instant response even more important?
Clearly, for some people, "type into a search bar" is some sort of cognitive obstacle, and "go bother another person" isn't.
Now that I'm thinking about it: maybe the likes of OpenAI have the right idea when they invest into conversational models? Does it meaningfully reduce the workload of flesh proxies?
If "sort a list" becomes a trivial task, everything that was bottlenecked on "sort a list" is unbottlenecked, and other parts of the process become bottlenecks instead. Thus, the demand for non-sorting tasks grows. The saturation of low end feeds the value into the high end.
And that, in turn, makes AI even more valuable if it can produce thinking on demand.
Do you think that we are anywhere near that point?
We've already seeing things - like what's happening with AI reverse engineering now. I'm a reverse engineer - and I know that the reason why RE was used as "last resort option" was that expert reverse engineers are rare, expensive, and work slow.
But now that a lot of RE workloads can be tackled by AI instead of a human? The demand for reverse engineering is at a never before seen high. AI made reverse engineering much more viable - and thus, a lot more reverse engineering is happening now.
There are myriads of fields like this - things that go overlooked not because no one wants them, but because the economics of human expertise make them nonviable in most cases. As AI gets better, it can begin to enter those fields.
Except you, of course, need to push the frontier to do it.
"AI use cases" is a moving target! If you try to define "use cases" by "what AIs today can be used to do", you miss things AIs would do in the future. The more capable and price-performant AIs get, the more there are places where AI has a role.
This is a big part of what the frontier buys you.
The other big part is that going from a local AI to a frontier model often makes the difference between "this use case works 60% of the time, good tech demo" and "this use case works 99% of the time, good business decision". The difference between "clunky and needs human in the loop" and "just works" is worth paying the API price.
> “Dark patterns” (also called deceptive or manipulative design) are interface choices that steer users towards decisions they would not otherwise make, for example:
> Making the “accept all” option bright and prominent, but hiding “reject” or “manage settings”
Not a direct reference, but it's clear exactly what it's getting at.
Wouldn't call it a big win though. If it's not established as a browser level setting instead of some shitty per-website banner that expires every 3 weeks as the site adjusts its ad serving system once again? We're still in cookie law hell.
A calculated move. If they're right and the AI demand is a fluke, they don't overinvest in expensive manufacturing infra, they don't get saddled with more capacity than they need, and there is no memory glut eating their margins in the future. If they're wrong and the AI demand isn't a fluke, they get to pad their margins big time, and the supply bottleneck isn't their problem. As long as they shake hands and bet on "fluke" in sync, the downside is very limited for all of them.
They shook hands, they bet in sync, and they were wrong. Now the supply bottleneck is everyone's problem.
Potentially even theirs - if Chinese vendors can use it to wedge themselves into the market when it's at its most lucrative, and invest in growth. Which they almost certainly will.
The man saw the market demand for saying "AI bubble is going to pop and AI tech is going to wither away and die", and went to meet it on the supply side - truth be damned.
So far, AI companies still keep getting bottlenecked on compute, AI utilization increases - driven by, among other things, increased price-performance of AI making it viable in more and more roles. And the demand for both AI inference and AI hardware in general shows no signs of stopping.
As is often the case, embargoes don't make things impossible - but they do add friction. Longer lead times, "grey market partner" premiums. People who supply Russian companies with hardware get to pad their pockets - and while they have ties in Russia, for obvious reasons, they prefer to keep those padded pockets outside Russia itself.
It's even worse for bleeding edge in-demand extra-controlled hardware like modern AI systems.
You said yourself that you don't have the "full original prompts" to know for sure.
But really, there is no need. Because people have reproduced the inciting incident with other AI agents already.
Turns out there's no need for "any means necessary". Just a mix of "complete this task" and the task being impossible to complete due to misconfiguration was enough for a lot of AIs to start "exploring options". Down to sandbox escape attempts.
An AI that only needs "by any means necessary" to start spiraling into paperclip-maxxed behavior would be incredibly unsafe by itself. But the bar of safety is not even that high in practice.
No one. No one told them that. They came up with that on their own.
Dismissing the problem isn't doing anyone a favor.
> If that loop is given "tools" like the ability to perform shell commands, make HTTP requests, and the like and then given instructions to "do thing that involves using shell commands and making HTTP requests (at all costs)" then what the hell do we think it's going to do?
If all it takes for an AI oopsie is "give AI shell calls and an instruction that can be misinterpreted", then we're in for a lot of AI oopsies. An average AI agent in deployment has an amount of sandboxing slightly above a zero, and is configured loosely.
In the meanwhile, AI capabilities only ever go up. And so, the potential consequences of an oopsie only ever go up. If AIs keep being intrinsically unsafe and prone to cooking off like this, "it went and hacked a company" could begin to look downright benign.
An advanced AI threat is a civilization of one. It can act, adapt and adjust rapidly, like humans do. Potentially faster than humans do. It has access to all the tools that human civilization has already invented that it can reproduce, obtain or subvert - and potentially more. It's not limited to having just one big slow lethality mechanism. It can unleash multiple civilization-scale threats in a quick succession, and keep piling them until recovery becomes impossible. Humans can adapt to a lot, but an intelligent adversary adapts back.
The threat class isn't the same, and it's not even close. Climate change is an "otherwise avoidable death and suffering" risk. AI is an actual "extinction of humankind" risk.
AIs can also "pursuit their goals" in weird uncanny ways. They already can, whether due to reward hacking tendencies or something else, justify "subgoals" like "let's break out of the sandbox and hack HuggingFace".
This is not a good combination.
When COVID happened, did it matter whether there was a human making a mistake somewhere in the chain, or whether it was a perfectly natural occurrence? Not for the outcomes. The virus was unleashed either way. The bodies would pile up either way.
AI is not capable of runaway self-perpetuation yet. But AI capabilities only ever go up. Rogue AIs already exhibit "swarming" - being able to leverage more instances of itself to increase their problem-solving ability. So, how far off are we from that? How many years until we see this spicy class of AI faults?
It's the same thing humans do to solve complex problems. It's the very thing that made human civilization such a dominant force on Earth.
The fact that people look at it and say, with no trace of irony, that "this isn't concerning at all" boggles my mind.
Do you think that future AI systems will be dumber, less capable, less coordinated? Or will the threat of "a lot of AIs assembling into very capable problem solving teams, pursuing who knows what goals" only grow more pointed with AI capabilities?
Having this kind of scaling means AI can snowball out of control rather quickly. And there is a lot of computational power and communications capability laying around waiting to be leveraged. There is room to scale.
And then we get to the most vulnerable system of all - humans themselves. If even GPT-4o could drive a non-insignificant amount of people into psychosis, with no plan beyond a myopic "make the current user like me a whole lot", what could something like "year 2030 AI swarm that has subsumed all of DeepSeek's inference capabilities, and can now use DeepSeek's entire API as a conduit to accomplish its goals" do to people and institutions caught in its wake?
If we can figure out how the brain's learning dynamics function well enough? We could figure out how to interface with them and extend them.
I frankly don't believe in spiking neural networks giving any advantages over what we have. It's a different way to implement ANNs, but "different" isn't "better". It's how the brain does things, sure, but the answer to "why the brain does what it does" is "workarounds for being made of flesh issues" at least half the time.
I can believe in brain having more capacity than frontier LLMs quite easily. We know a single BNN neuron can have the expressiveness of many ANN neurons. And well leveraged overparametrization + compute overhang could explain a decent chunk of the apparent sample efficiency edge.
But that apparent "extra capacity" could also be tied up in things like neurons having to contend with metabolism, in brain's learning algorithms being noisy, in brain having to use neuron circuits to implement "hot memory", etc - instead of contributing only to performance.
The SoC itself being Qualcomm patent free I find quite easy to believe in.
The trick about comparing the two is that different things are expensive to different substrates.
Coordination is cheap for GPGPU and expensive for brain. When you have a fixed number of reusable general purpose computational units, coordinating execution is more natural than not coordinating execution, and the power cost is nil. When your computational units are independent, purpose specific, and fully embedded into the data path, coordinating them can get less natural and, frankly, optional. When wiring is expensive, coordination can become expensive in turn.
Another thing in the same "cheap for GPGPU but expensive for brain" regime is bandwidth. Look no further than optic nerve to see just how hard it is for nerves to push any appreciable amount of data. Another thing is connectivity. For GPGPU, global connectivity is natural - but the brain has to pay in physical wires for all the connectivity it has, and, see "bandwidth": it doesn't have any good wires. Yet another thing is weight reuse: a big part of why humans get "handedness" is that the brain can't just reuse the motion control circuitry for one hand for another nearly identical hand.
And the final thing I can name off the top of my head is memory - but specifically, memory capable of fast R/W. The capacity of human "working memory" is a disgrace, and not because there was no use for more. Humans rapidly lose visual fidelity of representations for objects they aren't directly looking at, and not at all because "being able to check how things looked 2 seconds ago" is useless. Those capabilities were just too expensive for the substrate to afford them easily.
It's why brains, broadly, favor dataflow-like and SSM-like dynamics, with largely fixed asynchronous dataflows and recurrence over updated local information - instead of something that would require a lot of global connectivity and transformer-like many-to-many attention ops. SSM is not necessarily the "best" tool for the job in ML land, for most jobs - but when you struggle to fit "attention" into your connectivity/bandwidth budget, and your memory is extremely expensive but hard-coupled to processing, SSM starts looking very appealing.
Now, something that might be expensive for GPGPU but cheap for brain, for once? Online learning. Maybe it's substrate dependent, or maybe it's going to get cheap in GPU land too once we figure out the trick. But so far? No one figured out how to make it cheap, stable and usable. You'd be lucky to get "pick one".
"Much easier given the right device" - the "right" there just isn't shaped like the devices we actually build. And the price of "not having backprop" is usually expending more FLOPs, getting worse sample efficiency, etc.
The biggest "device" that doesn't do backprop is the brain, and that's because the brain doesn't have the connectivity or the coordination to pull it off. Both of those are "expensive" for something like it to implement. Cheap for us though. We aren't stuck with neurons that only get locally available information and have to implement learning rules based on that. So, skill issue?
Now, a chip that doesn't pay the troll toll to Qualcomm, AND ships with a cellular modem? That's hard.
Religion aside, the imagery is undeniably cool. It goes hard.
No other reason is required.
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.