663 karma · joined February 16, 2026
Some relevant github issues: https://github.com/aria2/aria2/issues/1399 https://github.com/aria2/aria2/pull/2004
> The way “recursive self-improvement” is being used recently seems to be intentional marketing. A model helping code/train/test the next generation of its model family isn’t technically “self”-improvement.
The term RSI is ~20 years old at this point, and the concept is even older. If an AI can't continuously modify itself and instead has to build a successor that builds a successor, that changes some specifics (for example, the alignment problem becomes more relevant - how can the AI ensure that its successor shares its goals?) but it's still a situation that may produce exponential growth, and hence most RSI-related considerations still apply.
(In fact, as a toy example, note that it's in theory possible to get exponential growth in capabilities even if you hold the software fixed, if e.g. you have an AI that can design and build better hardware to run itself faster on, which lets it design even better hardware even faster.)
Yabloko is the sole officially registered party to openly oppose the Russian war in Ukraine, openly calling "for peace and freedom". The party planned to partake in the 2026 Russian legislative election under their anti-war slogan.[29][30] However, in August 2026, the party was barred from running by the Supreme Court of Russia due to a lawsuit by Rodina.
Also, several politicians associated with it were imprisoned, e.g. this guy: https://en.wikipedia.org/wiki/Lev_ShlosbergThis comes across to me as a surprising question - why do you assume there's such a bio at all, rather than it being developed by someone anonymous, as is the case for many questionably legal privacy tools?
Well, yeah. It's almost as if LLMs are different from knifes.
> The person responsible who lost control is still responsible for losing control.
That's valid, and yet one can advocate for AI companies to be culpable for actions of their agents, while still admitting that rogue AI agents are a thing that exists.
Increasing effective depth like this is bad for safety because it can ruin CoT monitorability: the reason why looking at the model's CoT actually gives you info about what the model is thinking is that the model can't do enough thinking in a forward pass alone to solve complex tasks, and hence has to do multi-step reasoning in CoT. The more thinking the model can do in a single token's forward pass, the more opaque the model's reasoning is, and the less reason there is to believe that what it writes down in the CoT has anything to do with reality.
For Astra specifically, the impact seems to be limited to a moderate monitorability hit, like the concerning result from the model card that Astra is notably better than any model before at solving problems under the constraint of not mentioning the answer in the CoT. The really bad scenario, however, is that this may create a race to the bottom where OpenAI and Anthropic feel the need to use more recurrent depth in each generation to not get outcompeted on capabilities, completely bricking CoT monitoring for both model families. Or, worse, the pressure to compete might push them into one of the worse techniques, like training on the CoT[2], or eliminating human-readable CoT and letting the model think entirely in neuralese.
For more details on recurrent depth in Astra, see "Part 2" here: https://thezvi.wordpress.com/2026/09/08/astra-is-hard-to-mon... , or this article mentioning some expert responses: https://techcrunch.com/2026/09/02/openais-new-reasoning-tech...
[1] The model card doesn't mention it at all; it was reported by The Information prior to model release, then confirmed by OpenAI researchers.
[2] https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-f...
(Incidentally, if you asked me a few years ago I'd have predicted that reliable AI detection was impossible. I still suspect that it's impossible in general and that Pangram would break as soon as companies figure out how to make LLMs stop all sounding the same - but until then, it'll work fine.)
For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml - I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation, even given hundreds of years of theoretical research and several attempts in the real world? I think the closest real-world example of this is space exploration - it's extremely expensive and takes years to put stuff in space, and yet probes and rovers we launch end up working most of the time. On my model, it's because we have a decent grasp of the underlying physics in enough detail to predict real-world complexity, and we have toy testing areas we can test the tech on, and if you do this cautiously enough you can, in fact, launch a thing that ends up working.
I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
I don't think this is plausible, most notably because "AI research" is one of the tasks that clearly belongs to the digital world. You're suggesting that in the future all AI research is done by AIs because they are better at it than humans, and yet capabilities plateau at that point, instead of going into the RSI regime. (And it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics.)
Hasn't that already been done several times? There's feather-rs (fairly mature but reduced set of features) and Pumpkin and FerrumC. And, for that matter, Paper - that one is Java and I haven't checked to what degree it's just borrowing from the original implementation, but I do know that Paper is actually used for real servers.
This is good and admirable, but it'd really suck if by trying to build god without knowing how we end the human species. We could simply wait some more decades until we actually have any idea what we're doing, and then do that without the risk.
A "mathematician" is a human who decided to spend their lives studying mathematics. Mathematicians also tend to be smart, but intelligence is innate, not acquired, so studying mathematics doesn't make you smarter. This makes it obvious why they don't rule the world - if you want to rule the world you'd want to focus on that (for example, doing business or finance), and becoming a mathematician is just a waste of time.
LLMs don't work like that. Like in humans, all of their capabilities correlate, and unlike a human, their overall capabilities grow over time. Looking at LLM mathematical ability over time* therefore gives you info about the progress of their general capabilities, and ability to take over the world would be determined by the latter.
* In fact it'd be better to look at a mix of different capabilities, but that's growing too at about the same rate, see https://epoch.ai/eci
All there really is is arguments from theory (most notably, the orthogonality thesis and instrumental convergence). It's not like we have a spare universe to conduct experimental research on the likelihood of human extinction. That said, the 5-10% range is not some specific person's wild guess, it's the median wild guess of a poll of AI experts: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
> or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
They are not "trying to position themselves" that way - back when those companies were founded, the people who did it were already convinced they were trying to make the ("inevitable") development of AI go better. It's not some novel galaxy-brained marketing strategy, it's just what they thought the whole time.
Really? Here's a longer Bill Gates quote (from https://www.nytimes.com/2026/09/29/opinion/ezra-klein-podcas... ):
Ezra Klein: "So why is anything needed beyond — and is anything needed beyond? — the simply natural incentives under capitalism and normal corporate reputational management?"
Bill Gates: "Well, I almost can’t believe you’re asking that. This is the most dangerous thing that humans have ever gone near. [...] You can take an open-source model that can create bioweapons and disable any monitoring of any kind, and this exists today. So no, there is no filtering of any kind. And so say you kill 100 million people — you want to use a lawsuit? I almost can’t keep a straight face."
You're saying that if one were to describe this scenario in more detail, it'd be less of science fiction? That's a bit against the grain - usually it's the more detailed arguments that get dismissed as science fiction, while the less detailed ones get dismissed as abstract theorizing.
If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.