First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.

Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.

Let them compete in the market.

Start a new company with version N+1 leading it, and give it some funding. If it outcompetes version N company, it can acquire it.

I think in order to achieve AGI we have to simulate what life does. Imagine having a bunch of robots with the ability to adjust themselves, just hanging out without a goal, experiencing world on their own. Eventually evolution will filter out those robots that aren't good at, you know, "being intelligent in general" and break without self-repair, while the rest will grow into a society.

Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.

The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.

The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.

This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.

It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).

There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.

This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.

Yes, and therefore RSI isn't RSI. You need to set up multiple environments to generate data sets, multiple RL envs and just train the models on different datasets, architectures and then compare the final results against each other. The RSI part is just generating these pipelines end to end automatically.

What the hell are you talking about? There’s like 600 benchmark tests you can do.

Picturing advanced AI models eventually resembling Philosophy majors arguing about the number of angels that can dance on the head of a pin while the rest of us humans are left wondering if they truly have achieved N+1.

    > LLMs are super lazy about caring about wall clock time.

My bandaid for problem is to prefix every bash command with "timeout 60 " at the harness level and remind the LLM to hurry up. Of course, this is not a perfect solution, since LLMs might try to run jobs in the background or split up work into smaller pieces instead of running smaller experiments. And of course this does not help with the multi-million dollar training runs, but it has served me well most of the time.

I tend to think it's going to turn out to be something along the lines of a perpetual motion machine. In the sense it seems very possible on paper but the subtle facts of reality prevent it.

People will make things that look a lot like self-improving machines but on closer inspection that won't truly be the case.

I suspect this violates the Labor Theory of Value in the same way that perpetual motion violates the laws of physics.