One of the big points is that looped architectures don't necessarily remove CoT style reasoning. They definitely change it, but the model can still be prompted for this kind of output. It just spends more computation on individual tokens. I think the hope, in that particular scenario, is that the model spends less tokens on things like backtracking.
But there are lots of approaches to looped transformer architectures. Some have some kind dynamic loop (i.e. they might loop until they satisfy some kind of equilibrium, or they might have some kind of routing function that makes different tokens loop differently, or any number of other approaches). Others are simpler "loop this many times" architectures. It's a really active area of research with some pretty deep roots, if you want to dig in! This paper blew up a while ago, and it's a looped transformer architecture (though not like the ones discussed here) that isn't autoregressive, as an example: https://arxiv.org/html/2510.04871v1
1. coarse grained processing
2. fine grained processing
Tokens are coarse grained processing. While I agree that during a token generation, that processing is fine grained.
But Looping is fine grained processing or reasoning and it is not limited to discrete tokens within passes. This is qualitatively different because a lot more "stuff" happens between tokens. You are right that it is similar to a larger model but then even COT is similar to using a larger model.
I thought a looped transformer is in fact reasoning more in latent space, though "reasoning" in a somewhat different way from the standard reasoning token approach.
I certainly think you could say both of those are reasoning in latent space. But of course people can have different definitions of what they consider reasoning. Maybe this is because I think it's hard to pin down what "reasoning" is so I instead alias it to "computation" which both of those examples do.
• Parameter efficiency. 1.4B and 2.6B parameter looping Ouro models match 4B and 8B transformers on most benchmarks.
• Adaptive computation. Looping is trained in a novel way to avoid exiting too early or too late.
• Reasons to use looping. Looping does not increase raw knowledge storage (approximately 2 bits per parameter for looped and non-looped models) but dramatically enhances knowledge manipulation capabilities.
• Improved safety and adherence. They find that safety and alignment of CoT to final output improves as recurrent steps increase.
There are some safety/alignment attractor basins that are showing some signs of potential importance(such as scale-invariance) with the current/future prospective systems moving forward, and an ever increasing peak utility regarding such architecture(s).
An addition of some kind of, "spoof injection reclamation effort" might also supplement this, as well as, some ways to shunt the "dragging" effects from adversarial timing/fatigue attack(s) down at these interaction scale(s)/level(s).
We don't know the details of OpenAI's implementation, mind. But the "limited layerwise depth" problem was known for a while now, and both OpenAI's approach and this are getting at it. And that's far from being the first attempts to address it.
Training and stability were the main issues that stopped it from being adopted rapidly and widely the way something like MoE was. But I think it was always a matter of time.
- yes, you can definitely do that
- but you'll never be sure the two systems aren't colluding to hide things
- yet, even for layers that don't output text directly, you can find meaning in weights - there's an old example where you provide the LLM samples like 123, 321 and sampling from the middle layers lets you find the word "reverse" which is not in the output