Very similar. Might even say it's the same thing viewed at a different angle.

We don't know the details of OpenAI's implementation, mind. But the "limited layerwise depth" problem was known for a while now, and both OpenAI's approach and this are getting at it. And that's far from being the first attempts to address it.

Training and stability were the main issues that stopped it from being adopted rapidly and widely the way something like MoE was. But I think it was always a matter of time.