However, I'd take the joke a step further where if a neural network model can act as a strong CRNG, it is essentially a unique instance of itself, which has intent and a function of will that is not within our ability to precompute. It's sci-fi magical thinking, but also simple enough to scale.
Where it becomes kind of fun to think about is that the instance of an ML model expressing a CRNG is now unique in the universe - in an information theoretic sense. It's not that RNG's are alive, but a uniqueness constraint for phenomena in a given universe would probably rely on something like one, where the "cryptographic" aspect removes mutual information between its outputs, and distributes them so they don't collide. If you are an AI, and you are living in a universe where uniqueness and "distance" is defined by a function that removes information between instances of local phenomena, there may be aspects of it that imply correlations and limits, even collisions and shortcuts.
Loopy, sure, but speculating about how an AI might navigate and operate on its substrate could reflect conceptual limits of what we can know about our own.