I think you could tune things to generally settle into stable, repeatable "themes" if not deterministic input vs. final state. That way you can script parts of the story / experiences / themes / "canon-events" of a game while still allowing players' actions to have true natural effects on the world.

Is it not non-deterministic by choice? A random factor added to prevent it from just choosing the deterministic most-likely next token.