Animals get up and go do stuff mostly because they need food and sex and things in order to stick around, so we're surrounded by organisms like that. Imperatives like hunger and mating instinct create spontaneous action. The ability to sample from a modeled probability distribution is often useful in those pursuits, whether it's a model of the spatial world or of a corpus of pick-up lines that work well on a Friday evening, and the hyper-scaled transformers have more than demonstrated that kind of modeling and sampling.

Take it a step further and you've got AlphaZero or something: it's sampling from a modeled distribution of moves that win games. But there's still a `while`-loop somewhere saying: time to sample a move. That part is not novel or mysterious.

There is no demonstrated technology that I'm aware of where the "alignment" needs to be with the big model: the alignment needs to be with the person writing the `while`-loop. If someone has a machine repeatedly sample from a distribution of moves that DESTROYS_ALL_HUMANS_BEEP_BOOP, then your beef is with that person, not the model.

Now if someone trains an RL agent with a loss around chasing sex or fame or fortune, we might have an issue. But that's still sci-fi stuff AFAIK, and it's a little hard to take the mix of real stuff and sci-fi that passes for much-if-not-most "AI Safety" discussion seriously, especially when you consider that the `while`-loop authors stand to gain a great deal by focusing attention on the modeling part.