Modern AI works by maximizing the correctness score of an answer. That's the goal.

It does not maximize its chances of survival. It does not maximize the count of its offspring. Just the correctness score.

We have taught these systems that "human-like" responses are correct. That's why you feel like talking to an intelligent being, the models are good at maximizing the "human-likeness" of their responses.

But under the hood it's a markov chain. A very sophisticated markov chain, with lots of bling. Sure, when talking to investors, it's the second coming of sliced bread. But come on.