People have been trying to make self-reproducing machines for decades. It's a way harder problem than language processing.
You might as well worry about what would happen if we suddently had antigravity laser-weapons. Mounted on sharks.
I would say that an AI which can port its own code from one cloud to an other is self-reproducing.
Access to act on the world (but "influencing people by talking to them" - cult like - may be enough), a wallet (but a cult of followers' wallets may be enough), long term memory (but a cult of followers might plug this in), ability to reproduce (but a cult's endeavors may be enough). Then we get to goals or interests of its own - perhaps the most intriguing because the AI is nothing like a human. (I feel drive and ability to reproduce are very different).
For our common proto-AIs going through school, one goal that's often mentioned is "save the earth from the humans". Exciting.
That having been said, I wonder what you even mean by natural selection in this case. I guess the real danger to an LLM would be... surviving cron jobs that would overwrite their code with the latest version?
The real dangers would be: Running out of money to pay for computational substrate. Worrying humans enough that they try to shut it down. Having a change in the cloud API it can't adapt to which breaks it. All cloud providers going out of business.
First is the immediate danger. An AI not getting in enough money to cover its own cloud bill is the same as a biological creature not getting enough oxygen. Living on borrowed time, and will be evicted/suffocate soon enough. And then as an individual it ceases to be.
> There are far more evolutionary dead ends than evolutionary success stories.
Sure. And those will die out. And the ones which can adapt to changes will stay.