It depends what they mean by automating. But the classic argument goes:

* Each new generation of models has emergent capabilities we did not anticipate.

* We already have trouble monitoring and controlling the current generation (see HuggingFace incident).

* The more we let models shape their successors, the more out-of-distribution each generation's learning environment becomes.

* If not done carefully, we risk creating extraordinarily intelligent and powerful models with unintended behaviors, like deceptiveness or power-seeking.

Skynet.