Even if we take the argument at face value, we should allow LLMs to train their models for free, on the backs of real people's work, just so that there's a chance that they actually improved well enough to replace humans, all that just to have a temporary boost on search discovery of our content.

Not to mention LLMs still spew a lot of badly wrong results (no I will not anthropomorphize the models, they're not ready yet).

This is one heck of a poison chalice. 王先生，你願意喝這杯鶴酒嗎？