"I keep reading these opinions that LLMs are just doing some advanced form of copy paste. Actually, we don't know what they are doing. Are they actually doing some form of modelling and abstraction? Seems likely to me."
This is exactly the problem with AI. For business or government, the answer is as important as the methodology employed. A black box does not work for the majority of use cases.
Until it can show its work, it's a sideshow.