> Except, since we're talking about scientific research, and not all humans perform scientific research, the only relevant group to consider would be humans who do scientific research.
Sure, but I expect that to change the distribution, not much else.
> I don't think it's common for researchers to randomly fabricate sources, citations and results. It happens, but it isn't mere coincidence when it doesn't happen.
Hopefully not, but even in science, misunderstanding the cited work is a thing, as is disbelieving it. For the former, "perceptrons can't do XOR" became "AI is impossible", while the Black-Scholes equation was misapplied to ultimately lead to the global financial crisis. For the latter, the salient examples are older (e.g. the importance of washing hands between autopsies and midwifery), or things where I can't find the historical scientific consensus because of the modern cultural noise (e.g. thinking men and women have different numbers of ribs because of the book of Genesis and not checking).
In this regard, I think the replication crisis is valid, even though it (presumably) involved real research and real results, because widespread citations didn't fully account for the limits of that research.
To put it another way, while I agree with this:
> To equate that process with the result of actual human beings doing science and publishing research seems like you're just reaching to justify the validity of replacing researchers with AI.
With scientists I'm referring more to the game of telephone that necessarily occurs between any given researcher and anyone using their work.
For the general population it's only that game, hence all the people who think scientists in the 70s were worried about an ice age, or the newspaper stories that collectively divide the world into things that cause or cure cancer.
If you are referring specially to people who want an LLM to actually perform novel research, rather than merely write up lab notes in LaTeχ, then you would be correct that this is a terrible idea. Even for pure mathematics, you need a different type of AI, better suited to that task — they do exist though, and have been used successfully, but IIRC are not anything like as general with regard to maths and experiments as LLMs are with language.