You are pointing at something that is orthogonal to this paper. The LLM did not randomly recommend or bring this disease up to people - it merely assumed the disease was true when the preprint was pointed at it.
If the person put their product as th definitive cure for the made up disease, the LLM probably would have mentioned that too.
> merely assumed the disease was true when the preprint was pointed at it.
What do you mean by preprint pointed at it? It being the disease?
This is not true - the model was not trained on this fake disease. It brought it up because it found it during real time search.
>What do you mean by preprint pointed at it? It being the disease?
On this I'm wrong - it turned out that the model brought up this disease even when not mentioning it explicitly.