"The researchers showed that GTP-4 ADA “created a seemingly authentic database,” that supported the conclusion that one eye treatment was superior to another. In other words, it had created a fake dataset to support a preordained conclusion. This experiment raises the threat that large language models like GTP-4 ADA could be used to “fabricate data sets specifically designed to quickly produce false scientific evidence.” Which means that AI could be used to produce fake data to support whatever conclusion or product you want to promote."
Note also that faking data is not only about science. I'm pretty sure we're going to see fake datasets for fake market analyses, fake polling data for fake political campaigns, fake data from fake data breaches, etc., etc.