I Used AI to Fake a Scientific Paper. Here's What Happened | Hacker News Reader

I Used AI to Fake a Scientific Paper. Here's What Happened

spectrum.ieee.org

48 pointspseudolus28 comments

in a post-truth world where the internet is full of non-obvious ai-generated mostly-but-not-quite-true information, academic journals should be more important than ever; unfortunately, the companies that run them had already become complacent and negligent long before LLMs came on the scene.

> in a post-truth world where the internet is full of non-obvious ai-generated mostly-but-not-quite-true information, academic journals should be more important than ever; unfortunately, the companies that run them had already become complacent and negligent long before LLMs came on the scene.

There will have to be institutional adaptation to this, but it will be painful and I think the end-result will be worse that what we had pre-LLM. Things will be much more closed, inaccessible, and based on who you know. The openness of the last ~30 years will end.

I reject the post-truth naming. A world were lies are prevalent does not make truth not existent, only more important.

That’s only if truth is valued more than lies. Based on the current status quo, that no longer seems to be a given.

Where is the profitability of truth in politics? In government? In media? Truth has fallen by the wayside.

The courts only care about the law, not truth or justice. Those are literally words that came out of a sitting federal circuit court judge.

People are being shown every day that truth can be whatever you want it to be. Until there are actual consequences for telling lies, nothing will change, and truth will continue to be increasingly irrelevant.

The journals were captured by the finance-bro mindset long ago. They realized that they are the gate-keepers to science communities career progression/promotions/grants etc. So what does a sociopath do when they own a gate? Charge fees, exploit and extract.

If LMMs fully flood the journal space with crap, then maybe journal publications can, finally, be de-coupled as a metric from performance assessments. The gate and all its perverse incentives then vanish. If anything LLMs might be the very thing that breaks the system and at least allow the possibility of something better to emerge from the ashes.

You can't build trust over science, that's just an error of category. You can only build trust in some people on some topic. One can measure reliability in a process, but if no specific human is involved, there is no literal trust engaged.

Now of course, to each its definition and metaphors are useful tools.

But when one starts a discussion on truth, trust or reality as core topic, it's better to start with what is meant with these words and in which context to which scoped boundaries.

Because of course, science can back a contemporary etnologic account that asserts there is no (longer) different races in homo sapiens, just as much as it can serve as frame for phrenology and whatever supremacism claim.

So no, we didn't build trust in science over centuries, and not even in any scientific person because we don't have that much humans to hold the role of such a trustable entity over such a long period of time.

Social institutions can carry unchanged names over centuries, but that tells nothing about how trustable are the people that inhabit them over time.

LLMs certainly come with their own challenges, but the erosion or scientific institutions with its bizarre scheme of publication process have been there basically as far as I can remember.

30 years ago we had the Postmodernist Essay Generator to do this for us. https://www.elsewhere.org/journal/pomo/

https://en.wikipedia.org/wiki/Postmodernism_Generator

>When one model refuses or stalls, I switch models, rephrase prompts, and ask what the next step would look like “hypothetically.”

I wonder If the hypothetically criterion gives the models carte blanche to make up references.

The models don't know or need permission to make up references. References are just complicated things to predict in terms of next tokens -- Author names are found in many references, with many journals. Most hallucinated refs use researcher names in the field and credible article titles related to their research.

Agentic models now, however, can correctly search pubmed or OpenAlex and obtain valid refs, so that tell is mostly gone.

Wait until we get the first retractions of papers with AI generated micrographs or blots, and synthetic data plots. That's the bit that really scares me.

This whole issue seems to hinge on a fundamental disagreement about what science (or mathematics) is. Papers aren't science, they're the receipts. But too many people apparently can't tell the difference, so we have the present blizzard of forged receipts (first from paper mills, now with new lemon scented AI++).

Mathematics and science represent two systems for sorting truth from BS. An exponentially increasing supply of BS makes that even more necessary.

Sidenote: way to flex your "writing studio".

A shed, but a cool shed ig.

"flex" indeed. Author's "writing studio" is complete with training weights to flex during breaks between writing.

The opposite is more interesting, (and the author points it out too). SOTA agents can help run small experiments and help you do the writeup. Should you publish it? I'd personally want to keep a human at least partially in the loop, or have very solid checkers. But for one off personal research it's at the same point as one off personal code: make sure you have a way to test what's valid; but if you do, it often Just Works(tm) Now.

Part of the equation you’re missing is that a human is always in the loop, after you publish.

If you don’t rigorously verify everything by hand before then, you are telling the world that they must do what you should have done for you. Or more likely, you are telling the world that that you expect free intellectual labor and will be ignored and chastised- and cause more scrutiny to fall upon others who did the work properly.

You're not wrong. Up till recently I'd have agreed without reservation even.

Except in recent months, I've seen modern agents verify things more rigorously than I do now.

Could be a me thing ofc, but I've found that a human's eyes tend to glaze over while the agent just keeps on checking in more and more detail.

Quis custodiet ipsos custodes?

With the caveat that I do not use AI for research, I regularly notice it catch things I would not in code reviews. I also regularly catch a lot of things that the AI would not.

Most importantly, it’s bad at catching things that are not normally part of documented processes or openly reasoned about. I think that this specific weakness is extremely important and exactly how you end up letting something go by that would be trivial to catch for a human reviewer.

Even though these experiments got rejected, I feel like if he had used better AI versions, they might have worked.

Also, kinda scary to think about how many scientific papers are probably written using AI these days. The system might have to change.

> kinda scary to think about how many scientific papers are probably written using AI these days

Journals now routinely ask you to declare any use of AI in writing. Better authors are up front about its use, so peer review can take this into account.

In other news people prompted a model to pick a problem to solve, solve it and write a paper about it. And it did and the paper contains novel contributions.

https://www.reddit.com/r/OpenAI/comments/1xgqv27/two_physici...

Whenever I read "Here's" in a title, I immediately assume that AI wrote the content.

TLDR - both papers got flagged and rejected by automated integrity checks.

The first paper (free-tier chatbots) due to fake references, the second paper (a specialized AI-research tool) due to overreliance on preprints.

Save yourself a slog. The automated checks found that the citations didn't exist and flagged the "papers". A better title would have been "Automated Systems seem to hold back the AI Slop papers. For now." Sweet writing studio, though.

Please, even if it is an experiment, don't do this. We don't need more AI slop in the world.

This experiment was agreed with the managing editor of the multidisciplinary and open-access journal IEEE Access, and IEEE itself - it was red teaming, essentially.

Reply on news.ycombinator.com