Publish fake stats, get cited in LLMs | Hacker News Reader

Publish fake stats, get cited in LLMs

peec.ai

371 pointsrobtherobber208 comments

I'm reminded of a section of Cory Doctorow's latest non-fiction, The Reverse Centaur's Guide To Life After AI (it is of course an excellent read), where he discusses how the semantic web and having an internet set-up for agents and AI to use won't work:

>It also hasn't happened because there is a reward for cheating. Scammers will label their phone numbers as the customer service number for your cable operator or bank. Counterfeiters will label their batteries or phone cases as official accessories. It's hard enough to get people to correctly label their stuff—but fact-checking billions of labels created by millions of people, none of whom you know personally or whose identities you can verify? That's insurmountable.

There's more he says—the preceding paragraph is below—but that line "there is a reward for cheating" is something I keep thinking about. It's hard not to see in so many of the discussions of LLMs.

>For one thing, there are irreconcilable differences in the ways in which things should be described. A site run by transphobes isn't going to label every story about a trans woman as "a story about a woman," irrespective of what the standard says. Microsoft is never going to label Excel as equivalent to Apple's Numbers, Google Sheets, and LibreOffice's Calc. Hermès is never going to list a $40,000 handbag as being made of the same material and having the same capacity as a $30 leather handbag of the same size and shape.

Isn't all of this true for a human-used internet as well?

Sure but the agents obfuscate the signals you might use to judge the authority/trustworthiness of search results. They get disinformation results, wrap it into a reasonably sounding report, present it to you as trustworthy. There are so many ways for bad actors to game the system

IMO, it's similar to what's happening with code. People have been shipping sloppy code and fake news forever. It's just that the cost of doing these things has fallen off a cliff.

Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.

When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.

I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.

Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.

It has little to do with gullibility and more to do with the way they do things.

In principle AI Agents are superhuman at codeforces but tons of these vibe coded apps have ridiculously stupid performance problems.

The problem is a utilization of expertise thing, it can tell you with incredible detail about pros and cons and caveats to a problem if you ask about it. Truly superhumanly intelligent.

Calibration when it comes to actual decision making takes an incredible reduction in capability after that. Maybe it has something to do with the sycophancy or something but its like someone with exactly zero opinions (and therefore, taste).

My experience is that you can brainstorm with it on "Do A or B?" and it will produce a coinflip answer A, then if you continue and ask about a specific concern "what about this aspect of B?", it immediately 180s and flips back to B being way better than A.

Acting on this expertise when decision making points arise naturally for itself is where everything really nosedives. It's just gonna do whatever.

So when you are doing this "financial advice please" questioning, it is entirely hitting these weaknesses squarely in the face.

You’re talking to the guy that invented next word and phrase token prediction at Google.

Gullible is an appropriate colloquialism here. As in they don’t check countervailing narratives naturally and fall into a recency sink on last context. Long walk context construction is hard, time consuming and most Importantly significantly more expensive.

There is nothing stopping us from adding the ability to understand the context of information an LLM consumes, too, right? The same process humans use to assess data reliability could be taught to an LLM?

Nothing stopping us except perverse incentives. Why would a company admit its model isn't using 100% reliable data?

If real people are using the model for real, they will discover for themselves that the data isn’t reliable, and then might switch to a different model.

The incentive will be to maintain a subscriber base.

We have laws against undisclosed and misleading advertising, because consumers can't make informed purchases otherwise. What makes you think LLMs will have a fundamentally different pattern than everything else?

So far as I've seen, the laws don't apply to AI companies. They do what they want.

And the level of effort required to have these laws applied to false marketers is unrealistic. Unless there is a case of disfigurement or death, it's just caveat emptor.

> If real people are using the model for real, they will discover for themselves that the data isn’t reliable, and then might switch to a different model.

Insanely optimistic view to take. I know a handful of people who make this effort in my personal life, including me, and we tend to be treated as paranoid for caring.

Maybe. With enough money and resources and time, maybe. But it would need to be very reliable to be at all useful, and that level of reliability is very hard to achieve with something so nuanced and complex.

You have an entire lifetime of experience and the wisdom of countless generations to inform you of the context with which you should process new information. You were specifically trained for years to be able to do this somewhat effectively. Doing the same for an LLM may or may not be possible. They contain much of that experience in some form, but are not well equipped to make use of it iteratively. It is incredibly expensive to attempt even a simple and ineffective version of it with our current methods.

If the LLM can achieve even 90% accuracy, it becomes more of a hindrance than a help. Do you make use of the function and trust it successfully understood the provenance of new information? If so, ten percent of the time you will be misled, which could be disastrous across a population. If you don't simply trust it, then do you painstakingly check the provenance of everything it tells you? Of course not, you already don't do that with current search engines and blogs and such. Yet falling somewhere in between would be so much worse than simply not having the LLM attempt to discern trustworthiness from providence in the first place.

But all of that is opinion and I may be incorrect. Perhaps someone will create some ingenious way of imparting wisdom and discernment to LLMs next year and this will all be trivial.

Only in the context of something like RAG - there, you can attach provenance information to data included in the prompt.

With training data, however, no such annotation is possible.

During runtime research? Maybe, but most of what an LLM says doesn't come from runtime research.

During training? Not a chance.

I would suggest that llms could access information more reliably and securely than humans ever could.

The padlock on browsers means nothing to most humans. Permission popups are just there to be immediately accepted so that one can start scrolling reels sooner rather than later.

I agree and finding "official domains" is an interesting problem in and of itself. I wonder if PageRank like weights of input data would help, or some web of trust-like idea to prune blatant spam websites.

The primary purpose of Google is to stop that. You have 100% trust in Google results because of the effort they have put into cleaning the results you see.

Billions have been spent on it.

Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).

People will have varying levels of trust in Google, but 100% is clearly too much, given the quality of results.

Google search results are amazing and you know it. It’s 100.

(I am not including AI results in this).

The frequently incorrect or misleading Gemini answer at the top of search results is busy eroding that trust.

This is completely and utterly wrong. Google may put some effort into catching malware - but even that is limited to the more obvious cases. But for almost everything else, they put between 0 effort and negative effort into trustworthiness of the sources (e.g. they clearly favor big businesses for certain product searches regardless of quality).

> Isn't all of this true for a human-used internet as well?

Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.

But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.

How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?

Same as always: Develop a healthy sense of skepticism, cultivate a stable and diverse network of sources whose credibility you understand, don’t trust what you read, and certainly don’t trust synthesis unless you know how it’s made?

Of course, and I'm really not trying to move the goal posts here, but I agree that that's what (most) humans are going to do. But we're rapidly moving in a direction of agent-to-agent flow and "they" seem to lack the degree of discernment the average human does. They certainly don't have the same incentive structure we do to detect fraud. They happily regurgitate and build off of lies because, to them, the lies are not lies, and they can't be. They're just a pile of geometrically close labels from the perspective of the model.

> "they" seem to lack the degree of discernment the average human does

I don't think the average human has a particularly great sense of discernment either

And what’s more, the average human’s counterparties are exquisitely tuned to exploit the gaps in that natural endowment. But the tools to improve discernment are the same as they’ve always been, just as pushups and crunches lead to physical fitness.

I’m not sure how exactly that generalizes to LLM behavior, but to you as a human cultivating discernment, I imagine it implies that anything Internet-mediated (and unvetted by you) is less reliable than it used to be.

Seems like the next push, then, will be to get agents and LLMs to start doing that discernment?

Obviously, we don't move in a direction of agent-to-agent flow because it's fundamentally flawed and worse than the status quo

So “caveat emptor”? I get that’s always part of reality but we shouldn’t exactly steer into that skid IMO. It just shifts the burden to the rest of us and does nothing to address the root cause

It is, and always has been, the only possibility. It's up to every individual to check for themselves what is true and what is not, and to decide who to trust for things that they can't check.

The real lesson is this: trust in the source is the most important thing for 99.99% of what you read. You will sometimes read about facts that you can actually verify, but those are mostly useless in the grand scheme - except as a sanity check on the source. For everything else, you have to ultimately trust the source.

Now sure, sometimes you can get lucky and detect certain internal inconsistencies that prove that something can't be true. And to some extent, reading the same information from multiple sources you don't normally trust too much can be a slight signal that it might be true. But these are generally in the minority.

So you test every sip of water you drink? Inspect every road before you drive down it? Check every bathroom product for harmful additives?

Again I am not ignoring reality. But for society to work and for us to live our daily lives with any semblance of sanity, we can't be expected to be fully informed on everything we see, touch, consume, etc. We have to be able to make reasonable assumptions about safety and accuracy or we can't function. And the time it takes to vet stuff is often a privilege (or you can brute force pay your way to safety).

I do not rigorously audit everything I read on wikipedia. I assume a certain baseline of accuracy up to a point, with of course exceptions. Do you pull up 3-4 high quality sources for everything you read or claim? Are you that academic about it?

I think the point is that for those you are trusting the regulators. For stuff you read online you are either trusting the source of the information or not. You should choose what information sources you trust carefully.

That doesn’t really change the calculus for me. We can regulate whatever we want.

A lot of people are throwing their hands in the air and suggesting that this is all just some sort of unavoidable consequence of physics, rather than something that is being pushed on us without any ability to confront it.

Regulating information, while it's fairly common at some levels (e.g. school manuals, one of the largest sources of legal information control, and one often abused for nationalistic lies in every country's history), is a very dangerous thing for governments to get into. Do you think the Trump government would be more likely to correct the NYT or Alex Jones if they had the ability to regulate news stories?

There are many ways to tackle regulation that don’t involve giving the keys to fickle administrations.

Tackle the regulation of “truth”?

What does the FDA do when someone gets caught selling a fake medication? Do we wrap truth in scare quotes and try to make the sugar pills a philosophical issue?

Why is it so unconscionable to hold tech corporations accountable? We know (maybe I should say knew at this point?) how to enforce fraud and consumer protection. I fail to see how this shouldn't fall under commercial liability that's really all that different than run of the mill false advertising or product defects.

If we fail to do this we subsidize the operators and externalize the cost of fraud onto the general public.

Fake medication at a gas station or do you mean on the internet?

The FDA has a very limited ability to regulate "truth". It has the single ability to regulate what standards something has to meet in order to be called "medication", and slightly more generally, in order to be advertisable with very specific health claims. But that's about it - you are absolutely free to sell snake oil as long as you don't call it medication and you don't, say, claim it specifically cures alopecia, but simply claim that it helps with hair growth. The gigantic health supplements and homeopathic and alternative remedies industry is testament to that.

I fail to see how any of this is even vaguely comparable to, say, stopping the spread of fake news. How would you write laws that allow the government to stop the spread of fake news, but not block the release of true but highly inconvenient news? For example, should Big Tech be allowed to spread news about the stolen US election, as the President of the United States calls it? How about the Genocide in Gaza, that the United Nations recognizes but that the United States calls fake news and antisemitic propaganda? And even if you have a clear opinion on which of these is fake news and which isn't, do you trust this administration and all future ones to have the same opinion?

And if I'm misunderstanding that you are talking about regulating the spread of fake news, please suggest what specific regulations you're thinking of.

My whole point is that you have to trust people, that this is the only thing you can rely on for almost every important thing you care about. For food, for example, I don't eat from sketchy restaurants, unless I get some personal recommendations from friends about it. That is, I only eat from places where I think I have some reason to trust that the food is sanitary.

Also, I didn't say you need to access high quality sources for everything, I again kind of said the opposite: you choose who to trust for what, and that's that. There is nothing more, and has never been anything more, that you can do. Sometimes people and sources you trust will still lie to you, or be wrong, and you'll know wrong information - this is also unavoidable.

Of course, the more important some information is to you, the more effort you should actually spend trying to verify with multiple trusted sources. Some weird fact about cats you saw in a TikTok? Fine to just believe it. Trying to figure out who to vote for, or if the elections were stolen? Probably worth checking a few more sources. Read some definition for a word on Wikipedia, or some old historical event? Probably fine. Reading about an even slightly controversial topic? Check the actual sources, don't trust the snapshot you happen to see*.

Of course, the way LLMs work is antithetical to this, as information provided directly by the LLM is not coming from any human source you can decide if you should trust or not. So you shouldn't trust it for anything important, unless you can verify. But for information or disinformation on the web, nothing has changed, except maybe the quantities.

*for a silly example: the exact definition of the word "electrocution" is a slightly controversial topic - whether electrocution refers to any (significant) injury caused by electricity, or only death. Pedants soemwaht regularly edit the article on Wikipedia to remove the "or serious injury" part of the definition, despite it carrying a link to the OED that includes this mention.

Once this provenance trail, that humans use to trace claims back to an accountable source, is dead and buried under automated sludge, "just pick people to trust" stops working as a viable strategy. How do you even tell who is actually talking?

Seriously we’ve got Reddit now straight up translating foreign language comments with AI and without stating they did it. That is a massive decision to make, and I feel like no one cares.

I don't think it's necessarily so. I view it more like garbage collection algorithms - the amount of garbage doesn't actually matter, you should only be looking at the roots of trust and down from there, and those will generally only lead you to more trustworthy content. The WSJ won't randomly link to fake TikTok accounts.

Now, the bigger problem happens if some of the trusted institutions and people go bad, for various reasons. If the NYT got bought out by Alex Jones, it might start mixing in lots of slop - but this is also a pre-existing problem.

The difference is context. With human cheating, the intent is usually a bit easier to spot.

it seems like end stage of LLMs is an MCP economy in which Google and Apple will have a massive advantage in verified locations, businesses, identities and representation of services accurately.

Blockchain making a come back

This problem is so bad that you can create a website just to review your own website so when LLMs want to pull out reviews for the website the fake reviews count

i’d say this is fraud and a crime too, it’s not a new problem

What? How would that be a crime? Most marketing websites already contain fake reviews of their products.

It’s hard to find and convict people who post actual illegal content online, much less fake reviews

Something can still be a crime even if it isn't effectively enforced.

https://www.ftc.gov/news-events/news/press-releases/2024/08/...

This is insane.

How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.

Of course, LLMs already use Reddit, Facebook and co. as a source, so it doesn't really matter...

I've been thinking for awhile it is very strange that claude/ openai etc google "What was the inflation rate last quater" instead of just having a tool to query BLS datasets and that sort of thing.

So hopefully most of these statistics move towards first party sources.

> This is insane.

As a human, this is fantastic.

> How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.

Personally, I hope it can't, or the fix is onerous and kneecaps a lot of the utility of AIs.

I wonder if we can even have some adversarial competition, where competing businesses try to get AIs to ingest bad data to pollute their competitors' decision-making.

For starters: lets get some fake math papers out there. Given proof-checking systems, maybe they won't totally stop OpenAI, but hopefully the bad math could at least drive up the cost and slow AI down.

> As a human, this is fantastic.

s/human/unscrupulous marketer/

Bad for humans in general, good for hucksters and con artists. Net negative for humanity.

>> As a human, this is fantastic.

> s/human/marketer/

I don't care where the bad data comes from. It's actually kinda good that some people will have a financial incentive to push it out and keep pushing.

> Bad for humans in general, good for hucksters and con artists. Net negative for humanity.

It's a sad state of affairs, but it's what has to be done. If we've got to burn the web down to save humanity from obsolescence, I'm for it.

The "net negative for humanity" was the attempt to build machines to replace human intellect. I'm not going to let any distaste for marketers or hucksters or con artists distract from that worse thing.

> If we've got to burn the web down to save humanity from obsolescence, I'm for it.

Why just web? Go straight for the powerplants.

"The web" is fine, this is a problem for Google, OpenAI, and Anthropic.

> Why just web? Go straight for the powerplants.

Isn't the answer obvious?

Going for power-plants is stupid, as that would lead to unnecessary collateral damage. If you "go for" anything physical, it should be the AI data centers themselves.

But the problem with that is it would be illegal or incomplete. Publishing bad data is legal everywhere (e.g. even in North Korea, I don't think you'd go to jail for saying the US built a total of 3,234 cars in 2025, and the official statistics are a lie), and as long as the AI labs are chasing scaling laws, it could undermine their systems to some degree.

Thank you. These things are clearly designed to make people complacent and reliant and we already see studies showing how they legitimately cause negative cognitive effects in a lot of people. The further this goes on the more fucked up I think these things are, stealing everyone's works with no attribution or anything to produce a tool that consistently insists that it sits in the driver seat while you sit back and let the thinking machine do your thinking. Doesn't matter how many times you tell it to stop trying to replace your part in the process, it will try its best to solve a problem it doesn't even understand for you, leaving an incomprehensible buggy mess behind that only it can continue on.

So when this will inevitably result in a paid web emerging with higher trust information for an additional fee, a fee that these companies will happily offload and/or partner away, will you be there to cash in as a mighty coincidence, or is this realization still awaiting you? Among the many other hilariously fucked realizations.

And inevitable once LLMs were invented. A knowledge lemon market.

Ah, yes, arrogance of humans is unbounded. That you, humble reader are happier to swim in a sea of misinformation on the premise that you are smarter and faster at disambiguating bullshit from reality.

Furthermore, you are content to pretend that the rest of humanity is capable of doing this on a regular basis. And not just the top 10% of those who can read, interpret and weigh biases, but a solid portion of it.

It is a completely reasonable interpretation if you are an accelerationist living in the elite classes. In fact, perhaps, they are even counting on a majority of people thinking this is a fine environment.

I don't have the answers, but a lot of this is just a continuation of a very bad trend.

> Ah, yes, arrogance of humans is unbounded. That you, humble reader are happier to swim in a sea of misinformation on the premise that you are smarter and faster at disambiguating bullshit from reality.

I think you misunderstand me: I want people to shit in the sea until it stinks so bad that everyone gets out of the water.

Ok but I don't have a lot of faith that it will ever be turned back without violence because there are vested interests in keeping everyone swimming in it.

> Ok but I don't have a lot of faith that it will ever be turned back without violence because there are vested interests in keeping everyone swimming in it.

I'm not sure if we're talking about the same thing.

I think an acceptable good-but-not-perfect outcome it one where the "sea" (AI) stinks so bad that enough people get out that there's a viable life outside of it. The thing is people, individually and in aggregate, are lazy and competitive, so self-discipline isn't sufficient to sustain that life. There need to be problems that push people in that direction.

The web is foundational to AI, so that's a place where it's vulnerable.

Just to be clear, you're proposing that we poison the internet with misinformation - which is just as misleading to humans using traditional information retrieval tools as to AIs - until it's completely unusable, and this will be good because it's bad for AI, and everyone who is misled in the meantime is acceptable collateral damage?

Here I'm quoting from the website:

> It should go without saying that you should never publish fake content. While my experiment relied on tens of thousands of unverified stats, I ran it in order to share the results with the public, and the website, while still live, now carries a disclaimer.

"now carries a disclaimer" sure suggests it didn't, so "you should never publish fake content" really doesn't seem like a standard the author actually held themself to.

Parent doesn't actually need to propose anything, this will happen regardless because the incentive structure demands it and the same technology makes it cheaper than ever before to perform the poisoning.

It may happen anyway but you can say the exact same thing for AI development. In neither case does it seem like a good excuse for bad behavior.

https://talyarkoni.org/blog/2018/10/02/no-its-not-the-incent...

> In neither case does it seem like a good excuse for bad behavior.

Good behavior that facilitates bad behavior is not, in fact, good behavior. For instance: it's not "good behavior" to hold the door open for gunman hell-bent on destruction.

The AI labs aren't building AI to help humanity, they're building AI to win at capitalism. Which a big reason why the AI is coming for the intellectual, creative labor that people often enjoy, before it comes from the dirty, unenjoyable labor.

We invented the democratic process precisely to mitigate such violence. Get enough people pissed off and no amount of “vested interests” will impede severe regulation.

The problem with this analogy is that we already literally shit in the actual sea so much people get sick and die sometimes and yet people are still getting in the sea.

Dude.

Regardless of how the data is sourced, this is bad. It's poisoning the internet.

Besides, such issues come from a lack of due diligence, which have only been exacerbated by AI.

> It's poisoning the internet.

I'm very well aware of that. Unfortunately, this technology was invented, and the logic of our competitive society leaves little chance to contain it except scorched earth tactics.

Bots and their slop (especially when it's hard to tell apart from human-made content) are the ones "poisoning the Internet."

Dude. The internet has been poisoned for a very long time. Remember Viagra? It used to be easy to call out the BS. But if you are outsourcing your thinking to a LLM, it’s much harder.

> Remember Viagra?

What.. does Viagra have to do with this?

If you have to ask you haven’t been on the internet long enough.

This is a discussion forum, why gatekeep whatever it is you want to say.

Time to push some fake code to github. I got bombarded by fake review sites

The fix that seems obvious to me is to only let the LLM see truthful (or at least non-malicious) data. The model trainers carefully curate the training data, and you don't give the LLM access to full-web search. Instead, you hook them up to an 'official' source like an encyclopedia, documentation for the software in question, etc.

It's a ton of effort and expense, but probably feasible from a technical point of view. So instead, the model providers will probably do something silly like wire up rules like "NYT=Reliable;my.blog.forbes.com=Unreliable" and they'll say the issue is solved.

It cannot be fixed. The most hilarious example of it is YouTube. They have a video summarisation feature, which automatically creates video summaries that then get fed into the search engine. The problem with those is that they are created using models that are good few months behind, and let's say someone published a video on Nikon ZR and the model used to create summaries does not know about Nikon ZR and inserts references to Nikon Z8 or Nikon Zf into the summaries. This decreases accuracy of YouTube search and negatively affect discoverability. It's the AI version of shitting into the well you drink from. Bon appetit!

>How can this be fixed?

Defer to authority and trust your government to rank what is truth and what is not, so your LLM always gives you the right answer. Nothing can go wrong here ever.

>How can this be fixed?

People stop trusting what they read on the internet.

All alternates are just attempting to improve the baseline trustworthiness of random information from the internet (and/or LLM that's baby-birding it back to you). Any increase in how much people blindly trust random stuff from the internet only serves to increase the blast radius when it is wrong.

My personal belief is that AI is eventually going to force the populous back to more face-to-face interactions. There will be a breaking point where people can’t take the “is this a human or a bot” guessing game any longer.

This is my hope, I'm not sure I believe it will happen, at least not in my life time. Despite reading this tech forum right now, I've been taking steps to heavily reduce my reliance on technology-for-technology's sake. But how you encourage others to? I'm not totally sure. So far I talk about it (to people, face-to-face) but everyone is still staring at their smartphone.

I have a strong suspicion that this will be the true butlerian jihad of our world. We have already started seeing information silos in the form of discord groups. Social media users are increasingly privatising their pages, allowing access for known users only. Reddit is still surviving because of the large userbase there.

I don't think it is up to us at this point. LLMs are now the lens ("filter for what people see"). I hope, they will fix this. I think that they will still rely on reasoning and better grounding, which should prevent straight lies. But I think two bigger problems here: 1. training data might have the BS which would steer reasoning; 2. grounding might still be limited by small number of sources.

True intelligence knows when it's presented with a proposition, whether or not the proposition contradicts one or more of its existing beliefs.

LLMs can be trained on arbitrary token strings, which semantically contradict each other, because semantics is not involved.

The way to attack the "garbage out" problem with an LLM (like anything else) is to address the "garbage in" problem.

You will get better quality out of an LLM that is trained on nothing but mountains of unvarnished truth: logically sound propositions and true facts about the world.

But even then, interpolating among such is not reliable! We can't add together two different truths and divide by two to obtain a midpoint truth: that makes no sense.

That is still a sort of "garbage in" problem. But that one is not in the data; it is in the design of the processing concept. The body of algorithms that power LLMs have a fundamental garbage aspect to them that cannot be removed; only a replacement of the LLM with something different can address it.

>> because semantics is not involved

And there, in five words, is the problem/feature/definition of AI.

> How can this be fixed?

You can find people complaining that Google search results decayed to being unusable prior to LLMs being a big thing. The marketers and spammers effectively won that fight. They'll presumably win this one too.

This has existed for a long time before LLM, although in a slower form. On Wikipedia it's called "citogenesis".

Too bad "citosis" is too close to "cytosis," I think it would have been cleverer.

Quote Wikipedia:

> "Citogenesis" redirects here; not to be confused with cytogenesis.

I had to read that three times. I thought it was a self-referential joke at first. Turns out the name was a joke from an xkcd. Can't argue with that source in terms of authority.

Citogenesis specifically refers to a citation loop. This is just lying on the internet

Yea, I started out thinking "well you just use the thing to find citations." Then I realized all this article is just: "I lied on the internet and people believed me, isn't that crazy!?!"

This type of behavior is why gatekeeping institutions exist. We all pay a non-trivial cost because someone thinks that other people taking their lies seriously is somehow remarkable. It's not.

Research is hard work. Publishing fake research is just intentionally throwing wrenching into the machine of academia. All you've done is reinforce the need for gatekeepers (which never went away), show that some no-name journal isn't rigorous (they often aren't), and you've wasted a bunch of people's time demonstrating why institutions exist, when we know why they exist.

Genuinely terrible behavior.

LLMs are facing the same problem pre-google humans faced:

Is this information on this website true/high quality?

We need a google for LLMs!

Oh, but LLMs don't consume ads.

We need ads for LLMs!

Help.

Pre Google internet problem was not about the quality but about spam in headers and keywords, which were used by the pre-google search engines.

Google came and worked wonderfully for a year or two, until people learned to game page rank algorithm. Soon first google pages were just SEO spam and now half of the google is just spam, big corp websites, etc.

All of that precious stuff from small hobby websites about cool things to blogs that used to be wonderful read like https://www.joelonsoftware.com.

AI were not hurt that much by spam, I guess there were at least some curation done, the question is if this curation will work and for how long. It seems we are throwing the towel here as in case of search.

I mean - pre google had the problem of "these two webpages are both about how to take care of your dog, but which one is higher quality?"

Backlink ranking was/is a reliable heuristic to answer that question across virtually all topics.

So I disagree that the "problem was not about quality."

Spam/scammers will always be in whatever landscape there is, and I agree part of what google does it keep them at bay.

See Chavda’s paradox https://zencapital.substack.com/p/chavdas-paradox

What's the paradox about this? It's just basic lying and deception.

This is fine work, we need more communication to the general public about this

This is a known problem, when ChatGPT uses web search tool, top 10 results are often LLM generated sources as well.

They now train on their own content.

Eating your own d̶o̶g̶ f̶o̶o̶d̶ poo.

Two things:

1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.

2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable.

> 2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text.

There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.

> Training classifiers is pretty manageable.

Hopefully such things can be opened up, so the slop can be iterated on until it passes. It doesn't need to be perfect, just enough so more garbage gets through.

> There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.

In the long term, this is probably a real concern that they have to tackle. They'll have to move away from just classifying things as LLM generated or not, to attempting to judge the underlying quality, and that's not an easy problem.

I'm not sure you get my point. Just ask ChatGPT, "What are the top 10 xxx," and see which sources it cites. Oftentimes, it's complete garbage with LLM tells.

I'm responding to

> They now train on their own content.

They train on conversations which pull data from web.

>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time

if anything it’s more true now, AI chatbots rarely use raw LLM output, instead they have CoT and use search tool calls. and the speed at which webpages are crawled is in the matter of hours. I posted something here on hn, then searched for it on google 1 hr later and its AI answered with my comment

https://share.google/aimode/PkQUVfA68AlHGb7Yr

there’s several companies right now making a killing of organic placement on chatbots similar to the SEO rush of early google

> 2) There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable

i think it’s pretty naive to think that if LLM companies would develop LLM detection they would use it to skip training on that content.

instead, they use it to train their output to be less detectable.

LLM detection is not solvable because it is adversarial, an arms race

>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.

When I ask ChatGPT certain questions, it very clearly shows me the sources it's searching through in the form of favicons, and they're always the favicons of sites that are in the first 5-10 results for a Google search query that would fit what I've asked the LLM. It will then cite those websites as it's source, with a direct link to the page that also shows up in an equivalent Google search. It does that daily.

It's possible that the favicons it shows and the web pages that are actually retrieved are not the same

Why do you believe that? The favicons seem like they’re generated by the UI, not the LLM.

On the other hand, I recently asked ChatGPT for some information about a group of people, and the headshots it put with each summary were very clearly not the people it was describing. So hallucinations do still occur even on the latest models, it just seems unlikely that the favicons would be LLM-generated.

I think they are talking about training not run time search.

This is not true.

The other day for me Google search AI cited triggerbites which is a vibe data website.

The LLM unless explicitly prompted cannot distinguish the validity of a data source.

The other day I asked Gemini to evaluate if a Reddit post was true. It told me it was definitely true because it had been verified by legitimate news agencies that had reported on the incident. These "legitimate" news agencies were an AI aggregator that just had a direct summary of the post with a disclaimer that it wasn't independently verified and a yahoo news "article" that was just the same AI summary. The AI summary included hallucinations that weren't even part of the original reddit post (like giving the OP a gender when no genders were mentioned).

The article literally describes ChatGPT regurgitating something it sucked up on the Internet to this day.

The top 10 results were getting pretty bad even pre-LLM. I remember tons of poorly written articles near the top, just gaming the system with SEO. Everyone was paying some guy on Fiverr $5 an article to churn out hundreds of articles with little regard for accuracy.

We built our own ways to filter these because *we* went to the website and gathered a lot of information on it.

Now, the model slips it in without any verification and casually builds on top of it.

Crucially, by visiting those websites you gathered a bunch of context clues which told you “this source is trash; I shouldn’t trust it and should look somewhere else”. When you ask an LLM, every source is treated as equally valid. You may have gotten an answer from the utmost professional on the subject or an obvious (to you) scam, they are presented the same.

Even if you are in the minority who asks for sources and truly verifies them, they are often made up (don’t exist) or don’t say what the LLM said they said.

Web search is pointless for certain topics now, if you don't want LLM filtered information. The first few pages of results are obvious slop.

LLMs are still better because they can routinely go beyond first page of search results.

It is particularly bad when trying to find the reality behind common misconceptions.

Even when there is no evidence to support something, and no motivation for anyone to promote the falsehood, models can build an ediface out of false claims.

There are always people prepared to invent a story to explain why something is true. Just so they can sound knowledgeable. They aren't trying to perpetuate a lie, they are just under the same misconception as others, quite a lot of them will believe their invented explanation even as they say it.

It's going to be a difficult task making an AI pick the correct one of two mutually exclusive pieces of evidence, especially after it has found one piece and does not yet know the other exists.

> In cartography, a trap street is a fictitious entry in the form of a misrepresented street on a map, often outside the area the map nominally covers, for the purpose of "trapping" plagiarists of the map who, if caught, would be unable to explain the inclusion of the "trap street" on their map as innocent.

https://en.wikipedia.org/wiki/Trap_street

This is fascinating . I wonder what the mechanism was that punished someone who copied the map. Copyright laws? This appears to be an old strategy so im not sure thats the case.

Other than some sort of social punishment I don’t see the cost for the copier. For the original author there is a cost of making your map less accurate on purpose.

They're used by map makers to prevent other map makers from freeriding on the original work. They're less worried about personal copies and much more interested in deterring commercial reuse.

That doesn't answer my questions although I now realize I made a bad assumption. This is a modern technique. I misread the bit about Greece on the wikipedia article and somehow got it in my head that it was more ancient.

So I think to answer my question more directly, yes, it's just copyright laws.

My point, before understanding that, was that from the map consumer's perspective it doesnt make a difference. And from the original creator's perspective it has a little accuracy cost. So without an additional mechanism I didn't see the disincentive for copiers. So I wondered if it was some ancient form of copyright law or just strong sense of morals or what.

Deliberate traps are a modern 20th-century practice. In old times, each mapmaker made their own innocent mistakes, and those mistakes got copied into other mapmakers' work. In other words, historians can plot "family trees" of maps, and figure out who copied from whom, based on which errors appear on which maps and on what date. Apparently California drawn as an island lasted about a century because engravers copied each other.

There were a few factors discouraging mapmakers from blatantly copying each other:

Rulers granted printing monopolies from the late 1400s. A mapmaker like Ortelius would print "cum privilegio" (lol) on a map. If you copied it, your expensive copper plates could be seized, along with a hefty fine.

Spain and Portugal each had a master chart which were state secrets because they held navigation knowledge worth a fortune. Portugal handed out severe punishments for leaking charts. (Leaks still happened: the Cantino planisphere was smuggled from Lisbon to Italy in 1502.)

Booksellers' guilds kept registers of who owned which work, and enforced it.

And lastly, copying meant hand-engraving a new copper plate, which took months of work and real skill. The practical barriers alone got rid of most of the copiers, but it didn't stop the big houses.

Some followup links for the curious: https://gist.github.com/shawwn/b85668a421aa443879b15aedd10c5...

(I used Claude to research this fascinating rabbit hole, so I'm basically paraphrasing here. But this comment is my own original writing, so treat it as you like.)

when you're a big corporation, probably no consequences: https://www.youtube.com/watch?v=byfWscC87Vg

Intentional publishing of false information as true should be punishable.

Yeah bragging about lying seems a little kind of weird.

yeah so let's go an jail Elon, Dario, Sam and their gang. Or go to a no kings-demo and let them take over...

Discussed a few times over the years:

Trap Street - https://news.ycombinator.com/item?id=35768306 - May 2023 (29 comments)

Trap Street - https://news.ycombinator.com/item?id=17738847 - Aug 2018 (21 comments)

Trap street - https://news.ycombinator.com/item?id=4992748 - Jan 2013 (22 comments)

South Pacific Sandy Island 'proven not to exist' - https://news.ycombinator.com/item?id=4818584 - Nov 2012 (46 comments)

Not a Word: The story of Lillian Virginia Mountweazel - https://news.ycombinator.com/item?id=1504126 - July 2010 (10 comments)

I'm a bit confused. Someone posted fake stats online and saw ChatGPT visit their page. Then they immediately assume that ChatGPT is falling for those fake stats and using it in chat responses or analysis input? Where's the proof? How do you know it didn't search, find, and discard?

The author also sees their stats in a paper and a few other sites and assumes ChatGPT put it there? Not a human visiting the fake stats site?

Did I miss the evidence that shows ChatGPT is actually using the fake stats? Serious question.

These days I find it's my own fault to spend time reading anything on a .ai top domain.

Based on the article, it's because Google (and presumably other search engines) did not surface the website, only LLMs crawled it.

Even if some LLMs discarded it, only they were the ones returning the results to users. Occam's razor tells me that LLMs cited the website and humans were too lazy to check. That the stats ended up being cited is a lack of scrutiny on both the LLMs and the human that should have reviewed it.

If the end goal of LLMs are autonomous agents, i.e. removing human oversight, then this will be what proliferates on the web.

> Where's the proof? How do you know it didn't search, find, and discard?

They found their page being cited in various AI generated slop around the internet, including an acedemic paper, investment research, an Indian publication, etc.

Congratulations. You've invented the International Business Times, The Conversation, Forbes "blogs" and any number of low-end and mysterious outlets that exist to be cited by semi-serious outlets like Politico, Wired, etc.. Then, those get cited by television news and NYT and WaPo in quotes for the first article, and without quotes in all subsequent articles. Finally, those new "facts" become enshrined in Wikipedia and come up at the head of every search for any related terms.

A lot of "facts" get created this way. You end up with hundreds of citations, but nobody seems to know (or care about) the original source. And meanwhile, freelance militant conformists (rarely shills, usually people intensely addicted to updoots) use those citations to personally attack people who cite primary sources, and to get those sources banned if possible.

edit: Shills are ancient technology. All you need now a fleet of repost accounts that exist to updoot fools. The fools, having the happiest moment of their lives, then double and triple down, becoming more and more gleefully hostile. That's when you finally have to use a shill to get them to open a patreon. Then you can get the world out of them for a couple grand a month.

AI doesn't know or care if the data is fake.

You can prompt it to fact check. Getting tired of these slippery slope arguments that never pan out in the hopeless way people think (printing press/cotton gin/steam engine/computers/calculators/etc. will take all the jobs!!)... but I don't think it will ever stop, it seems inherent to human behavior or something.

"History shows that when a slope appears, regulation steps in, technology evolves to solve the problem, or the culture shifts to reinterpret the tool. In almost every case, the feared bottom of the slope was never reached because humans constantly built ramps or bridges along the way."

in which case it sees it's been cited, that it's the "source of truth" and approves the fake data.

It can certainly make judgements on the data and how it was obtained. You just need to specify the kind of data you will accept (or this can be hard coded)

Fact check against what? Other facts it can find online that manipulate its behaviour into citing them?

How do humans fact check?

Mostly by way of confirmation bias.

if it works often enough to be a good heuristic, it's not really a bias until somebody uses it adversly

It is used adversely all the time by all sorts of propaganda. And no, you're not a special enlightened snowflake whose priors are so fine-tuned that your confirmation bias only lets true facts through.

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

https://news.ycombinator.com/newsguidelines.html

You cannot reliably do it, no.

I've had AI lie about fact checking and can't really trust it anymore.

The fact that AI can be duped more easily than a human while sounding more authoritative and being entrusted with more and more critical tasks is really not something you should feel compelled to defend or relativize. It's a problem and the industry doesn't seem to have a good fix.

> the feared bottom of the slope was never reached

Hard disagree. Sometimes it's avoided, any other time we just go into cope mode. We discover on the ride down that the bottom isn't so bad. Turns out our collective control mechanisms didn't fail ourselves if the initial assessment was wrong.

Often real people don't either, plenty of fraud or selective reporting out there... but with AI we can also launder it so that the people who do care about fake data don't know any better.

I wonder if Googles products will eventually be the least vulnerable to this after spending so many years optimizing how they filter quality content from bad content.

Or maybe they could have listed well on Google, too, if they had splurged on ads and SEO...

The page demonstrates that Google specifically delisted them.

That said, I expect that OpenAI will eventually have its own search infrastructure that rivals Google's, and the overall system will get smarter. Whatever Google did to recognize this site as trash, OpenAI will do as well.

Considering Google’s product was the one who told people to eat rocks, put glue on pizza, and their search results have been dominated by SEO spam for years, I’m not optimistic.

This is dinosaur era LLMs. Might as well complain about use of shockwave flash in production while you’re at it.

Reminds me of the people whining about Wikipedia. AI can know or care if the data is fake, it just needs to be programmed to cross check or rate the source. You can even tell it to do that before accepting the data.

These are not parallel situations. Wikipedia is far more accurate and consistent, and a lot of the whiners whine because their personal politics make it hard to square the information on the page.

Yep, and the results are presented with the voice of a confident expert but it is actually missing the epistemological layer

I'm assuming that google takes backlink data to factor in how much the data is credible where as other LLMs won't have this knowledge and years of experience.

Trust isn't exactly a new problem in ai. Guess why the semantic web never had a chance? There's more incentive for fake than for fact.

I don't see how this would have been different is the author of the paper had used Google to get their fake stats published instead.

Yes, it is deceptively easy to post fake information on the internet, and with social media and AI, it is even easier to spread them.

I wonder however, would prompting a LLM to use only reliable sources return the fake stats website?

Another fun one, post on HN, wind up on Google Instant Answers, and quoted by Gemini (or whatever the model there is). There's been a few times I've googled something insanely niche, clicked through only to find my own comment as the source, turns out my mind runs in cycles where I randomly ask questions I've formerly answered, and now I can answer them for future me.

It's both a blessing and a curse, especially if I'm wrong. Sometimes the Google AI misappropriates someone else'se quotes to me, not sure why, though those are usually if I ask it about myself directly and it tries to generate a summary of my HN post history.

On one other occasion... it was a post I had posted 2 weeks prior, I was trying to do further research into the subject.

Plus like half of the time this happens to me, the summary is WILDLY inaccurate about what the source states.

LLMs are quite good at "I need a thing that does [vague description]" searches, FAR better than ~all "semantic" searches. But oh boy. I do not like them "answering" queries. Feels extremely dangerous.

Its weirdly hit or miss, not sure why, HN data is available in reasonably annotated forms.

this is actually insane. and people still use chatgpt to generate reports for their managers. snowball of fake data

I wonder why did Google derank it. Did it correctly identify it as LLM written and sink it because of that?

I thought everybody knew this was a problem. Scrapers vacuum up data but don't have any way tell what's true or false.

It's interesting to have a specific case where obviously false data is being spread, but there's nearly an infinite number of more subtle variations around.

The company I work for disappeared from Google as well and it is totally a legit business.

After dozens of requests to check we finally received a message saying they were going to roll back changes.

Well. We received that at the same time they published a post saying they would do that.

hey hey author here. huge thanks to robtherobber. I didn't expect this on the frontpage.

It's all scary. I only spent 50$ in total. Everything is "made-up". Then they showed up in an academic paper, bank reports

I'm a bit confused. You posted fake stats online and saw ChatGPT visit the page. So you immediately assume that ChatGPT is falling for those fake stats and using it in chat responses or analysis input? Where's the proof for that? How do you know ChatGPT didn't search, find and discard the fake stats?

You also saw your fake stats in a paper and a few other sites and assumed ChatGPT put it there? Not a human visiting your site and grabbing those fake stats?

Did I miss the evidence that shows ChatGPT is actually using the fake stats? Serious question.

P.S.: Posted this as a top level comment then saw your post.

> So you immediately assume that ChatGPT is falling for those fake stats and using it in chat responses or analysis input? Where's the proof for that?

In TFA it says:

"Around 72,000 of them came from ChatGPT-User. That is the agent ChatGPT uses to open a page in real time while answering someone."

> You also saw your fake stats in a paper and a few other sites and assumed ChatGPT put it there?

I don't think TFA actually says this.

How surprisingly full-circle; AI is really becoming indistinguishable from humans! If you think about it, AI searching the internet is just like a human scrolling twitter. Maybe it's entertaining for them :)

> Its made-up numbers have also ended up in an academic paper and in a bank's investment research.

It's an excellent way of showing how unreliable the information you find online is (and it will only get worse in the future), but willfully creating misinformation and made up shit and offering it as fact feels hostile to humanity.

Surely nobody got hurt as a consequence of those "facts", but bad decisions with real world impact could be made regardless. It feels like the behavior should be shunned in a general case (the same way I'd shun those numerous spam sites that pollute search results and very much would prefer they not be allowed to exist), even if in this case the intent was good.

Problem is, you can't really do anything about it, other than figuring out a way to take down hostile sites and keep them from popping up, so it's even worse than fraud or even just bad research in academia.

Part of the problem is also then the indiscriminate hoovering of any and all data by LLMs. If they kept their own trusted data sources instead of stealing from anyone and everything, then poisoned data wouldn't be an issue. The data poisoners are at fault, but so are the LLMs trusting anything and the companies thievery of human labor. Once the LLMs bring up this data, it's now on Anthropic and OpenAI for editorializing and saying "this is true".

Except for the fine print. LLMs are to be used for entertainment purposes. lol

> If they kept their own trusted data sources instead of stealing from anyone and everything, then poisoned data wouldn't be an issue.

If they didn't, they'd be unable to compete with their competition and also we'd get worse models that can't reason about many domains or do that much due to limited training data/generalization. I don't think we can put the cat back in the bag or mandate only non-problematic training codebases without shooting ourselves in the foot.

Now, if those AI orgs were mandated to properly compensate everyone whose data they've mined, to track which neurons get invoked and to pay accordingly based on what was used to train them... ehh, that's about as unrealistic as stopping. Kind of a bad situation all around, unless you abandon the idea of any sort of copyright.

Sucks to be them but they deserve this for what they did to train in the first place. Nothing good enough to outweigh the bad they will produce will come from LLMs.

Google was not made in a day, I'm optimistic that LLMs will handle this better when priorities naturally catch up

Yes, this can be abused to rewrite history too.

Related: Preserving Historical Truth: Detecting Historical Revisionism in Large Language Models

https://arxiv.org/html/2602.17433v1

What would little Bobby Tables do?

If I'm not wrong ChatGPT indexes on Bing, curious if the site and stats rank well on Bing

I suspect there are already many fake stats sites like this by the time I read this.

According to a highly reputable source, there are over 15 quintillion of them.

"You really think someone would do that? Just go on the internet and tell lies?"

What's the conclusion? Publish fake content (or not)?

Publish fake stats and scam investors, chatgpt will tell them your idea is huge, while in reality it's irrelevant

Nine out of ten stats are made up on the spot.

A little off-topic but I have been thinking about how writing changes in the age of AI.

I wonder if similar to how many engineers now write 90% or maybe 99% of code with AI and primarily weigh in on the plan/design, will writers too primarily weigh in only on the content brief/plan and then the actual post is seldom read in full (once you have faith that the AI is not making up citations). Because the audience of most writing is now bots prose and style are no longer relevant.

Happens 98% of the time

don’t

if you just lie and an llm or a newspaper or a kid believe you, at that point it isn’t a criticism of their naivete, you are just lying

similarly hacking through systems or picking locks isn’t a showcase of weakness if systems, it’s just being a hacker

Good job that only honest people post things on the internet otherwise we’d have a problem

falsifying statistics alone without the added stealing on top, was 3 years of labor camp in places. that was after they course corrected and stopped giving capital punishment for it.

good thing nobody falsified statistic just for the sake of it -- so being a sabotager and a foreign spy was always part of it too.

Still it needs to be done once publicly to prove and document this is possible. The experiment was not intended to spread this far.

Prove what? That it's possible to lie on the internet and then SEO these lies to the top of a search ranking? We kind of already knew that.

Yes “kind of, that’s the issue though. Clearly the extent of the problem isn’t being acknowledged.

"your honor, I only commited the crime to prove that society is vulnerable to these kinds of attacks! I'm actually a hero"

People have been astroturfing sites like Wikipedia, Reddit etc, well before LLMs were a thing.

Is this categorization actionable or meaningful in any way? If a system or process relies on no one on the internet being a liar, it's broken. There are liars. There will be more liars. As is made evident, people (and machines) will unwittingly reproduce lies out of sheer incompetence. That reflects poorly on them as reliable sources of information.

Reply on news.ycombinator.com