Comment by michaelt | Hacker News Reader

Comment by michaelt | Hacker News Reader

As far as I can tell, what they're proposing is:

Today, for each output token the LLM produces a probability for each possible output, then a 'sampler' makes a probability-weighted random choice. If the next-token probabilities are 90% for foo, 9% for bar and 1% for baz then the sampler chooses a random number between 0 and 1, if it's <0.9 it outputs foo, 0.9-0.99 it outputs bar, 0.99-1 it outputs baz.

But what if instead of using random numbers, you had a source of evenly distributed random numbers that was deterministic, based on some secret key?

Each candidate token would remain just as likely as it was before - there would still be a 90% chance of foo being chosen. So the output shouldn't degrade in quality.

And sure, some tokens will have 99.999% probability and their selection doesn't tell you much. But in most real-world use multiple wordings are possible and so on. So across a large enough sample of the output, you could detect whether the sampler was following your secret deterministic pattern.

Of course the downside is you've got to check on exactly the same LLM, and only people with the secret key can perform the check. And it's only applicable to closed-source LLMs.

I'm also not quite sure if it works when you don't know the exact prompt - so maybe my understanding of the paper is all wrong?

My understanding from the "Watermark Detection" section is that it only requires the key and the output text in order to do the detection. In particular, it seems like the random seed used for each token is only based off the previous 4 tokens and the LLM specific key, so for any output larger than 4 tokens, you can start to get a signal.

I don't think the key actually needs to be secret, as it's not trying to cryptographicly secure anything. So all closed weights LLM providers could just publicly share the keys they use for water marking, and then anybody could use them to check if a particular piece of text was generated by a particular LLM.

That being said, I think you are right about this only really being useful for closed weights models. If you have the weights, you can just run an LLM through a standard sampler and it won't be watermarked.

Why would anyone ever use such a model? And then, given the significant reduction in users, why would any closed model service do this?

Seems like a cool theoretical trick that has little practical implication.

There are situations where the model output being watermarked doesn't matter. For instance, I hear people on HN asking LLMs to explain things to them all the time, (which I think is a bad idea, but YMMV) and people use LLMs to write code quickly. (which I think is at least possibly a good idea) There are also some content farms which churn out low quality books on Amazon on various topics, and I don't think they care if they get caught using LLM outputs.

Thus it might reduce usage some, but it certainly wouldn't block all usage. Additionally, there are only a few providers of truly massive LLMs on the market right now. If they decided that doing this would be a social good, or more likely that it would bring bad PR to not do this when their competitors do, then they would at least be able to watermark all of the massive LLM outputs.

You say that as though there isn't a choice though. There will always be a good offering who doesn't water mark.

And there is no good reason for a provider to watermark - they aren't helping the customer. They'd be helping some other party who isn't paying them.

This will never be a thing.

> There will always be a good offering who doesn't water mark

There's a possible future where this gets legislated, right? Of course, there are lots of implementation challenges to this and it's probably a bad idea...

Sure, it's possible. I'd lay 100-1 against though.

>There will always be a good offering who doesn't water mark.

I wouldn't bet on that! I can see legislation to require this for many reasons ... related to intellectual property, cheating, detecting the root of hate-speech or harassment, "stealing" from employers by not performing work or putting them at legal risk, "stealing" from artists by duplicating their style, political speech that can not be traced (it could be from a bad actor!), tracking down generated revenge porn (or much worse!), tracking down people using LLMs to grift the elderly, and on and on. Why, if you are not using a watermarked LLM, it could be an op by Russia, China, or Iran! In fact, part of the legislation could be a requirement of social media or office tools or government tools or political tools or educational tools to check for a watermark and not work if an approved one is not found. Ideally this list will be private, because you want companies to be able to automate away workers, and do the least possible for customers, you just want to make sure you're doing it above-board, you see.

>And there is no good reason for a provider to watermark - they aren't helping the customer.

No one cares about customers, they care about money. And you know what helps make a lot of money? A legally defined moat for yourself and a couple of others that blocks anyone else.

>They'd be helping some other party who isn't paying them.

Yes! That party is themselves!

It seems extremely unlikely in the US that such legislation could come about.

Even then, open source will almost certainly always exist. Services running offshore will exist. It would seem impossible to enforce.

I'm totally happy having huge amounts of my use of llms identifiable as from an llm. I don't see many important cases for me where I need to pretend it wasn't from an llm.

I will happily lose those cases for increased performance, that's the thing I care about.

Are there normal cases where you picture this as an issue?

Not a problem for me. I am not a student anymore.

And I am not against LLM output being identifiable as such. (although I think an argument could be made based on the ruling about the monkey and the camera, which IIRC would say that the copyright belongs to whoever created the situation).

But after the

1. British Post Office scandal and

2. some really high profile cases of education institutions here in Norway abusing plagiarism detectors

I do not feel ready to trust neither

1. complex software (and especially not closed sourced software) to tell us who is cheating or not

2. nor any humans ability to use such a system in a sensible way

While cheating isn't usually criminal court, students also usually does not get a free defense.

For this reason I suggest cheating should have to be proven to have occurred, not "suggested to probably have occurred" by the same people who creates the not very reliable and extremely hard-to-reproduce LLMs.

Increased performance? Watermarking will not increase performance. They are talking about tilting the decoding process in minor ways. It won't help (or hurt much) performance.

Increased relevant to other providers of different llms. So I'd pick watermarked X over non-watermarked y if x performs better than Y.

The seed is computed using the last 4 tokens, but to get the predictive distribution they should still need the full history?

I haven't been in school since LLMs became useful, but if I were to "cheat", I'd ask the LLM for a very fine grained outline, and then just translate the outline into my own words. Then ask the LLM to fill in my citations in AMA format.

And honestly, this still retains like 95% of the value of writing a paper, because I did write it, the words did flow through my brain. I just used the LLM to avoid facing a blank page.

I've also thought about asking LLMs to simulate a forum conversation about the Civil War (or whatever the topic may be), and include a wrong comment that can be countered by writing exactly what the assignment requires, because I seem to have no trouble writing an essay when duty calls and someone is wrong in the internet.

These things don’t have to be foolproof to be effective deterrents.

What happens if I tell the LLM to reword every other sentence? — or every 5th word?

I must be missing something, because this seems to assume a contiguous output.

It's possible that this might break the method, but what seems most likely to me is that the LLM will simply reword every 5th word with some other word that it is more likely to use due to the watermark sampling. Thus the resulting output would display roughly the same level of "watermarkedness".

You might be able to have one LLM output the original, and then another to do a partial rewording though. The resulting text would likely have higher than chance "watermarkedness" for both LLMs, but less than you would expect from a plain output. Perhaps this would be sufficient for short enough outputs?

What happens when we all reading llm output all the time. Simply start to adapt more to llm writing styles, word choice, and possibly without realizing it watermark our own original writing?

You might be right, but my first instinct is that this probably wouldn't happen enough to throw off the water marking to badly.

The most likely used word is based off the previous four, and only works if there is enough entropy present that one of multiple word would work. Thus its not a simple matter of humans picking up particular word choices. There might be some cases where there are 3 tokens in a row that occur with low entropy after the first token, and then one token generation with high entropy at the end. That would cause a particular 5 word phrase to occur. Otherwise, the word choice would appear pretty random. I don't think humans pick up on stuff like that even subconsciously, but I could be wrong.

I would be interested to see if LLMs pick up the watermarks when fed watermarked training data though. Evidently ChatGPT can decode base64, [0] so it seems like these things can pick up on some pretty subtle patterns.

[0] https://www.reddit.com/r/ChatGPT/comments/1645n6i/i_noticed_...

Reply on news.ycombinator.com