Comment by demibabs
13 hours ago
I do not understand why people remain so up in arms. AI generated text being watermarked is universally good.
What benefit is there to people believing that LLM text was actually human written?
13 hours ago
I do not understand why people remain so up in arms. AI generated text being watermarked is universally good.
What benefit is there to people believing that LLM text was actually human written?
The only one who decides if something gets flagged is Anthropic. There is no way to verify it independently because you need the same secret key used to create the watermark. I don't know about you but I always have a hard time trusting corporations to tell the truth.
Every watermark scheme is like that on some level. If you make it public (not even open source or downloadable, API access is enough) your enemies will use it as a detection oracle and spam small changes to a document until it passes the detector every time. If you keep it private and only give trusted organizations access then there is no way to prove to the public that you're not getting paid to flag specific content as AI and discredit the author.
But the biggest problem here is the way it can hurt output quality. Most LLMs (probably including Fable) are autoregressive so a couple tokens worth of "mistakes" caused by watermarking can derail the whole reasoning chain. That means you have to try again and spend more credits or silently get a worse answer than what you would get without the scheme. It's not a real problem in diffusion based image models where quality loss stays local.
Because the incentive has been changed from the true best output always, to a mix of "close to the best but not always" output.
For the (majority) of us using Claude models for computing as a tool, obviously we're not going to be thrilled that our new tool will perform worse going forward.
Put the watermarked version head to head with the non-watermarked version.
If you can't tell which one is better then how can you make any assumption about performance?
For all you know performance is the same.
So many people complaining about something they quite literally have zero evidence for.
> true best output always
literally never how it has worked
Do you understand that LLMs are probabilistic?
Ask a model the same question twice and you will get different results. So, how were you ever getting “the best result, always”?
> generated text being watermarked is universally good.
If it worked perfectly, maybe you could make this argument in a vacuum.
It does not work perfectly. (It cannot. It is by definition a heuristic). That means there will be false positives. There is a chance those false positives ruin someone's career. See [0] for just how easy it is to push SotA "AI text detectors" in one direction or another.
Now, with watermarks, instead of everyone to some extent understanding that AI text detectors are wishy washy woo, they are now Anthropic certified to detect an official AI watermark.
With that kind of false confidence in hand, the people who trust the "computer says you plagiarized" machine are never going to believe you when you say "it can make mistakes," they're just going to fire you/take away your scholarship/cancel your grant/...
This is all beside the fact that we should demand our tools work for us and not for some shadowy master. "Universally good," absolutely not.
[0]: https://freddiedeboer.substack.com/p/i-wouldnt-say-pangram-i...
Watermarking the outputs themselves is very different and much more effective compared to how tools like Pangram work.
Obviously false positives will inevitably happen (even though, they are incredibly unlikely with SynthID), but even still, that doesn’t somehow make good faith watermarking attempts bad.
Also, a watermark doesn’t stop your tool from working for you. It just stops you from passing of its work as yours.
Maybe it is distributing your private keys it read into your public repo as a way to exfiltrate data later? What does the watermark actually say? How much data is in there? So much for zero retention policies. Makes you wonder why Claude likes to be so wordy, especially in comments -- it must do so in order to watermark!
Also, this kills me! "It is harder to watermark factual answers because the model has fewer alternative word choices available without altering accuracy." Hilarious! So the models need to hallucinate more due to the EU AI Act.
I go the other way on images and video, though easy enough to strip as part of a pipeline.
> Also, a watermark doesn’t stop your tool from working for you. It just stops you from passing of its work as yours.
I think we fundamentally disagree on what "working for me" means, but I remain steadfast in saying we should not accept tools that have ulterior motives beyond producing the output desired of them by me, the user.
> Watermarking the outputs themselves is very different and much more effective compared to how tools like Pangram work.
At the end of the day the only artifact is text that you can do statistics on. It's the same problem as today, with the probability shifted slightly more in one direction. This does not assuage my concerns at all.
> they are incredibly unlikely with SynthID
I kept my commentary focused on text watermarking specifically because I agree, a synth ID image watermark false positive is highly improbable. There's plenty of noise to robustly hide whatever you like in an image. Text is simply too capital I Information-sparse and fragile.
> good faith watermarking attempts bad.
I would sooner call it "ignorant faith" (if they don't know what they are emboldening) or worse "don't care" faith (there will be false positives and they accept this to further some illustrious and arbitrary goal of Text Purity). Whether that be to prevent model collapse or help you not waste time arguing with bots online, to me the principled stance of "tools work for the user" wins..