Comment by NewsaHackO
8 hours ago
I am pretty sure they did A/B testing to show it didn't. I could gave sworn they even released a quiz were the user has to try and guess which answer is watermarked or not and it was impossible to tell.
8 hours ago
I am pretty sure they did A/B testing to show it didn't. I could gave sworn they even released a quiz were the user has to try and guess which answer is watermarked or not and it was impossible to tell.
That's not the metric, though.
You either output the best version, or you output something else.
You can't do both.
That's not the case, because LLMs are non-deterministic.
It only alters outputs when the last layer of the neural network give significant weights to multiple tokens, and it would anyway have picked a random answer.
Instead it picks a non-random one, but non-random in such a way that you can't tell without the private key of the watermarking.
This mostly adds randomness these days for branches in syntax that make no difference, and the model has no reason to believe make a difference. Anything that matters, it is much more confident in the last layer of weights on the token to use.
I feel as though you are overlooking simple statistics/confidence intervals. It absolutely possible for two different works to be not have a distinguishable difference in quality.
Agreed, but not if one of them was altered to contain a secret message. That one will have a distinguishable difference in quality. Maybe (almost certainly) negligible, but still there.
1 reply →
that's fair. but it outputted non-best version in the first place sometimes too. Now it still output non-best version sometimes.