Note that while that page is from Pangram, the studies on it are not. If they are excluding studies that show Pangram performs poorly in realistic conditions, please share those studies!
This is incorrect. You cannot identify the presence of text watermarks if you do not have access to the models so that you can spot where expected tokens diverge from actual tokens based off the weights. That does not preclude Pangram from being able to reliably identify LLM-generated text from models whose outputs they’ve trained their own detection model upon.
2025 called and it wants it's take on AI detection back.
https://www.pangram.com/blog/third-party-pangram-evals
Note that while that page is from Pangram, the studies on it are not. If they are excluding studies that show Pangram performs poorly in realistic conditions, please share those studies!
If Pangram was snake oil (it's not but could be more mindfully marketed), it's definitely not now that every major lab has added watermarking.
You wouldn't even need to be able to read the information the watermark encodes to recognize it exists in a piece of text
This is incorrect. You cannot identify the presence of text watermarks if you do not have access to the models so that you can spot where expected tokens diverge from actual tokens based off the weights. That does not preclude Pangram from being able to reliably identify LLM-generated text from models whose outputs they’ve trained their own detection model upon.
They have unwatermarked text from the same base models, human text, and watermarked text.
If they couldn't train a reliable detector with that they wouldn't exist.