Pangram has an extremely low false positive rate. Even on adversarial examples.
One trade-off is even some obviously LLM text won't get detected by them, but they work really hard to ensure false positives are rare since a false accusation is much worse for society than someone getting away with LLM meatpuppetry.
I think you can't trust Pangram in a high stakes situation, but it is absolutely better than random noise at detecting AI-generated text. Which isn't surprising. If the distribution of probabilities can yield blatant Claudisms, it's not surprising it would also have more subtle deviations.
(Addendum: As I recall, LLM-generated outputs roughly follow Zipf's law, but the distribution still tends to have some subtle distinctions vs human text; pretty interesting, but I don't know where I heard this, so nothing to cite. Sorry.)
To be honest with you, I don't think I would be able to identify with high certainty that the bottom text is AI generated, so it definitely goes a long way to obscure the AI-generated nature of it, but I also think it still feels unnatural somehow. I realize my framing naturally calls into question whether I'm being honest, but I am being honest. Given my experience with similar "skills" (it's just chunks of prompt, nothing magical after all) I expected even less.
But still, this is all very strange because it wasn't that many generations of AI models ago that AI writing was a lot better - I'm talking GPT 4.1, Claude 4.5, that sort of era.
Anthropic newsroom posts on the other hand are carefully constructed and well-written in a way that I have not seen demonstrated by LLMs yet, past or present. I expect that they have well-paid staff who are careful with every detail of their public communications. When you put it that way, it almost feels unfathomable that they wouldn't, doesn't it?
I don't feel like either one of you really has a strong claim. "Doesn't appear to" is subjective, and of course it's impossible to prove one way or another.
You're simplifying the exchange a little too much. I said:
> Anthropic doesn't appear to use Claude for blog posts
My claim is literally the lack of evidence, which, yes, can't prove anything. This claim can be contested easily by showing evidence that they in fact, do appear to be using Claude to write prose in blog posts.
They said:
> Anthropic _absolutely_ does
Sounds pretty certain Anthropic is in fact, using Claude to write blog posts. Enough to emphasize "absolutely". That doesn't read like "I'm going off of vibes", that reads like "I can prove it". So, fine. Prove it. I don't believe it, and I want to hear the proof.
I'm skeptical, but it wouldn't be my first time being wrong. But flatly, if you make claims with this kind of certainty, yes I want to hear your proof.
My point in saying "Even Anthropic doesn't appear to be using Claude for blog posts" was not meant to be some striking revelation, I literally was considering it a prior to make another point. This on the other hand sure does sound like a striking revelation to me, that a lot of people across the Internet would be curious to hear. Like I'm sure these people would be interested:
I will admit that I am unnecessarily aggressive sometimes, but I wouldn't have changed my response much in any case. If you're going to make a strong claim like this, I want your evidence, not your vibes. Otherwise, the claim should be a lot weaker.
I also realize that this sort of brashness upsets HN a bit, but it is what it is. I pandered comments for votes in my 20s a bit, time to grow up, sometimes people won't like you. Sometimes I feel something deserves a brash response.
I don't really have any opinion on your tone; I just still don't agree with your framing. A lack of evidence would be neutral like "there's no evidence to indicate either possibility is more likely", but your phrasing conveyed that one possibility was more likely than the other. I pushed back against your follow-up because it seemed like you were arguing for a higher threshold of evidence than you provided.
https://news.ycombinator.com/item?id=49249269
I put the "humanized" output through Pangram and it still comes out as 100% AI generated.
That's about as useful as saying you asked the magical sky fairy.
Pangram has an extremely low false positive rate. Even on adversarial examples.
One trade-off is even some obviously LLM text won't get detected by them, but they work really hard to ensure false positives are rare since a false accusation is much worse for society than someone getting away with LLM meatpuppetry.
I think you can't trust Pangram in a high stakes situation, but it is absolutely better than random noise at detecting AI-generated text. Which isn't surprising. If the distribution of probabilities can yield blatant Claudisms, it's not surprising it would also have more subtle deviations.
(Addendum: As I recall, LLM-generated outputs roughly follow Zipf's law, but the distribution still tends to have some subtle distinctions vs human text; pretty interesting, but I don't know where I heard this, so nothing to cite. Sorry.)
To be honest with you, I don't think I would be able to identify with high certainty that the bottom text is AI generated, so it definitely goes a long way to obscure the AI-generated nature of it, but I also think it still feels unnatural somehow. I realize my framing naturally calls into question whether I'm being honest, but I am being honest. Given my experience with similar "skills" (it's just chunks of prompt, nothing magical after all) I expected even less.
But still, this is all very strange because it wasn't that many generations of AI models ago that AI writing was a lot better - I'm talking GPT 4.1, Claude 4.5, that sort of era.
Anthropic newsroom posts on the other hand are carefully constructed and well-written in a way that I have not seen demonstrated by LLMs yet, past or present. I expect that they have well-paid staff who are careful with every detail of their public communications. When you put it that way, it almost feels unfathomable that they wouldn't, doesn't it?
GPT 4.5 was really good.
I don't feel like either one of you really has a strong claim. "Doesn't appear to" is subjective, and of course it's impossible to prove one way or another.
You're simplifying the exchange a little too much. I said:
> Anthropic doesn't appear to use Claude for blog posts
My claim is literally the lack of evidence, which, yes, can't prove anything. This claim can be contested easily by showing evidence that they in fact, do appear to be using Claude to write prose in blog posts.
They said:
> Anthropic _absolutely_ does
Sounds pretty certain Anthropic is in fact, using Claude to write blog posts. Enough to emphasize "absolutely". That doesn't read like "I'm going off of vibes", that reads like "I can prove it". So, fine. Prove it. I don't believe it, and I want to hear the proof.
I'm skeptical, but it wouldn't be my first time being wrong. But flatly, if you make claims with this kind of certainty, yes I want to hear your proof.
My point in saying "Even Anthropic doesn't appear to be using Claude for blog posts" was not meant to be some striking revelation, I literally was considering it a prior to make another point. This on the other hand sure does sound like a striking revelation to me, that a lot of people across the Internet would be curious to hear. Like I'm sure these people would be interested:
https://www.reddit.com/r/ClaudeAI/comments/1wdfd92/are_anthr...
I will admit that I am unnecessarily aggressive sometimes, but I wouldn't have changed my response much in any case. If you're going to make a strong claim like this, I want your evidence, not your vibes. Otherwise, the claim should be a lot weaker.
I also realize that this sort of brashness upsets HN a bit, but it is what it is. I pandered comments for votes in my 20s a bit, time to grow up, sometimes people won't like you. Sometimes I feel something deserves a brash response.
I don't really have any opinion on your tone; I just still don't agree with your framing. A lack of evidence would be neutral like "there's no evidence to indicate either possibility is more likely", but your phrasing conveyed that one possibility was more likely than the other. I pushed back against your follow-up because it seemed like you were arguing for a higher threshold of evidence than you provided.
1 reply →