Comment by kamranjon
13 hours ago
By evidence I mean logs, I mean IP addresses, I mean timestamps. They claim millions of requests, let’s see literally any of them?
I don’t consider a tweet by Denise Wu, who works at Anthropic, to be reproducible evidence.
I don’t consider “Caveat: fully AI-generated research.” To be someone taking time to analyze anything in great detail.
Because two AI models produce vaguely similar front-end styles when generating similar prompts I also do not consider to be of much value?
I think this is what I mean when I say the U.S. has its head in the sand. The Chinese labs are releasing ~60 page research reports with citations and analyses and evidence and Anthropic is throwing up defensive blog posts with zilch. I’ve seen more detail in a tech blog from Uber than anything I’ve seen from Anthropic.
You've backtracked significantly here.
"Zero evidence" as you claimed earlier isn't accurate. You've moved the goalposts from "evidence" to "raw internal logs I can independently audit," which is a different and very high standard. Sure Anthropic didn't publish logs, IP addresses, timestamps, or account IDs of the accounts involved. But that's true of any cybersecurity breach/abuse disclosure ever made. Companies are furtive to reveal how they detect fraud, because doing so exposes the signals used to detect bad actors, and makes future abuse easier. Not revealing the "evidence" you're asking for is industry standard practice. You're complaining that Anthropic is following industry standard practice, and conveniently defining the "evidence" you need as something Anthropic is never going to publish.
> I don’t consider a tweet by Denise Wu, who works at Anthropic, to be reproducible evidence.
Is the issue here that she works at Anthropic? Because Denise Wu doesn't work there.
> I don’t consider “Caveat: fully AI-generated research” to be someone taking time to analyze anything in great detail.
The experiments were run by Ryan Greenblatt, who is a real AI safety researcher (at Redwood Research).
The identity experiments and Greenblatt analysis are trivially reproducible. The methodology, code, and metrics are all there in the Github repository. You can ask your preferred AI to independently replicate these results, and it will give you a result within an hour.
You’ve also reduced the evidence to “two models producing vaguely similar front-end styles,” which is not what either analysis shows.
From the analysis, Kimi K3 identifies itself as Claude 15% of the time. How do you explain that? Qwen and GPT identify themselves as Claude 0% of the time.
If a long document is too much analysis for you, someone else made a simple chart which measures the KL divergence between Kimi K3 and other major models. They found K3 is unusually similar to Fable 5 & Opus models. That is, Kimi K3 has an very similar style and phrasing to that of Anthropic models. That behavior is expected from a model distilled from Claude.
https://typebulb.com/u/lab/you-re-relatively-right/full
"From the analysis, Kimi K3 identifies itself as Claude 15% of the time. How do you explain that? Qwen and GPT identify themselves as Claude 0% of the time."
Qwen and GPT have special guards that trigger when asked to identify, Kimi doesnt. I dont understand the argument. Kimi is an LLM and does not know what it is. It will give you the most likely answer which sometimes is Claude.
I wouldn’t say I’ve backtracked- I think I’ve been incredibly consistent here. Chinese labs are releasing open weight models, research and analysis. Anthropic is not. They haven’t produced any actual evidence of distillation themselves and what they have presented is tenuous at best.