Comment by simonw
2 years ago
What are those comments meant to be illustrating?
I spent a bunch of time playing with Golden Gate Claude. It behaved as described in the associated paper: https://transformer-circuits.pub/2024/scaling-monosemanticit...
> For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge!
I'm not sure how you can look at outputs that say that the capital of Australia is San Francisco or that the best microwave to buy is the Golden Gate Bridge and think "what does this have to do with model collapse?"
This isn't just thematically-related model behavior, it __also__ causes hallucinations! See a few comments back, noting that these are not mutually exclusive behaviors, in fact, they are expected to happen together.
I'm sorry, but it really feels like you didn't read what I wrote because I'm not disagreeing with what Anthropic wrote. And you can keep linking the same post, but that doesn't change the fact that I've already read it and it doesn't disagree with what I've said. Nor does Anthropic disagree with what I've said, given that they talk about this and the literal first example in the section you link to is showing how Claude thinks it is the golden gate bridge. Just actually read my last comment.