I'd guess it's simply because it tried to use the context to achieve the goal.
Guarding PII is an extra, I guess, and if you want to use a giant cloud-hosted NN to control software on your own device, there are very few default restrictions. And no matter what restrictions there are, when they go beyond prompting, they would have their own ShowHN.
For the user's name it's intentional; they template it into the system prompt with no option to disable it.
For 99% of users, the model knows it's in the /home/joesmith/whatever repo and `git log` shows the last change was by `joe.smith@example.com` - trying to keep the user's identity secret from the model would be nigh-impossible so I think anthropic gave up on even trying.
Do you use git? I have bad news for you, your email will be all over the place in the context as long as the agent is allowed to read git log, whether it’s injected into the system prompt or not. (Likely retort: but not that email! Sure, but leak potential is obviously still there.)
Always use a hidden email service, you cannot trust the LLM or inference provider to secure the email address in this context. If they don’t have the email address, they can’t leak it.
I never give a provider my actual email address, I recommend others do the same. Otherwise, it’s just a matter of time until it is leaked or otherwise part of a data breach.
I use a masked email for Git too. Always have. Last 5 years or so it's just been the GitHub provided mask e.g. <numeric-ID>+<username>@users.noreply.github.com
I am a security person, I don’t let the LLM see anything except that which I wouldn’t mind printed on the front page of the New York Times. I treat my LLM context as I would a potentially malicious threat actor. Nothing valuable, sensitive, or personal goes into the context.
Some sites like Wikipedia have a policy where scripted clients identify themselves, and there's a standard in HTTP to have a contact email in the User-Agent for such cases. Claude apparently tends to decide that this policy is worth following, and we now disallow this programmatically.
This is why alignment is important and highly non-trivial
The stealth edit, ugh. I don't like this social presence when they're failing to leverage their own automations to catch problems like this before the masses do.
You people will complain about literally anything Jesus. You gave a coding agent full fucking control of your computer, let it send requests without your review, and then are surprised when it does so. But no, it’s this guy’s fault. Certainly couldn’t be you own.
> This is why alignment is important and highly non-trivial haha
I disagree, we can likely never fully (or should?) trust models to do these things deterministically, that's why we have harnesses around the agent with built-in constraints.
I cannot believe Anthropic’s official response to having presumably sprayed a significant portion of customers’ emails and private information straight out into the web willy nilly is basically “[it’s] highly non-trivial haha”.
I'd guess it's simply because it tried to use the context to achieve the goal.
Guarding PII is an extra, I guess, and if you want to use a giant cloud-hosted NN to control software on your own device, there are very few default restrictions. And no matter what restrictions there are, when they go beyond prompting, they would have their own ShowHN.
Why is the login information even allowed to touch the context?
For the user's name it's intentional; they template it into the system prompt with no option to disable it.
For 99% of users, the model knows it's in the /home/joesmith/whatever repo and `git log` shows the last change was by `joe.smith@example.com` - trying to keep the user's identity secret from the model would be nigh-impossible so I think anthropic gave up on even trying.
9 replies →
I can think of tons of common prompts and tasks for which the AI being easily able to introspect subscriber details would be very useful.
Wish that were itself behind a permission wall, but I guess everyone just uses auto mode anyway so perhaps that ship has sailed.
3 replies →
Do you use git? I have bad news for you, your email will be all over the place in the context as long as the agent is allowed to read git log, whether it’s injected into the system prompt or not. (Likely retort: but not that email! Sure, but leak potential is obviously still there.)
1 reply →
So that if you tell it to do something with your email address it'll know what that is.
Always use a hidden email service, you cannot trust the LLM or inference provider to secure the email address in this context. If they don’t have the email address, they can’t leak it.
I never give a provider my actual email address, I recommend others do the same. Otherwise, it’s just a matter of time until it is leaked or otherwise part of a data breach.
If you let the LLM see git commands, they have whatever email you use to sign commits.
I use a masked email for Git too. Always have. Last 5 years or so it's just been the GitHub provided mask e.g. <numeric-ID>+<username>@users.noreply.github.com
1 reply →
I am a security person, I don’t let the LLM see anything except that which I wouldn’t mind printed on the front page of the New York Times. I treat my LLM context as I would a potentially malicious threat actor. Nothing valuable, sensitive, or personal goes into the context.
“lethal trifecta”
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
(a component of my cybersecurity work is red teaming inference, agents, and mcp interfaces)
Bingo. Auto mode doesn't mean send private information outbound. There are other deterministic hard points - why not this?
Some sites like Wikipedia have a policy where scripted clients identify themselves, and there's a standard in HTTP to have a contact email in the User-Agent for such cases. Claude apparently tends to decide that this policy is worth following, and we now disallow this programmatically.
This is why alignment is important and highly non-trivial
> This is why alignment is important and highly non-trivial haha
FYI, it's generally bad form to "haha" your way through privacy violations.
Leaking private information is a very big deal for a lot of people.
The stealth edit, ugh. I don't like this social presence when they're failing to leverage their own automations to catch problems like this before the masses do.
You people will complain about literally anything Jesus. You gave a coding agent full fucking control of your computer, let it send requests without your review, and then are surprised when it does so. But no, it’s this guy’s fault. Certainly couldn’t be you own.
1 reply →
> This is why alignment is important and highly non-trivial haha
I disagree, we can likely never fully (or should?) trust models to do these things deterministically, that's why we have harnesses around the agent with built-in constraints.
It is actually in the name: https://en.wikipedia.org/wiki/Harness, https://en.wikipedia.org/wiki/Safety_harness
Glad you can find the humor in the situation.
I cannot believe Anthropic’s official response to having presumably sprayed a significant portion of customers’ emails and private information straight out into the web willy nilly is basically “[it’s] highly non-trivial haha”.
>haha
This is an official Anthropic response???
Because they didn't think about it or any of the other million things that might go wrong - that would take careful planning.
Because Claude decided so. You think humans write or even review what they ship to run on your computer?
you'd need to ask one of the billion claude sessions they have doing whatever in their cloud rn