Comment by bflesch
15 hours ago
Still, this is a vuln in what I imagine is their most frequently used path:
Attacker provides link to website, their software crawls the website, and during the crawl there should not happen security issues as fundamental as this.
It's baffling that the Website crawler can make 50 changes to the URL in a query that tries to compare several public entities and on top of this manages to leak user secrets.
To me this shows a striking lack of defense-in-depth thinking:
- why is single URL crawl with 20+ redirects not flagged as problematic and/or aborted?
- why is a query about a coffee place based on its public URL even seeded with the users' context and confidential information?
- why dont they just look up the coffee place on a trusted source like google maps and continue from there?
- why is the basic "social" engineering style attack working?
- why is the cloudflare impersonation not challenged if the website is clearly not from cloudflare and there are zero references from cloudflare to this website in the training corpus?
In terms of web crawling, cloudflare is like the government. You shouldn't be able to walk up to someone and say "Hey I'm the tax man, please pay your income tax in cash to me right now!" without being challenged.
I know there are fundamental reasons in the LLM technology why this kind of attack is possible, but there should be so many more checks around web crawling in Claude.
How can security engineers at Anthropic say they know about this kind of vulnerability but have not implemented any of these defense in depth mitigations for it? Is everybody out shopping for a new yacht?
addressing any of the points would require locking the AI down and making it less general and less "agentic". Your concerns make sense if you look at the AI as an information retrieval engine.
There could be legitimate use cases for interacting with a website like this that could serve the user.
Because it provides context in how the agent interacts with the site (in this case to detrimental effect)
The ai was explicitly instructed to check the given url
because ai can not separate prompt from information, they share the same input channel. state of the art ai has some amount of "common sense" as to when it is being prompt injected or engineered, but this isn't exhaustive
because the ai didn't think to check if the website is truly behind some sort of cloudflare product or not