Comment by bruce343434

addressing any of the points would require locking the AI down and making it less general and less "agentic". Your concerns make sense if you look at the AI as an information retrieval engine.

   - why is single URL crawl with 20+ redirects not flagged as problematic and/or aborted?

There could be legitimate use cases for interacting with a website like this that could serve the user.

   - why is a query about a coffee place based on its public URL even seeded with the users' context and confidential information? 

Because it provides context in how the agent interacts with the site (in this case to detrimental effect)

   - why dont they just look up the coffee place on a trusted source like google maps and continue from there?

The ai was explicitly instructed to check the given url

   - why is the basic "social" engineering style attack working?

because ai can not separate prompt from information, they share the same input channel. state of the art ai has some amount of "common sense" as to when it is being prompt injected or engineered, but this isn't exhaustive

   - why is the cloudflare impersonation not challenged if the website is clearly not from cloudflare and there are zero references from cloudflare to this website in the training corpus?

because the ai didn't think to check if the website is truly behind some sort of cloudflare product or not