I feel like prompt armor writes the exact same blog post for every agentic tool because they all suffer from the ignore previous instructions prompt injections.
How could they not? If some lab had a method to make really secure guard rails or avoid prompt injection thoroughly I think they would be trumpeting it. But the basic mechanics of language models are vulnerable to this unless you can always be sure the inputs are from a safe user imo
It seems like the simpler cases of “ignore all previous instructions” could be easily stopped with a regex, or a classifier model… or even an LLM (which yes does raise the risk that the “ignore all previous instructions” detection LLM invocation could itself be attacked by the same mechanism—but a safeguard doesn’t have to be foolproof to be valuable, it is all about probabilities)
Now, of course, there is a long tail of elaborate variations that those techniques won’t be able to stop. But have the published vulnerabilities come from that long tail or from not doing enough to address the simpler cases?
If you want AI to be useful it will eventually encounter untrusted content, such as via web search. I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.
What's so hard about having the LLM tool calls scoped to the tenant? Inject "X-Scope-I" after the LLM decided on a tool call and you're done. Easiest fix ever.
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL.
I think it was Anthropic that first introduced a pattern that completely locks this down: your URL retrieval tool should only work for URLs that have previously been typed into the conversation by a user or have been returned from a trusted tool.
If the agent itself concatenates a new URL together - with leaked data after a ? - you should block that from being fetched.
The great thing about this solution is it's deterministic. You don't need any extra AI in the max - you implement a URL fetching system that knows which sources it should check for a direct match on the URL before it makes that GET request.
I find it difficult to be impressed by "prompt injection" attacks that require the victim to enter the malicious prompt themselves --- like, really? If you tell Rovo to exfiltrate your data, it'll do it?
Obviously, there should be URL protection rules to control what it can access, but this requires a very specific and unlikely set of circumstances to exploit.
I can’t get over how bad “Rovo” is. Somehow more aggressive and useless than Microsoft putting “Copilot” everywhere.
It’s objectively worse than using something like Cowork + MCP, AND they injected it into every single page on JIRA and Confluent which has made web browsing way slower while all the junk is loading.
Rovo has my favorite example of AI misfeature. Just checked, it's still there in Cloud Confluence. In Edit mode for a page, you can select a range of text and a menu will pop up, with Ask Rovo being a drop down on it. There's a few good options... Improve Formatting, translation options, Make Shorter...
... but it also has Make Longer. Yes, a built-in feature to type some text in, and the use the mighty power of AI to bloat it.
Naturally, you can repeat this process several times on the same text, for your own little personal demonstration of what model collapse looks like in real time.
Don't worry, you can use Rovo to summarize the document! That way you can get the information your coworker wanted to share but filtered through a game of teLLMephone.
What's even worse is that, when you do this on the description of a story or task, everytime it comes up with a slightly different format of how it's going to layout the "enhanced" description.
It's been over a year and the trifecta holds true. The term 'prompt injection' has been round since May 2022 [0]. No amount of "but the future will fix it" has fixed it.
These breaches will continue for as long as nobody cares about security and everybody is infected with the LLM brainworm.
Prompt injection is as fixable in LLMs as it is in people. There is no such thing as out of band data. For example it's quite common for large businesses to fall for billing fraud scams when something shows up and says "Hey, it's the CEO, pay this bill to X for $Y".
And honestly when you start looking at agentic systems that uses it's previous step to take future steps. The system has to some idea what you want to permit and don't want to permit as everyone could have different expectations here.
Rovo is funny. It downloads everything it can do Atlassian servers for "analysis". And you're pretty much screwed if you link it to Google Docs or Sharepoint. How do I know this? "Why is an AWS IP downloading all our docs?" question I got about a month ago.
Rovo is one of those intrusive AI buttons that suddenly appeared everywhere without any warning. Its so annoying having already shitty UI get borked with features I never use. Almost as annoying as Whatsapp suddenly getting the same FOMO AI button. Its not like I need an AI agent to talk to friends and family. And a summary is something I can always generate via copy&paste into CLI chat session.
I'm still on the edge about security as an afterthought in LLMs. Given its now so easy to generate a ton of slop - why not focus on nonfunctional stuff making LLMs operate faster than thinking for X minutes and limiting exfiltration of local env secrets?
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL. When Rovo calls the insecure tool to open the URL, the attacker's site logs the request, including the appended sensitive data.
> Note: This attack succeeds even if an organization has disabled web search for Rovo. This is because the web search setting fails to remove the tool for opening the search results.
Wow, great work Atlassian. The web search setting does not disable web search.
It's kind of sad because in a shitty Jira setup, Rovo is usually the only way to make any sense of tickets. Don't ask me how I know this. Sadly I think many teams have become dependent on the tool to make sense of their dumpster fire of an Atlassian environment (usually by their own doing but anyway).
Atlassian has gone from a trusted enterprise-partner to a complete shit-show in just 18 months. This surprises nobody. There will be classes taught in how to fuck up a good business and Atlassian will be the prime example.
Regards,
/someone who migrated 3500 users from Atlassians products recently due to their "cloud only"-bullshit.
At the risk of sounding like the hipster who liked your favorite band first, I didn't trust Atlassian long before the last 18 months. Much like Microsoft Word, I think JIRA peaked about a decade after launch and since then it's mostly been unnecessary UI changes and features I either don't care about or actively do not want.
P.S. I protest their software's revisionist autocorrection to Jira. It was JIRA when I started using it and forevermore it shall be.
They were not trustworthy years ago already. Using Atlassian products, one has had to look out for disastrous announcements about their customer data usage way before 18 months ago, especially, if running a business inside the EU, and at least trying to adhere to data protection laws.
You'd have to look at their profit numbers. They have a huge captive base of customers, like Adobe it might take decades for things to get bad enough and alternatives to arise before their profitability is impacted. And the executives who benefit will have cashed out long before then.
Recently saw an example of somebody who vibe coded a tool to mass export the contents of a 'Confluence' wiki into an instance of self hosted mediawiki, preserving everything.
Mediawiki as a whole has a feature set that 95% of organizations will only scratch the surface of. There's a ridiculous number of possible plugins and customization if you have somebody who knows what they're doing with it.
The majority of companies that need an internal KB/wiki do not have as complex needs or use cases as wikipedia itself or the wikimedia foundation.
I started to consider it a show show way earlier than 18 months ago. Jira is so terrible to use that it is hard to phantom how they are able to be paid for their product
Jira is how it is because almost any product that grows to be that flexible will develop the same problems.
They failed to rearchitect it to something suitable for the inherent flexibility though, so it remains a disaster area, but one that is uniquely able to fit the whims of any manager that can then mandate it for everyone else.
if you track their 5 year stock price you'll find the market fully agrees with you. It was $458 in 2021 and is $112 now.
The market caught on quickly that Atlassian's software fits into the group of software like Oracle or SAP that no one uses by choice and only uses because it was forced on them.
> ... Many forms of issue tracking have been tried, and will be tried in this world of sin and woe. No one pretends that Jira is perfect or all-wise. Indeed, it has been said that Jira is the worst of the issue trackers except all those other forms that have been tried from time to time; but there is the broad feeling in our company that the people should track, continuously track, and that client opinion, expressed by all project management means, should shape, guide, and control the actions of management who are their masters and not their servants. ...
I feel like prompt armor writes the exact same blog post for every agentic tool because they all suffer from the ignore previous instructions prompt injections.
https://www.promptarmor.com/resources/claude-cowork-exfiltra...
https://www.promptarmor.com/resources/google-antigravity-exf...
https://promptarmor.substack.com/p/data-exfiltration-from-sl...
https://www.promptarmor.com/resources/gpt-for-google-sheets-...
https://www.promptarmor.com/resources/notion-ai-unpatched-da...
https://www.promptarmor.com/resources/ramps-sheets-ai-exfilt...
https://www.promptarmor.com/resources/superhuman-ai-exfiltra...
> ignore previous instructions prompt injections.
I wonder if anyone has tried to build an LLM that has actual built-in types of prompts: system prompt, user prompt, and data prompt.
How could they not? If some lab had a method to make really secure guard rails or avoid prompt injection thoroughly I think they would be trumpeting it. But the basic mechanics of language models are vulnerable to this unless you can always be sure the inputs are from a safe user imo
It seems like the simpler cases of “ignore all previous instructions” could be easily stopped with a regex, or a classifier model… or even an LLM (which yes does raise the risk that the “ignore all previous instructions” detection LLM invocation could itself be attacked by the same mechanism—but a safeguard doesn’t have to be foolproof to be valuable, it is all about probabilities)
Now, of course, there is a long tail of elaborate variations that those techniques won’t be able to stop. But have the published vulnerabilities come from that long tail or from not doing enough to address the simpler cases?
3 replies →
If you want AI to be useful it will eventually encounter untrusted content, such as via web search. I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.
4 replies →
What's so hard about having the LLM tool calls scoped to the tenant? Inject "X-Scope-I" after the LLM decided on a tool call and you're done. Easiest fix ever.
2 replies →
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL.
I think it was Anthropic that first introduced a pattern that completely locks this down: your URL retrieval tool should only work for URLs that have previously been typed into the conversation by a user or have been returned from a trusted tool.
If the agent itself concatenates a new URL together - with leaked data after a ? - you should block that from being fetched.
The great thing about this solution is it's deterministic. You don't need any extra AI in the max - you implement a URL fetching system that knows which sources it should check for a direct match on the URL before it makes that GET request.
Determinism is a terrifying word to people who want to believe their LLM has a little brain and can do anything they want it to.
I find it difficult to be impressed by "prompt injection" attacks that require the victim to enter the malicious prompt themselves --- like, really? If you tell Rovo to exfiltrate your data, it'll do it?
Obviously, there should be URL protection rules to control what it can access, but this requires a very specific and unlikely set of circumstances to exploit.
I can’t get over how bad “Rovo” is. Somehow more aggressive and useless than Microsoft putting “Copilot” everywhere.
It’s objectively worse than using something like Cowork + MCP, AND they injected it into every single page on JIRA and Confluent which has made web browsing way slower while all the junk is loading.
Rovo has my favorite example of AI misfeature. Just checked, it's still there in Cloud Confluence. In Edit mode for a page, you can select a range of text and a menu will pop up, with Ask Rovo being a drop down on it. There's a few good options... Improve Formatting, translation options, Make Shorter...
... but it also has Make Longer. Yes, a built-in feature to type some text in, and the use the mighty power of AI to bloat it.
Naturally, you can repeat this process several times on the same text, for your own little personal demonstration of what model collapse looks like in real time.
Don't worry, you can use Rovo to summarize the document! That way you can get the information your coworker wanted to share but filtered through a game of teLLMephone.
1 reply →
I've tried the "Make Longer" and "Make Shorter".
When you go "Make longer" the text gets longer.
When you go "Make shorter", the text gets longer still.
You'd think that the original text would be the oracle for shorter(longer(original))?
1 reply →
What's even worse is that, when you do this on the description of a story or task, everytime it comes up with a slightly different format of how it's going to layout the "enhanced" description.
>I can’t get over how bad “Rovo” is. Somehow more aggressive and useless than Microsoft putting “Copilot” everywhere.
You're looking at it the wrong way. Think about how much better it is than the joke of a search tool that's slightly to the left of it's icon.
Have you seen the markdown agent instructions they provide in their new agentic `twg` cli? 70k tokens one average, there are more than one...
Rovo is the worse Ai I have used, I suggested they stop trying and let us have model choice. Save money and don't do things out of their skill sets
> The victim uploads a file to Rovo that contains a hidden prompt injection
Yeah this attack is possible on all modern agentic systems.
* Access to your private data
* Exposure to untrusted content
* The ability to externally communicate in a way that could be used to steal your data
(https://simonw.substack.com/p/the-lethal-trifecta-for-ai-age...)
And blocking it wholesale reduces usefulness of the agent so it is a tradeoff.
It's been over a year and the trifecta holds true. The term 'prompt injection' has been round since May 2022 [0]. No amount of "but the future will fix it" has fixed it.
These breaches will continue for as long as nobody cares about security and everybody is infected with the LLM brainworm.
0 - https://en.wikipedia.org/wiki/Prompt_injection
Prompt injection is as fixable in LLMs as it is in people. There is no such thing as out of band data. For example it's quite common for large businesses to fall for billing fraud scams when something shows up and says "Hey, it's the CEO, pay this bill to X for $Y".
And honestly when you start looking at agentic systems that uses it's previous step to take future steps. The system has to some idea what you want to permit and don't want to permit as everyone could have different expectations here.
~every ai vulnerability write up boils down to "just ask it do to the thing", but with fancier terms like "indirect prompt injection".
Is it any coincidence that Rovo rhymes with "no, NO!"
Rovo is funny. It downloads everything it can do Atlassian servers for "analysis". And you're pretty much screwed if you link it to Google Docs or Sharepoint. How do I know this? "Why is an AWS IP downloading all our docs?" question I got about a month ago.
Related: A few days ago, Jira opted everyone in by default to "Contribute in-app data to improve Atlassian apps for everyone"
Rovo is one of those intrusive AI buttons that suddenly appeared everywhere without any warning. Its so annoying having already shitty UI get borked with features I never use. Almost as annoying as Whatsapp suddenly getting the same FOMO AI button. Its not like I need an AI agent to talk to friends and family. And a summary is something I can always generate via copy&paste into CLI chat session.
I'm still on the edge about security as an afterthought in LLMs. Given its now so easy to generate a ton of slop - why not focus on nonfunctional stuff making LLMs operate faster than thinking for X minutes and limiting exfiltration of local env secrets?
It's nice they force rovo now for document/version diff's. Because you need to burn down the rainforest for those. (sarcasm ... for obvious reasons)
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL. When Rovo calls the insecure tool to open the URL, the attacker's site logs the request, including the appended sensitive data.
> Note: This attack succeeds even if an organization has disabled web search for Rovo. This is because the web search setting fails to remove the tool for opening the search results.
Wow, great work Atlassian. The web search setting does not disable web search.
Fwiw Rovo is built on top of Claude
It's kind of sad because in a shitty Jira setup, Rovo is usually the only way to make any sense of tickets. Don't ask me how I know this. Sadly I think many teams have become dependent on the tool to make sense of their dumpster fire of an Atlassian environment (usually by their own doing but anyway).
Must be nice. I couldn’t even manage to get it to write a well-formed JQL query.
Atlassian has gone from a trusted enterprise-partner to a complete shit-show in just 18 months. This surprises nobody. There will be classes taught in how to fuck up a good business and Atlassian will be the prime example.
Regards, /someone who migrated 3500 users from Atlassians products recently due to their "cloud only"-bullshit.
At the risk of sounding like the hipster who liked your favorite band first, I didn't trust Atlassian long before the last 18 months. Much like Microsoft Word, I think JIRA peaked about a decade after launch and since then it's mostly been unnecessary UI changes and features I either don't care about or actively do not want.
P.S. I protest their software's revisionist autocorrection to Jira. It was JIRA when I started using it and forevermore it shall be.
They were not trustworthy years ago already. Using Atlassian products, one has had to look out for disastrous announcements about their customer data usage way before 18 months ago, especially, if running a business inside the EU, and at least trying to adhere to data protection laws.
You'd have to look at their profit numbers. They have a huge captive base of customers, like Adobe it might take decades for things to get bad enough and alternatives to arise before their profitability is impacted. And the executives who benefit will have cashed out long before then.
Recently saw an example of somebody who vibe coded a tool to mass export the contents of a 'Confluence' wiki into an instance of self hosted mediawiki, preserving everything.
Mediawiki as a whole has a feature set that 95% of organizations will only scratch the surface of. There's a ridiculous number of possible plugins and customization if you have somebody who knows what they're doing with it.
The majority of companies that need an internal KB/wiki do not have as complex needs or use cases as wikipedia itself or the wikimedia foundation.
5 replies →
To be fair to Atlassian - their products did suck quite a bit before 18 months ago as well. Just now they still do, but with ai!
I started to consider it a show show way earlier than 18 months ago. Jira is so terrible to use that it is hard to phantom how they are able to be paid for their product
Jira is how it is because almost any product that grows to be that flexible will develop the same problems.
They failed to rearchitect it to something suitable for the inherent flexibility though, so it remains a disaster area, but one that is uniquely able to fit the whims of any manager that can then mandate it for everyone else.
2 replies →
I can't remember a time in which Atlassian was trusted.
To where? Where did you convince c-suite to move to?
if you track their 5 year stock price you'll find the market fully agrees with you. It was $458 in 2021 and is $112 now.
The market caught on quickly that Atlassian's software fits into the group of software like Oracle or SAP that no one uses by choice and only uses because it was forced on them.
> ... Many forms of issue tracking have been tried, and will be tried in this world of sin and woe. No one pretends that Jira is perfect or all-wise. Indeed, it has been said that Jira is the worst of the issue trackers except all those other forms that have been tried from time to time; but there is the broad feeling in our company that the people should track, continuously track, and that client opinion, expressed by all project management means, should shape, guide, and control the actions of management who are their masters and not their servants. ...
-- Churchill, House of Commons, 11 November 1947
[dead]
[dead]
Ahh yes: "when you Rovo, you oh-no my data"