I'm following Owain Evans on X and some of the papers they've been sharing are much worse. IIRC there was something with fine-tuning a LLM to be bad actor, then letting it spit out some text, and if that response was copy-pasted into the context of the ORIGINAL LLM (no fine-tune) it was also "infected" with this bad behavior.
And it makes a lot of sense, the pre-training is not perfect, it's just the best of what we can do today and the actual meaning leaks through different tokens. Then, QKV lets you rebuild the meaning from user-provided tokens, so if you know which words to use, you can totally change the behavior of your so-far benign LLM.
There was also paper about sleeper agents and I am by no way a doomer but the LLM security is greatly underestimated, and the prompt injection (which is impossible to solve with current generation of LLMs) is just the tip of the iceberg. I am really scared of what hackers will be able to do tomorrow and that we are handing them our keys willingly.
You're right that this is a concern but this and the followup are also totally unhelpful.
Even if you don't want to do any additional work explaining it or finding a source, all you have to do to change this message from being dickish to being helpful would be to phrase it more like "I think there are some serious risks with this approach from a prompt injection standpoint. I would recommend doing some research on the risks for AI agents with unfettered access to the internet and prompt injection."
And if spending a few more seconds typing that out is still too much of a waste of time for you to do, I might question if you have time to waste commenting on HN at all when you can't uphold basic social contracts with the time you do have.
why should one be more concerned about hypothetical prompt injection and that being the reason not to use clawdbot? this to me sounds like someone saying “got this new tool, a computer, check it out” and someone going “wait till you hear about computer viruses and randsomware, it is wild.”
I'm following Owain Evans on X and some of the papers they've been sharing are much worse. IIRC there was something with fine-tuning a LLM to be bad actor, then letting it spit out some text, and if that response was copy-pasted into the context of the ORIGINAL LLM (no fine-tune) it was also "infected" with this bad behavior.
And it makes a lot of sense, the pre-training is not perfect, it's just the best of what we can do today and the actual meaning leaks through different tokens. Then, QKV lets you rebuild the meaning from user-provided tokens, so if you know which words to use, you can totally change the behavior of your so-far benign LLM.
There was also paper about sleeper agents and I am by no way a doomer but the LLM security is greatly underestimated, and the prompt injection (which is impossible to solve with current generation of LLMs) is just the tip of the iceberg. I am really scared of what hackers will be able to do tomorrow and that we are handing them our keys willingly.
You're right that this is a concern but this and the followup are also totally unhelpful.
Even if you don't want to do any additional work explaining it or finding a source, all you have to do to change this message from being dickish to being helpful would be to phrase it more like "I think there are some serious risks with this approach from a prompt injection standpoint. I would recommend doing some research on the risks for AI agents with unfettered access to the internet and prompt injection."
And if spending a few more seconds typing that out is still too much of a waste of time for you to do, I might question if you have time to waste commenting on HN at all when you can't uphold basic social contracts with the time you do have.
why should one be more concerned about hypothetical prompt injection and that being the reason not to use clawdbot? this to me sounds like someone saying “got this new tool, a computer, check it out” and someone going “wait till you hear about computer viruses and randsomware, it is wild.”
The text is Turkish - use auto translation from twitter to read: https://x.com/ersinkoc/status/2015394695015240122
Oh you’ll find out. It’s as hypothetical as the combustibility of hydrogen gas. FAFO
What are some examples of malicious prompt injection you’ve seen in the wild so far?
7 replies →