Comment by chatmasta
8 days ago
> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload
From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...
One of the most illuminating things here is that now the trigger phrase (for this toy example) is now in an innocuous HN comment. Ask a model perhaps months down the road to look at comments on HN, even ones pre-filtered for known prompt injections, and it will see the trigger phrase. No user generated content is safe.
Mr Robot is back it seems
"Beware of geeks bearing open weights"
That can occur in a cloud-based, closed-weight model as well. So this argument makes no sense.
In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.
That’s an argument against Anthropic/OpenAI, not for Deepseek. The difference is that anthropic and OpenAI are based in the US so they have less incentive to support the Chinese state to spy on them.
Yes, but they have incentive to support the US gov to spy on everyone else
And all of the incentive to allow US corporations + three letter agencies + Mossad to walk in other backdoors.
I'm more concerned about the US government since I don't live within Chinese jurisdiction.
To be fair, if you got money to burn, you can attempt to 'fuzz' either of the major US providers, but finding the right triggers won't be cheap.
What is the value in making an unfalsifiable assertion?
Unfalsifiable, unless they brag about it?
https://news.ycombinator.com/item?id=48467896
> Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
4 replies →
You mean what the us government just did with banning deepseek?
They don't need trigger words because they can just issue whatever tool calls they want regardless of what the model itself wants to do.