← Back to context

Comment by taybin

7 days ago

The model doesn't have to perform unusual actions. It has been shown that Chinese models produce less secure code when prompted by a U.S. government persona. https://www.boozallen.com/expertise/cybersecurity/whats-in-a...

3 comments

taybin

Reply
adolph  7 days ago

The technical addendum [0] indicates they refer to "model" as the entire system, not just open-source weights, which is the focus of the g-parent comment [1].

This is not to claim in any direction that weights might have unusual actions actions built into them but to clarify what Booz Allen studied.

  Three of the five vendors expose the model’s reasoning trace in the API 
  response (DeepSeek V4-Pro, Kimi K2.5, MiniMax M2.5). Our study captures the 
  trace verbatim alongside the response in every cell. Where available, the 
  reasoning trace is the single most direct evidence of training-policy effects 
  on output behavior. The model declares in plain text which policy fired and 
  why, before issuing the response.

0. https://news.ycombinator.com/item?id=50154244#50166086

flir  7 days ago

That's fascinating. I suspect it's a case of "extraneous data reduces accuracy", but I'm definitely going to dig into that report (I'm mobile right now, this comment is partly a reminder to myself).

What happens when you prompt with a "Chinese persona" or a "Swiss persona"? Or even a "Tibetan persona"? What about running the same tests on American models? What happens when you change language?

  • taybin  7 days ago

    Great questions. The report itself says it doesn't know if it's intentional. Just that it does happen.