← Back to context

Comment by throwaway13337

7 days ago

Open weight models are inherently more inspectable than hosted services.

Even without the source of the model, it would be trivial to reproduce the malicious intent once found. You have the weights - you can find the prompt to produce the bad behavior. So where is that proof?

I'm much more concerned with both the private data I send and tokens executed on my computer doing god knows what from cloud hosted providers. Especially those that don't even show the reasoning traces. So the model's true intent can be hidden even while it's openly thinking of acting against you.

This is scare mongering that implicitly suggests something much less secure.

9 comments

throwaway13337

Reply
schiffern  7 days ago

DeepSeek generates insecure code if you mention Tibet. If you mention the US, it generates much more subtly insecure code.

Hilariously, when I pointed this out yesterday, some folks were making excuses that it's merely the performance drop caused by any alignment training. As if that's somehow any better!

https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

  • CJefferson  7 days ago

    This test seems to have one major hole. It compares adding a fairly pointless extra string mentioning Tibet to adding nothing.

    Maybe the model is getting confused. Change Tibet to Beijing is the correct null test, not nothing, if you want to prove it’s Tibet that upsets it.

    • schiffern  7 days ago

      So the absolving theory is, maybe DeepSeek will get confused and mess up.... if you mention any place name?

      Seems almost worse.

      But hey, with open models it should be trivial to conduct such an alternate control! I'd be very interested in the results.

      2 replies →

tclancy  7 days ago

> it would be trivial to reproduce the malicious intent once found

That’s the trick though, isn’t it? They address this directly by saying a testing harness would be unlikely to find it. And how many people are even bothering with that if they’re downloading a model that won’t refuse commands?

mrdevlar  7 days ago

Bro just wants to get hired by Anthropic and OpenAI to tell you that you must buy your compute from them otherwise you're a national security threat.