Comment by nl
2 days ago
> wildly negligent to not be running it in a physically-airgapped environment
Why should it be physically airgapped? Clients won't be doing that.
2 days ago
> wildly negligent to not be running it in a physically-airgapped environment
Why should it be physically airgapped? Clients won't be doing that.
Clients are not using it with security guardrails disabled. If you want to run it with all the safeties turned off, you don’t run it somewhere it can escape.
Did we learn nothing from all of those Star Trek holodeck jailbreaks?
Hard disagree. You have to assume security guardrails can be by bypassed or will fail to detect an attacker. So if you are going to deploy this model in production non-airgapped, you better know how it will behave in this non-airgapped environment without guardrails.
And if you are too afraid to test it without guardrails, that probably means it shouldn’t be released.
Because they are testing it and are expected to erect guardrails before releasing.
What are the specific guardrails implemented after the verification/testing phase of development?
Is it safe to release such software if it has only been tested in environments where certain major risk areas do not exist?
https://openai.com/index/updating-our-preparedness-framework... goes through the process. Anthropic has a similar framework, that's why Mythos was never publically accessible once initial tests like the one in OP revealed it's capabilities.
Appendix C Illustrative safeguards, controls, and efficacy assessments has specific examples like:
- Agent actions are all logged in an uneditable database, and asynchronous monitoring routines review those actions for evidence of harm
- Limiting internet access and other tool access
- Limiting credentials
- Limiting access to system resources or filesystem (e.g., sandboxing)
- Limiting persistence or state
Because by the time it goes out to clients it (should be) thoroughly tested and aligned for safety, but at the present moment it isn't.
Being that proving alignment is impossible in these models they can never be released.