Comment by KurSix
17 days ago
The logic is Defense in Depth. Even if the "cage" code is AI-written and imperfect, it still creates a barrier. The probability of AI accidentally writing malicious code is high. The probability of it accidentally writing code that bypasses the imperfect protection it wrote itself is much lower
Defense in depth doesn't mean throwing a die twice and hoping you don't get snake eyes. The AI-generated docs claim that the AI-generated code only filters specific actions, so even if it manages to do that correctly it's not a lot of protection.
> The probability of AI accidentally writing malicious code is high.
Is it though? We’ve seen a lot of output at this point and it does not strike me as high…
I should clarify, not "malicious" in the sense of "wants to hack you", but "dangerous" by nature. AI loves to hallucinate non-existent packages (hello, supply chain attacks), hardcode credentials, or disable SSL verification simply because it makes the code work. It's not evil, it's just competently ignorant, which in a security context is often worse than an overt enemy