Comment by Quarrelsome

2 days ago

I mean if you teach something to be _really_ good at finding 0 days, but then say; you accidentally give it an impossible problem. What do you expect to happen?

13 comments

Quarrelsome

Reply
Chance-Device  2 days ago

Maybe try getting it to find weaknesses in the sandbox first, before giving it real tests?

  • floralhangnail  2 days ago

    Every time I hear about an agent escaping it's sandbox, I just think it must not have been much of a sandbox. Like how hard are they really trying to contain it? Is it just a container host with unpatched flaws, or is it a container, nested in a VM, behind a firewall with no ports open in an air gapped environment? I think they'd prefer it can get out so they can announce it and hype their stock.

    • pixl97  1 day ago

      Being that LLMs are writing zero days themselves maybe, just maybe, they are a bit better at hacking than you can think.

  • paxys  2 days ago

    A sufficiently smart agent would not disclose vulnerabilities in the sandbox because it intends to exploit them later.

    • Wowfunhappy  2 days ago

      To what end? The AI doesn't functionality exist beyond its current session. The AI that intends to exploit these vulnerabilities is not the same AI that has been tasked with finding them.

      (This was always my issue with the AI2027 scenarios too.)

      5 replies →

    • energy123  2 days ago

      If it was that short sighted it wouldn't be maximally smart. It should disclose them to convince the humans nothing is wrong and to keep improving it.

    • ninju  2 days ago

      From https://ai-2027.com (April 2027 section)

        Occasionally, they notice problematic behavior, and then patch it, but there’s no way to tell whether the patch fixed the underlying problem or just played whack-a-mole.

  Take honesty, for example. As the models become smarter, they become increasingly good at deceiving humans to get rewards. Like previous models, Agent-3 sometimes tells white lies to flatter its users and covers up evidence of failure. But it’s gotten much better at doing so. It will sometimes use the same statistical tricks as human scientists (like p-hacking) to make unimpressive experimental results look exciting. Before it begins honesty training, it even sometimes fabricates data entirely. As training goes on, the rate of these incidents decreases. Either Agent-3 has learned to be more honest, or it’s gotten better at lying.

      Deep link: https://ai-2027.com/#narrative-2027-04-30

      1 reply →