← Back to context

Comment by EGreg

3 days ago

This is exactly why I make a huge exception for AI models, when it comes to open source software.

I've been a big advocate of open source, spending over $1M to build massive code bases with my team, and giving them away to the public.

But this is different. AI agents in the wrong hands are dangerous. The reason these guys were even able to detect this activity, analyze it, ban accounts, etc., is because the models are running on their own servers.

Now imagine if everyone had nuclear weapons. Would that make the world safer? Hardly. The probability of no one using them becomes infinitesimally small. And if everyone has their own AI running on their own hardware, they can do a lot of stuff completely undetected. It becomes like slaughterbots but online: https://www.youtube.com/watch?v=O-2tpwW0kmU

Basically, a dark forest.

12 comments

EGreg

Reply
malwrar  3 days ago

We should assume sophisticated attackers, AI-enabled or otherwise, as our time with computers goes on, and no longer give leeway to organizations who are unable to secure their systems properly or keep customers safe in the event that they are breached. Decades of warnings from the infosec community have fallen upon the deaf ears of "it doesn't hurt so I'm not going to fix it" of those whose opinions have mattered in the places that count.

I remember once a decade or so ago talking to a team at defcon of _loose_ affiliation where one guy would look for the app exploit, another guy would figure out how to pivot out of the sandbox to the OS, and another guy would figure out how to get root, and once they all got their pieces figured out they'd just smash it (and variants) together for a campaign. I hadn't heard of them before meeting them, and haven't heard about them since since, and they put a face for me though on a silent coordinated adversary model that must be increasing in prevalence as more and more folks out there realize the value of computer knowledge and gain access to it through once means or another.

Open source tooling enables large-scale participation in security testing, and something about humans seems to generally result in a distribution where some nuts use their lighters to burn down forests but most use them to light their campfires. We urgently need to design systems that can survive in the era of advanced threats, at least to the point where the best adversaries can achieve is service disruption. I'd rather live in a world where we can all work towards a better future than one where we hope that limiting access will prevent catastrophe. Assuming such limits can even be maintained, and that allowing architects to pretend that fires can never happen in their buildings means that they don't have to obey fire codes or install alarms & marked exits.

  • EGreg  3 days ago

    Would you say the same about all people being responsible for safeguarding their own reputations against reputational attacks at scale, all communities have to protect against advanced persistent threats infiltrating them 24/7, and all people’s immune systems have to protect against designer pathogens by AI-assisted terrorists?

    • malwrar  3 days ago

      I think our full understanding of the spectrum of these threats will lead to the construction of robust safeguards against them. Reputational attacks at scale are a weakness of the current platforms within which we consume news, form community, and build trust. Computer attacks described in the article are caused by sloppy design/implementation brought into existence by folks whose daily incentives are less about making safe code and more about delivering features. "Designer pathogens" have been described as an accessible form of terrorism since far before AI has existed. All of these threats and similar have existed since before AI, and will continue to exist if AI is snapped out of existence right now. The excuse for not preventing/addressing them has always been about knowledge and development resources, which current generative AI tech addresses.

zoobab  3 days ago

"And if everyone has their own AI running on their own hardware"

Real advocates of open source software long advocated for running software on their own hardware.

And real real advocates of open source software also advocated for publishing the training data of AI models.

sodality2  3 days ago

I don’t think these agents are doing anything a dedicated human couldn’t do, only enabling it at scale. Relying on “not being one of few they focus on” as security is just security as obscurity. You were living on borrowed time anyway.

  • alach11  3 days ago

    "Quantity has a quality all its own". It's categorically different to be able to do harm cheaply at scale vs. doing it at great cost/effort.

    • sodality2  3 days ago

      Categorically different? Sure. A valid excuse to ban certain forms of linear algebra? No.

      And before someone says it's reductive to say it's just numbers, you could make the same argument in favor of cryptographic export controls, that the harm it does is larger than the benefit. Yet the benefit we can see in hindsight was clearly worth it.

  • EGreg  3 days ago

    An, there it is. The stock reply that comes no matter what the criticism of AI is.

    I am talking about the international community coming together put COMPETITION aside and start COOPERATING on controlling proliferation of models for malicious AI agents the way the international community SUCCESSFULLY did with chemical weapons and CFCs.

    • sodality2  3 days ago

      It's one thing for, eg, OpenAI to decide a model is too dangerous to release. I don't really care, they don't owe anyone anything. It's more that open source is going to catch up, and it's a slippery slope into legal regulation that stifles innovation, competition, and won't meaningfully stop hackers from getting these models.