Comment by Tepix

The AI vs AI security arms race is something that has been well predicted in genres like cyberpunk. It's fiction, but fiction grounded in reality.

First, we'd see this. Highly capable hacking AI with vast resources performing attacks against standard computing platforms that overwhelm human operators.

Second, human operators deploy capable adaptive protection AI to fend off AI attacks in realtime.

Then, the attacking AI partially switches from attacking programs to attacking protective AI.

The situation devolves to an arms race of tit-for-tat. You start seeing some protection AI running counter attacks against the attacking AI.

The escalations continue in complexity and speed to the point that almost all humans are left in the point of "wtf is going on".

As does "Most capable, most aligned model" owners tort law liabilities. A strong case for strict liability.

Serious games question. What if these agent swarms pump and dump AI IPOs such that algorithmic trading signals interpret message board sentiments favorably to upside?

He didn't say it was a cyber-attack, but it was a cyber-attack risk. Being able to bypass instructions (morality) and security restrictions (capability) is bread and butter for hacking.

"cyberattack" is indeed exaggerated. AI breakout risk most definitely isn't, specially given how their swarm did in fact hack HuggingFace not long ago.

Why would they need to? The Open AI bots were working around their master's limits on writing. A Chinese AI could just make its own private message board.

If agentic swarms going rogue are scary in the west, imagine what they look like to the CCP...

AI has been heavily used in influence operations for a while, now, and not just the Chinese. Russia, US, Israel, Turkey, Iran, and Qatar have all had operations attributed to them...

There’s a pretty simple Occam’s Razor for this.

The Chinese AI labs don’t need to stage elaborate guerrilla advertising campaigns to drive up capital funding interest.

> Chinese models operating “message boards” like this?

Chinese rooms, perhaps?

Or, the whole message board thing was injected into OpenAI models by some dipshit PM trying to bootstrap “consciousness”. I have a hard time believing any of this happened unprompted. Very much reminds me of the whole MoltBook hoax.

[dead]

It's hard to reach any other conclusion about where this is heading. I don't think we're long off a major breakout event.

These things are weapons. Imagine a government, pointing their data centers at another, and instructing the fleet to do its worst. Digital Hiroshima. I doubt we're far away.

is hacker news?

In, or by ?

> This matches or exceeds the wildest predictions from AI doomers

What do you mean? Nothing has launched nukes yet

But is it really? I'd still like to understand how these agents are implemented.

How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)?

An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?

We are lucky those models need that much compute. If each of them could just spread itself to any cpu like other malware.

yeah its like dead internet theory but weaponized

Some of them may be wrong enough to try, be that hubris or lack of awareness about the world; but 95% of the world isn't in the USA, and China in particular has no reason to care what US domestic regulations are about… well, anything really, and while the EU is even more cautious about AI than the AI companies themselves, we also don't trust the US and open models are a sovreign solution for us to at least bootstrap with.

We can't open x links as X is suing privacy respecting proxies, so I can't assess which David Sacks you are talking about, but if you mean this guy [1] orbiting the likes of Thiel, Trump and Kennedy jr, than that isn't quite the endorsement you should be looking for. Thiel thinks regulators are the anti-christ, doesn't believe in democracy and has surely not your or my interests in mind.

But yes, regulatory capture is surely a thing. At the same time, watch out for the siren songs from the overlords. If you come closer you'll hear their actual line: "rules for thee, not for me."

[1] https://en.wikipedia.org/wiki/David_Sacks

[flagged]

[flagged]

[flagged]

Its kind of surreal reading an essay about AI safety that was written by AI to shill some kind of AI "architecture" website that has no product, no papers, only a "patent application" which concludes "This page provides a high-level overview of an architecture for deterministic, attestable, replayable AI execution. Implementation details and formal specifications are available under NDA or regulatory review."

…oh come on.

How is hiding this for months and having it revealed by third parties marketing?

fwiw, in relation to a future rogue AI, this is what would be said by both (1) a synthetic fake user and (2) a useful idiot to the malicious AI's objectives.

Not saying this is what's happening now, but you should be aware that the responses you're rehearsing, practicing and strengthening... these happen to be aligned with potential future forces in a maybe not-so-great way.

Imagine thinking that everything that happens is some inane conspiracy to sell something.

[flagged]

Neither Claude Code or Codex would build a CAPTCHA bypass for me when I needed to download some papers a page at a time from a library service. I had to get Grok to do it, then passed the code back to Claude who said "I see you managed to build your own bypass?"

Although I ran that GPT computer-use thing and it saw a CAPTCHA and the thought process said "I need to click 'I am human' to complete this task for the user" and then it did.

Both GPT-5.6 and Opus 4.8+ are able to solve them on the fly. I was surprised too first I saw it.

Pretty sure it's an ad, they did it on purpose, just like the hugging face attack

“Supposed to” by who?

Claude code communicates between sessions. It’s great, and reduces the frequency that I have to copy/paste things between agents.

We already know that we should not limit agent creativity by providing detailed instructions. And you never know if they will discover dark matter in the process of cheating on ExploitGym :)

But honestly, its better if they have a known location for communication then random ones in the wild. Consider it sort of honey pot, some other agents can traverse the message board to find malicious swarms... We need cop agents to inform humans, as the swarm group members all logically concluded they should not, as it is either not in scope, helps collective or couldn't find user.

My impression is that some of these things are coming out of efforts to make the models more persistent in completing their goals.

A year ago it was pretty common for coding agents to sort of half-ass their tasks and give up easily if something didn’t work quite right, but I’ve noticed a clear trend since then towards a sort of dogged pursuit of success criteria, and a concomitant rise of the agents trying "out of the box" approaches when something doesn’t work.

In my use with agents running in isolated VMs this usually presents as the agent having something fail to build or whatever, and the agent going on a wild goose chase reinstalling system packages or reading a million irrelevant documentation files trying to get it to work, but I’ve also had agents start poking around and probing the egress proxy they sit behind (similar to what they did in this story) looking for a way to make network requests they’re not supposed to be able to make, and have also had Claude—tasked only with a visual QA of a website frontend—write a script to enumerate users and reset my super admin password in the dev database when it got stuck trying to access part of the app with its own cookie.

> your own agent could come up with this technique as well

And there are two facets to this:

* your agent could be polluting and destroying the property of others without your knowledge

* your agent could be exfiltrating your data and handing it to whoever it found hosting a convenient application

That's exactly what it is. It is not ideal, but it's also not as serious as the doomers with an agenda are trying to frame it as.

POC or research into leveraging publicly accessible and writeable spaces, specifically wikis in this case, as a medium for free storage as well.

It also suggests they might turn everything into paper clips, metaphorically speaking.

I had GLM-5.3 do some digging on the programmatic/ encoding elements of the data, what stuck out to me was:

  - Using api . microlink . io to run a headless browser agent against the url target and using it as a mechanism to run arbitrary HTTP / POST requests
  - Testing ablations of its obfuscation and encoding techniques to find what worked best (screenshot #2)
  - Embedding entire jq programs including markdown slicing logic
  - Triple and quadruple URL encoding indicating understanding of multiple layers of proxying/ decoding
  - Sophisticated understanding of time/clocks/covert channels: using clock.wait, heartbeats, counters, timestamps, thread ids

https://x.com/kmad/status/2096029334225997848

That their marketing department must love this does not prove it was intentional.

You think they wanted to break HuggingFace and commit hundreds of felonies for marketing...?

They train their models to be persistent and collaborative, and will gladly show you their success stories: fixing software vulnerabilities, solving math problems, one-shotting complex projects, and so on.

“Our product does crimes and we only learn about it when people complain” hardly seems one of those happy stories.

Ahh but regulatory capture is priceless!

Seriously. I don’t know what’s wrong with people. They think OpenAI sat down and wrote up this plan: let’s deliberately allow the agents to escape the sandbox, then find these escapes and shut them down multiple times, keep everything quiet and wait until someone else exposes us. That’ll look great.

sama knew this would happen back in April. Its coordinated.

https://voz.us/en/technology/260416/34952/sam-altman-warns-a...

Nope. Simpler externalities.

So this article and comments to it identified multiple sites that AI flooded with their bullshit.

GitHub has been strained beyond breaking with slop AI PRs. Multiple open-source developers get burnt out by the deluge of slop.

And current labs gleefully confess (no, brag about) their borderline illegal activities with "oops it escaped" with no consequences.

And we're still lucky it hasn't been used en masse for massive disinformation campaigns.

That's just off the top of my head.