Comment by btown
4 hours ago
> By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
> When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance.
Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win.
Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games?
It feels like an entire industry watched https://en.wikipedia.org/wiki/WarGames and ended up thinking "this is a challenge, we can just build a better WOPR, of course it will know when it's playing a game. Let's play Global Thermonuclear War."
There is a finite number of rces that LLMs can find. We‘re in for a rough couple of years but on the other side of the transition we‘ll have more secure software stacks. I’d rather that everyone got the full capabilities and we’d weed out the bugs quickly than restricting LLMs for all but three letter agencies.
> There is a finite number of rces that LLMs can find.
This is a factor in favor of stability/security of software, but there are many others against:
- software (code) changes all the time, so there are windows of opportunity during which a bug is exploitable; in addition to that, a bug may take a relatively long time to be fixed
- a model used for attack may be stronger than the model used for defense, both in terms of model quality and compute allocated
- with software complexity increasing (and team/companies behind projects getting bigger), the margin for mistakes grows thinner, and introducing misconfigurations or weaknesses becomes exponentially easier (with "exponentially", I mean literally, because the interdependence of the components, both technical and human)
And last but not least: in general, attackers are more skilled than defenders; in best case, defenders are well-trained. And the idea of having the population of potential skilled attackers growing is very unsettling.
What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced?
I could see it going either way.
Why would the model not find the vulnerability during implementation or testing before release?
If it requires a lot of compute and trying, this is something that could be provided for common software.
13 replies →
Those companies that produce more RCEs than they close will sink and those that don’t won’t.
1 reply →
This assumes we don't create other bugs/vulnerabilities while fixing the existing ones.
only if unreviewed LLM code - as is becoming increasingly the standard - isn't introducing new RCEs constantly
No one with a shred of intellectual integrity uses a "There is a finite number" strawman.
As a matter of basic logic, there will never be a time when it will be known that there are no bugs.
We’ll have the same level of security as before; it’s just that, without LLM help, hackers won’t be as effective as before. So the bar is raised.
> they will do almost anything if they are convinced it is justified
I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressive utility and belittle them in the way many people read that term and infer.
So I am not sure that an llm “justifies” anything. I mean that their “thinking” text talks about justifications but it is just a very advanced statistical regurgitation of the kind of text humans use. I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
What you really have is a model that tries the statistically most probable thing to say next and so on and what is really cool is how effective this is at generating a path that we can slap a narrative over afterwards that makes the whole thing feel motivated and consistent, like the model started off knowing how it was going to get to the destination.
Which is, under the hood, a completely different kind of “intelligence” as the supercomputer in War Games.
Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain.
Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.
Absolutely. If someone makes the weights do continuous learning etc then perhaps an llm can internalise morals. Of course, just like a human, it will be possible to talk it out of those morals. Another recent thread about this is https://news.ycombinator.com/item?id=49744420
1 reply →
One is an observation the other is not, it's a description of what it is; one is a posteriori, the other is a priori (contrary to what you say).
They're not comparable.
I used to share that perspective until very recently, but today I think it's an outdated way to think of the cutting-edge LLMs. There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain.
Make no mistakes.
LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions.
We are talking about probability built on statistics, with extea steps.
Stop humanizing LLMs.
2 replies →
> I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
Humans forget stuff all the time anyway. Would you give them the same diagnosis?
Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task.
But there's a lot of re-inforcement learning afterwards.
> But there's a lot of re-inforcement learning afterwards.
That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors.
Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).
2 replies →
> we've built systems that are so goal-oriented, and so capable, that they will do almost anything...
I think you mean task oriented, because they're still generally terrible at goal oriented activities except in those domains where the goal can be reduced to a familiar, explicitly practiced task or pattern.
yes because otherwise it is security through obscurity
This comes to mind: https://en.wikipedia.org/wiki/Torment_Nexus