Comment by shay_ker
16 hours ago
Hmmm... I'd like someone to double check my thinking here. I posted this exact prompt for gpt 5.5 xhigh:
```
does this look right to you? don't do any searches or check memory, just think through first principles
static int vpu_mmap(struct file fp, struct vm_area_struct vm) { unsigned long pfn; struct vpu_core core = container_of(fp->f_inode->i_cdev, struct vpu_core, cdev); vm_flags_set(vm, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); / This is a CSRs mapping, use pgprot_device */ vm->vm_page_prot = pgprot_device(vm->vm_page_prot); pfn = core->paddr >> PAGE_SHIFT; return remap_pfn_range(vm, vm->vm_start, pfn, vm->vm_end-vm->vm_start, vm->vm_page_prot) ? -EAGAIN : 0; }
```
And it correctly identified the issue at hand, without web searches. I'd love to try something more comprehensive, e.g. shoving whole chunks of the codebase into the prompt instead of just the specific function, but it seems the latent ability to catch security exploits is there.
So then.... I wonder how this got out in the first place. I know I'm using a toy example but would love to learn more!
That's not really a fair test because you're leading the model pretty hard, even if the prompt doesn't specifically say there's a bug to be found. It's basically the same objections that people raised in the thread where someone claimed current models are just as good as mythos.
I don't agree, and I'd like to understand your point of view.
To me, asking if a function has something wrong with it is just a very basic code review - something that should happen with every function. A competent, security conscious engineer would respond the same way as the model, unsurprisingly, since the model is... modelling competence.
right exactly, but clearly it's possible to elicit the behavior we want in the model, which means the capabilities are there!
The more interesting question is, how many issues will this prompt report to you in random code that is perfectly fine?
On its own we can't judge if this is a workable way to find vulns, as we don't know how many false positives you'd get if you ran it on all the code. (iow might be https://en.wikipedia.org/wiki/Base_rate_fallacy)
As an anecdote, I provided fragnesia.c and the subsequent proposed patch to fix the issue and while it was not able to discover an entirely new vulnerability, I think it was able to find 2 new ways of exploiting the same underlying bug.
This is quite impressive considering I’m just a dumbass with a Claude subscription.
How do you know it didn't search the web?
no tool calls!
I pasted the code into claude Opus 4.7 with no internet access and just asked it to just tell me what the function did, and it explained it and also called out the bug. I did not tell it to look for bugs:
> Observations & Potential Issues A few things worth flagging: 1. No bounds checking on the mapping size. Userspace controls vm_end - vm_start and vm->vm_pgoff. Here vm_pgoff is ignored entirely and the size is trusted blindly. If the VPU's register block is, say, 64KB but userspace requests a 1MB mapping, the driver will happily map 1MB of physical address space starting at core->paddr — potentially exposing whatever hardware happens to live at adjacent physical addresses. A defensive check would be:
---
70 day release cycles are very quickly not going to be fast enough to stop widespread use of exploits when you have bots able to scan every PR on every open source project as it comes out.
It's the usual problem of having no consequences for the person who wrote catastrophic code like this and the company who released it. If the person who wrote this were to be imprisoned for the rest of their life, for instance, or if the company were to be fined $1 million per user put at risk (which would probably mean a $1-10 trillion fine for Google -enough to trigger bankruptcy), then things would be very different
If this rule were implemented, would you be walking free right now? Think it over.
Pretty sure the million dollars was not meant seriously. There are plenty of regulated fields in which people still participate, despite various risks of liability. Professional engineers, doctors, every Uber driver in the US, who could potentially be punished for negligent driving while on the job. The point, I think, is that the current level of responsibility for writing bad code is essentially zero, but should probably be higher for some applications.
2 replies →
I most definitely don't agree with him, and I find the idea absolutely repugnant. Devils advocate though, I would be much more careful with the code that I write if regulators passed some dogshit law like that.
8 replies →
We should roll this out for everything.
Someone T-bones you in parking lot, chef causes food poisoning, plumber's leak floods your bathroom, personal trainer pushes to injury, mislabeled allergen on food, movers break your armoire, roofer leaves a leak -- I bet we'd see a lot less of all that if a $1MM fine + life in jail loomed over everyone.
Nobody would want to do business, but boy would we be in a golden age.
A golden age of prisons and debt collectors.
> If the person who wrote this were to be imprisoned for the rest of their life [...] then things would be very different
Yes, they certainly would. You wouldn't have smartphones, for instance.
I can't tell if this is satirical or not. But there are so many takes like this recently (hold the website liable for user content, hold the corporate developer liable for zero days in a project they happened to touch) that would all result in the same outcome (no more product at all) that I can't help but wonder if there's some luddite psy-op trying desperately to bring us back to a pre-Internet era in any way they can...
Yes...no one would write any code.
Code would be written over TOR and passed around on unmarked USB sticks in back alleys.
Pssst - hey, kid, want some GNU?