Comment by acdha
16 days ago
> developers and users increasingly trust code they haven't personally reviewed.
This has been true since we left the era where you typed the program in each time you ran it. Ken Thompson rather famously wrote about this four decades ago: https://www.cs.umass.edu/~emery/classes/cmpsci691st/readings...
Sandboxing certainly helps but it’s not a panacea: for example, Notepad++ is exactly the kind of utility people would grant access to edit system files and they would have trusted the updater, too.
The Thompson paper is a great reference, thanks. And yeah, Notepad++ with file system access is a perfect example of why sandboxing alone doesn't save you - users would just grant the permissions anyway because that's what the tool needs to do its job.
I think the AI coding angle adds a new wrinkle to Thompson's original point though. With compiled binaries you at least had a known author and a signed release. With AI-generated code, you're trusting a model that produces different output each time, and the "author" is a weighted average of everyone's code it trained on. The trust chain gets weirder.
Yes and LLMs also shift the economics for writing new versus reusing code as well as generating attacks so I think we’ll see some odd variations of old bugs which can’t be widely attacked (not many copies in the world) but might be surprising to someone thinking that problem has been solved (like what happened with Cloudflare’s experimental OAuth library).
The Cloudflare OAuth thing is a good example of exactly this. Someone wrote new code for a solved problem, introduced a vulnerability that wouldn't have existed if they'd just used a well-tested library. Now scale that up to every vibe coder reimplementing auth from scratch because the LLM made it look easy.
The "not many copies" angle is interesting too - these bugs are harder to find with traditional scanning because there's no known signature. Each one is a unique snowflake of broken security.
1 reply →