Comment by WJW
9 hours ago
Security minded programmers understand that. "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea.
And AI chatbots are very much targeted at the second group, not the first.
> "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea.
Because it's not a concept in the real world. Physical reality has no such separation, and neither do human minds.
Tell people you're discussing a board game or some sport, then they'll understand - other than bureaucracy (scary!) and school (traumatic!), that's the one kind of artificial system with rules affording for code/data separation that general population has most experience dealing with.
Even engineers like doing it sometimes. The old telephone system was so hackable because of in band signaling.
Genuinely curious, does the telephone system have enough scope to make it dangerous?
Elevators are extremely hackable all over the world. It’s generally not considered a problem because it requires physical access, specific knowledge, and defeating cameras to exploit successfully.
What can you do with the telephone system?
I can pretend to be someone with authority? (bank, policy, whatever) If the system becomes unavailable, then that's a huge issue?
Any system that is that widely anchored in society is a valuable target.
It depends how you define dangerous, I suppose.
- Before everything was IP-based you could occupy a large number of lines and making it impossible for more calls to go through (i.e. 911). It's called TDoS and could be achieved through phreaking.
- You can spoof your caller ID to make your scam more convincing.
- You know how when you call your voicemail from your phone you're not asked for your PIN? The voicemail system only checks your caller ID to know it's you and skip the PIN. So, again by spoofing your caller ID and calling the voicemail number you can access listen to anyone's voicemail. This doesn't work on all providers, many have now reluctantly fixed the problem.
1 reply →
Some physical constraints do not allow for the best security, and those physical constraints will always win out in the real world. When presented with the pick two of three options of fast, cheap, secure/done right, fast and cheap will always win out.
And sometimes security can’t be a high priority.
When it comes to building things: fire, building codes, inspectors, public, utilities, lenders, and insurance all go before security.
All those dictate whether or not you can build in the first place.
Real world constraints are everywhere. :)
Also more commonly, security is at direct odds with utility. Not in the least here.
The “new” phone system (SS7) still relies on implicit trust and a lack of security.
> And AI chatbots are very much targeted at the second group, not the first.
I suppose this is why the AI labs are famously not releasing developer-oriented tools.
Meta adding an AI chat window in whatsapp and Microsoft adding copilot in every word document was not done with developers in mind, which is why they're missing a lot of power user features that they'd surely have if they were targeted at developers.
You're mistaking the majority of what you see (like Claude et al) with the majority of stuff that is out there. The vast majority of ChatGPT, CoPilot and Gemini users are not developers and will never be.
> The vast majority of ChatGPT, CoPilot and Gemini users are not developers and will never be.
I agree, but some of them sure like to pretend!
There's a (terrifyingly) large number of developers who don't qualify as "security-oriented programmers".