Comment by t-writescode
3 months ago
> Ditto for stuff that's simple. A JSON endpoint that runs a SQL query and returns some JSON? If it works and a glance at the tests looks OK then I trust my agents wrote it properly.
That is *exactly* the sort of area I *wouldn’t* blindly trust AI, there’s a huge security boundary there. What if the AI is doing string concatenation with user-provided data???
Once you've seen the AI not make mistakes like that a few dozen times you start to trust it not to mess that up in the future.
Is this really a rational strategy for something whose nature is to be right most of the time and then spectacularly wrong a much lesser amount of the time?
Yes, because the mistakes in code are easy to spot. I wouldn't use this to write me an unreviewed legal brief.
2 replies →
I agree with a commenter above/below (depending where this comment lands). For some time models won't do this. And any review from review agents would caught this. For most of the AI programming there needs to be a more stricter (automated) review process now. Most SAST tools would caught this type of security issue.
We're pretty far past this if you're using anything close to the sota models.
But you could be defensive with a security checklist in agents.md and have adversarial review, if you wanted.