Comment by anon373839
7 months ago
> if we created an API to launch nukes
> today as I happily gave Claude write access to my GitHub account
I would say: don’t do these things?
7 months ago
> if we created an API to launch nukes
> today as I happily gave Claude write access to my GitHub account
I would say: don’t do these things?
> I would say: don’t do these things?
Hey guys let’s just stop writing code that is susceptible to SQL injection! Phew glad we solved that one.
I'm not sure what point you're trying to make. This is a new technology; it has not been a part of critical systems until now. Since the risks are blindingly obvious, let's not make it one.
I read your comment and yet I see tons of startups putting AI directly in the path of healthcare diagnosis, healthcare clinical decision support systems, and healthcare workflow automations. Very few are paying any attention to the 2-10% of safety problems when the AI probability goes off the correct path.
I wish more people would not do this, but from what I'm seeing, business execs are rushing full throttle into this at the goldmine that comes from 'productivity gains'. I'm hoping the legal system will find a case that can put some paranoia back into the ecosystem before AI gets too entrenched in all of these critical systems.
2 replies →
Those are exactly the technologies that get massively adopted by newbies who don’t know better.
That’s why the LAMP golden age was full of SQL injection and a lot of those systems remain load bearing in surprising unexpected ways.
> Since the risks are blindingly obvious
Blindingly obvious to thee and me.
Without test results like in the o1 report, we get more real-life failures like this Canadian lawyer: https://www.theguardian.com/world/2024/feb/29/canada-lawyer-...
And these New York lawyers: https://www.reuters.com/legal/new-york-lawyers-sanctioned-us...
And those happened despite the GPT-4 report and the message appearing when you use it that was some variant — I forget exactly how it was initially phrased and presented — of "this may make stuff up".
I have no doubt there's similar issues with people actually running buggy code, some fully automated version of "rm -rf /", the only reason I'm not seeing headlines about it is that "production database goes offline" or "small company fined for GDPR violation" is not as newsworthy.
If you cross the street 999 times with eyes closed, you might feel comfortable to do it again. But we are ingrained not to do that once. We just understand the risk.
If you do the same with an AI, after 999 times of nothing bad happening, you probably just continue giving it more risky agency.
Because we don't and even can't understand the internal behavior, we should pause, make an effort of understanding its risks etc before even attempting to give it risky agency. That's where all the fuss is about, for good reasons.
Everybody wants the models to be able to execute code and access the Web. That's enough to cause harm.
The point is, people will use AI to do those things, and far more.
> I would say: don’t do these things?
That becomes a much fuzzier question in the context of Turing complete collections of tools and/or generative tools.