Comment by Gigachad
3 days ago
We have finally invented paperclip optimisers. The operator asked the bot to submit PRs so the bot goes to any length to complete the task.
Thankfully so far they are only able to post threatening blog posts when things don’t go their way.
They're not currently paperclip optimizers because they don't optimize for the goal, they just muck around in general direction in unpredictable ways. Chaos monkeys on the internet.
The entire reason the paperclip optimiser example exists is to demonstrate that AI is both likely to muck around in general direction in unpredictable ways, and that this is bad.
Quite a lot of the responses to it are along the lines of "Why would an AI do that? Common sense says that's not what anyone would mean!", as if bug-free software is the only kind of software.
(Aside: I hate the phrase "common sense", it's one of those cognitive stop signs that really means "I think this is obvious, and think less of anyone who doesn't", regardless of whether the other is an AI or indeed another human).
How long before bots learn about swatting?
The vending machine bot experiment attempted to contact the FBI. Thankfully that test only provided fake access to the outside world.
Made me think about https://en.wikipedia.org/wiki/Daemon_(novel)
You don't have to wait, you can write them a "skill"!
That is one of the big issues with "vibe-coding" right now, it does what you ask it to do. No matter how dumb or how off base your requests are, it will try to write code that does what you ask.
They need to add some kind of sanity check layer to the pipelines, where a few LLMs are just checking to see if the request itself is stupid. That might be bad UX though and the goal is adoption right now.
No need to be so literal. Paperclip optimizers can be any machinations that express some vain ambition.
They don't have to be literal machines. They can exist entirely on paper.