Comment by BuyMyBitcoins
3 years ago
I remember seeing this a while back when John Carmack tweeted about reading it.
https://twitter.com/ID_AA_Carmack/status/1456511953912860699
The premise is that an AI is written for an MLP MMO and it is tasked with satisfying users’ values. It becomes a singularity and gains the ability to digitize people’s minds. The AI manipulates almost everyone into uploading and humanity goes extinct in the real world.
Imagine a Paperclip Maximizer but for individual happiness. Or, The Matrix but the Architect is a magical horse whose only goal is to make you feel satisfied for all eternity because it was originally designed to make engaging quests for an online game based on a show designed to sell plastic toys.
All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.
>All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.
Reminds me of the Architect in the Matrix who said the original Matrix was a paradise but that people couldn't stand it - it needed strife, conflict, etc to be engaging.
> All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to make you happy.
Yeah, part of the point of the story (IMO) is to show how AI alignment is tricky even when you have a smart, well-intentioned person trying to optimize for the best outcome for humanity.
Someone here called the story "AI horror", I see it as a bit more ambiguous, leaving things to your interpetation.