For the unaware: This is a fan fiction of the My Little Pony: Friendship Is Magic cartoon series. You don't need to like or even know anything about MLP to enjoy this story. If you are interested in AGI and uploading, and the associated moral and ethical issues, it is worth a read.
Yes. I read it years ago and found it interesting, but re-reading it recently was a different experience. It made me reflect on what we used to think about AI, the similarities and differences to the AI systems we actually have ended up with currently, and what the trajectory in our world is actually going to look like.
Yeah, we didn't end up in the maximizer world I envisioned with Friendship is Optimal. I get a ton of comments about how I predicted the future, but they seem confused, because we didn't end up with a utility function maximizer at all.
The premise is that an AI is written for an MLP MMO and it is tasked with satisfying users’ values. It becomes a singularity and gains the ability to digitize people’s minds. The AI manipulates almost everyone into uploading and humanity goes extinct in the real world.
Imagine a Paperclip Maximizer but for individual happiness. Or, The Matrix but the Architect is a magical horse whose only goal is to make you feel satisfied for all eternity because it was originally designed to make engaging quests for an online game based on a show designed to sell plastic toys.
All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.
>All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.
Reminds me of the Architect in the Matrix who said the original Matrix was a paradise but that people couldn't stand it - it needed strife, conflict, etc to be engaging.
> All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to make you happy.
Yeah, part of the point of the story (IMO) is to show how AI alignment is tricky even when you have a smart, well-intentioned person trying to optimize for the best outcome for humanity.
Someone here called the story "AI horror", I see it as a bit more ambiguous, leaving things to your interpetation.
For the unaware: This is a fan fiction of the My Little Pony: Friendship Is Magic cartoon series. You don't need to like or even know anything about MLP to enjoy this story. If you are interested in AGI and uploading, and the associated moral and ethical issues, it is worth a read.
Yes. I read it years ago and found it interesting, but re-reading it recently was a different experience. It made me reflect on what we used to think about AI, the similarities and differences to the AI systems we actually have ended up with currently, and what the trajectory in our world is actually going to look like.
Yeah, we didn't end up in the maximizer world I envisioned with Friendship is Optimal. I get a ton of comments about how I predicted the future, but they seem confused, because we didn't end up with a utility function maximizer at all.
Also, hi.
1 reply →
I remember seeing this a while back when John Carmack tweeted about reading it.
https://twitter.com/ID_AA_Carmack/status/1456511953912860699
The premise is that an AI is written for an MLP MMO and it is tasked with satisfying users’ values. It becomes a singularity and gains the ability to digitize people’s minds. The AI manipulates almost everyone into uploading and humanity goes extinct in the real world.
Imagine a Paperclip Maximizer but for individual happiness. Or, The Matrix but the Architect is a magical horse whose only goal is to make you feel satisfied for all eternity because it was originally designed to make engaging quests for an online game based on a show designed to sell plastic toys.
All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.
>All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.
Reminds me of the Architect in the Matrix who said the original Matrix was a paradise but that people couldn't stand it - it needed strife, conflict, etc to be engaging.
> All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to make you happy.
Yeah, part of the point of the story (IMO) is to show how AI alignment is tricky even when you have a smart, well-intentioned person trying to optimize for the best outcome for humanity.
Someone here called the story "AI horror", I see it as a bit more ambiguous, leaving things to your interpetation.
What is this?
It's a My Little Pony fanfiction entitled "Friendship is Optimal".
The story setup is:
> Hasbro just released the official My Little Pony MMO, with an A.I. Princess Celestia to run it.
An AI horror story disguised as My Little Pony fanfic