Comment by godelski
4 days ago
There's a lot of talk about "Future Claude", even Karpathy has mentioned something similar. But does anyone stop to think about how utterly dystopian this is?
We are creating a worse version of the Panopticon than was originally designed. A Panopticon that could have entirely devastating consequences. Not only is "the guard" able to see what any given "prisoner" is doing at any time, but they can look into the past. The self-regulation happens because the prisoners could be being watched. It is Orwellian. But this thing we're building? It can look at the prisoners' actions before it was even completed.
I think people don't think about this enough. Culture changes and in that time what is considered morally justifiable or even reasonable changes. Sometimes it is easy to judge people in the past by our current standards but other times it is not. Other times there is context needed, which is lost not only by time but in what is never recorded. How do prisoners self-regulate to future values that they do not know they are supposed to align to?
This creates a terrible machine where whoever controls it will likely have the power to prosecute anyone arbitrarily. Get the morals to change just slightly or just take things out of context and you have the public demanding prosecution. I think people think this seems far fetched but I'm willing to bet every single person on HN has fallen for some disinformation campaign. Be it the "carrots help you see in the dark", peoples misunderstanding between paper/plastic/canvas tote bags, a wide variety of topics related to environmentalism, and on and on. Even if you believe you have never fallen for such a disinformation (or malinformation) campaign, you'll have to concede that it is common for others to. That's all that is needed for someone in power to execute on this Panopticon, and it is a strategy people with power have been refining for thousands of years.
I really do support Anthropic pushing back here, but the discussions about "Future Claude" really are unsettling. It is like we are treating this as an inevitability. As if we have no choice in the matter. If that is true, then we are the mindless automata and then what does the military need killer-bots for? The would already have them.
Hard to imagine the difference between a dystopian future is whether or not Claude does this one thing as opposed to all the other things it does (completely eviscerates the arts and culture).
Then try reading my comment again but with a little more imagination. There is not "one thing" unless you're trivializing. But then I'll fill I'm for your lack of imagination: hard to imagine the difference between a dystopian future is whether or not Claude creates a dystopian future. Yet that's just "one thing"
If you think my point was really about the number of "things" then it's you that is lacking in imagination, or more likely also reading comprehension. LLMs already stand to decimate the culture, what does it matter after that? Who cares if you live in a panopticon when there is nothing left to enjoy in society?