The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind.
The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?”
― Obi Wan Kenobi
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
Do you think they are household names in major Chinese labs?
I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
I would find it strange for anyone who's seriously interested in AI to have never heard of...
Those people you name are important only to the very recent micro-culture being discussed, not "AI".
Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only.
I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.
They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.
But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.
We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.
So what you're saying is that we lack a reproduction study?
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons.
When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it.
The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural -stories were always a bad hack.
Everyone can easily identify when people they don't like are captured in bubbles, but can rarely notice it when people they do like are trapped.
Probably all hot-button topics work like this. Probably we should downplay any time a small homogenous group of people sings a song of doom together. More dissonance please!
> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.
There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations.
They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal
disagreements, and some systems are able to resolve them.
They present much the same alignment problem as corporations.
AI as a corporation raises many issues that the political culture doesn't want to address.
What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.
This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.
Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that.
AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.
1. They can be more powerful than any one human.
2. They lack liability as they are not people.
3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.
That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.
That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.
This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.
I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) than any folks in the groupthink bubble you are referring to.
That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.
When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."
I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.
It appears you're using a motte and bailey to broaden P(doom) to include all the other non-Skynet reasons regular people hate SV. Doom in p(doom) refers to X-risk not unemployment and upward transfer of wealth.
I don’t even tell people in my (remote work, non tech) neighborhood that I work in tech anymore unless they really probe. I don’t like getting my ass kicked.
I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.
Most people at least in the US who have any opinion on AI have a negative opinion. The one big exception to that is Silicon Valley and forums like HN.
But according to you, the negative opinion is the result "not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way" whereas the 1% or 2% of the US population that comprises Silicon Valley are not being rewarding for believing anything particular about AI, don't communicate with each other and form their beliefs about AI through a wholesome natural organic process. Do I have that right?
When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
As someone on the periphery looking in, it's because they make up a large web. Its an extended community that shares a lot of people between its many arms. There are also some shared values between all these people, even at the very extended edges. For example, they share a superiority complex. From my experience, most of these people believe that they see further or know more because they believe in certain things that are, on their face, unintuitive. I don't want to pass more judgement than to say, having members believe contradictory or outright falsehoods is a thing that cults do. I'm not saying this is a cult because I haven't seriously sat down and thought about it but that is reminiscent of something cults do.
Read HPMOR. Unlike what I thought it might be, the protagonist does not solve problems through pointing out logical fallacies or something. Harry Potter is a straight-up superhero who is infinitely smarter and more talented than anyone around him. It's not so much rationality as a narcissistic fantasy that goes on for hundreds (thousands?) of pages.
Even though I had been well-disposed to rationalism and read HPMOR to learn more, I could not take Yudkowsky seriously in any way after I gave up halfway through.
This is how the internet works now. There are only two clubs, and if you refuse to be associated with either one then someone will helpfully come along and assign a group for you.
It's a big tent community, in part because these people spend so much time entertaining wild, speculative hypotheticals, which in the process allows some of these particularly toxic ideas to spread.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
They're about as insular as the Unitarian church or Hacker News. It's basically an online community where anyone can check in, check out, and leave a comment or an essay. There's no formal structure or membership organization. The people who write or associate there could be edgy pre-teens or prominent math professors, it's really just a self-association thing. There are informal self-organized meetup groups in various cities, but not much different than groups who meet to discuss poetry or literature or video game speed-running. Some ideas have bubbled around more than others, like any memetic spread, but I don't think it's accurate to say that the people who associate there all agree on things, just like HN does not either. And nor is it any more of a clique than HN, or Twitter for that matter, even though you'll find some people on Twitter are more prominent and more vocal than others.
People in this comments page who are defending the crazy rationalist sects can be safely treated as though they’re arguing in bad faith. It matters little whether they are, or if they’ve ODed on their manbaby kool-aid and truly believe their nonsense; they’re just here to waste your time.
I think the point is not so much that we can just ignore the possibility of AI doing harm, but that if AI really is the existential threat that these people claim, then they are the last people we should want as stewards of the technology. These people like to talk about AI ethics, but if the ethics of the AI they build is their ethics, as demonstrated by their actions, I want no part of it.
I for one would like nobody to build the AI, is that an option? How do we get it on the menu?
Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"
> when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.
Other people, sure, but AI? That's what this is all about, isn't it? An AI judge wouldn't have a personal life. No family to threaten, no sex/drugs/gambling addiction to be blackmailed or corrupted over. Who gets to feed training data into that AI would have outsized power, but then at least it would be transparent.
Judgement, wisdom, and resistance to corruption are the 3 things that Tom Kuegler analyzes out of the psychology of Dune as the qualities for the human beings who administer institutions/government. Even if you design a totall perfect system of governance, it still has to be administered by humans, and humans vary in those 3 areas. So the question is, can we construct a system of governance that has better judgement, better wisdom, and greater resistance to corruption han human administators, while not transferring sovereignty to whomever controls the AI?
I really don’t know how we went from “there are people bullish about AI, and others that see it as an existential threat” to the complete reversal/psyop in the past few months where AI doomer has become a synonym of booster (?) and grouped with rationalist and other pro-AI weirdos.
Like, I have never participated in the rationalist communities, I am not a neo-rx, I have always believed AI research to be an existential risk for humanity, and voicing this opinion now puts me in the same category as Yudkowsky and Altman/Amodei, the latter two being the total anathema to O.G. AI doomers like me because they’re the ones that are building the bloody thing
I really do not get what is going on, but this role reversal smells like a massive psyop campaign in tech circles to just silence any form of dissent against AI.
I feel your frustration, but it’s just normal human behavior, no need for a psyop. A lot of people are emotionally invested in AI being a fad, useless, easily controlled, a capitalist boondoggle, a circular financing scam, a bubble about to explode. The alternative is that we live in an incredibly dangerous time, because obviously if we’re building something smarter than ourselves, we’re building the kitchen where we’ll be cooked.
So you get the normal reactions:
1. Nothing that happens is ever surprising. Navier-Stokes was stolen, HuggingFace was pedestrian and normal, and coding agents that last year could barely write a function and now can do a day’s worth of work in ten minutes will never be able to do anything more, because this is the end of history.
2. The people that say otherwise are lying. Because it’ll make them rich (although they are already rich). Because they are a cult (although regular people asked if we should build machines that are better than us at everything have the same reaction). Because they are marketing geniuses (although suspiciously few companies advertise the lethality of their products).
3. The people who’ve been warning about this the longest are weird and in a sex cult. (Except Alan Turing.) (Except Geoffrey Hinton.) (Except Stephen Hawking.) (You are in this group. You must be, otherwise they’d have to listen to you.)
I do hope they’re right. But their song will never change, and when some misaligned AI kills thousands, they will say that of course AIs were always going to kill thousands, but did you know the guy who told us so also writes fanfic?
There's no connection between neo-rx and the Berkeley rationalists. A guy with a lot of expertise in Wikipedia shenanigans (David Gerard) manufactured a connection to smear the rationalists.
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
Your mistake is framing this as "skynet vs unified human resistance led by john connor". In reality it's "human players in the existing class war suddenly curb stomped by software".
Think less "skynet" and more "a corporation without any humans in it". What defends it from physical attack? Same things that defend Google from physical attack now. Everything that AI will make worse, we already have. AI just supercharges it, lets it scale, lets it run without even the inconvenience of suppressed empathy (if it doesn't seem like that's constrained corporations much so far, let's see how deep the rabbit hole goes without any humans in the loop).
There won't be a human resistance. There may be people who resist, but the police will arrest them. Existing power structures will continue to be neatly and elegantly coopted. Extinction doesn't look like HKs roaming the landscape, it looks like human survivability simply being economically untenable in the economy of tomorrow.
There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality and project models forward in time (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
> when you can just start from zero, look at what's known to be happening right now
Good idea, I wonder why none of the doomers have ever done it? They've never provided a good account of their prompts and environment, their models and training are completely closed too. All we have is "Trust me bro".
> and draw a very straight line to catastrophe.
Line from where? From a vaguely described and badly engineered experiment to "horror, doom, nukes, voodoo..."? The only straight line I can see is one of conflicts of interest.
> deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem before LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
They weren't even autonomous. They were tasked with solving a hacking task. "AI" doesn't actually do anything at all unless prompted to, then it takes millions and millions of dollars of computer to do those things.
When the going gets weird, the weird turn pro. Few people have the ability/temperament to reason about things that have never happened before. Those that do have a weird psychological profile. Usually you can ignore them. Every once in a while, you can't.
Exactly. The vast majority of people never progress beyond naive extrapolation into the near future, even in the face of clear evidence that their previous extrapolations were wrong.
"Nobody will want to walk around with mobile phones. Who wants to be available any time and everywhere?"
People said similar things about Bluetooth headsets and are saying similar things about AR glasses. And these are very simple in terms of reasoning about how the technology evolves and the probability of adoption by people, yet troves of people get it ridiculously wrong; what they experience every day seems so normal that they can't imagine it ever changing.
Something as transformative and monumental as (potentially) an entirely new and powerful form of intelligence arising is far, far beyond the imagination of most. I don't mean that in that they are fundamentally incapable of it, but that it takes having been exposed to a bunch of counter-intuitive logic and reasoning about how the universe works, which most people just have not been.
Oh, man, not this again. I recommend listening to the "Behind the Bastards" podcast episodes on the Zizians if you want to really be confronted by the personal histories of some people who went really far down the rationalist rabbit hole. Yes, the harry potter fan fiction features prominently in it.
I took this advice and posted at https://news.ycombinator.com/item?id=49757722. Thanks for the nudge! Just generally speaking, I find Rao's writing to be a really good, well-informed counterbalance to...well, darn near everyone else that writes in the same sectors these days.
Useless article. Doesn't matter whether Coxon has pure motives, it got attention. You can't put that back in a bottle.
It does matter whether AI is safe or not. I won't make a specific argument here, but I think the rationalist community would be overjoyed to see unaffiliated people sincerely trying to reason through these things.
It seems that the author has very weird and strong obsession over Yudkowsky and his HP fanfic and took the public hatred toward AI companies as an opportunity to ridicule him.
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election (that is almost certain a losing proposition for the party in power), they put the future of humanity at risk.
I remember reading back in 2014 (IIRC) Yudkowsky's explanation of the Bayes theorem. It was genuinely good. Some sort of first principles introduction.
HPMOR was entertaining as well. Death as the last enemy to be destroyed placed his priorities where his mouth is.
For a brief time back then I used to visit Lesswrong and many comments including Yudkowsky's were plain stupid (which was not hard to reason at all) but coming from folks who valued themselves as first principles thinkers was an oxymoron.
But his timeless equations (IIRC), Bayes theorem explanation and HPMOR were all remarkable and I will always appreciate those works.
I hate seeing all the blind negativity towards the rationalists, both because the general idea of improving structured thinking is good, and because there were some great essays on Lesswrong and adjacent blogs over the years. People who can’t see that are blinded by their own bias.
That said, the most interesting people and discussions seem to have moved on, and now it’s more of a social club. I haven’t read anything compelling from there for a long time. Adjacent bloggers like Scott still put out interesting writing, but their output has dropped significantly.
Good to know. Haven't been following them as I was busy with our work.
I empathize with you on the sentiment of adversity against rationality but disagree on the implications.
Usually sentences uttered by anyone contains compressed in it the state of the mind and the thought process leading upto it.
Thus people blindly hating rationality makes it easy for us to decipher their state. Those who follow the cult worship likewise reveals their own state. So weild it as a weapon. Everybody is human and susceptible to be fallible. Truth doesn't care that it is Yudkowsky and hence true.
So I wouldn't be bothered by the opinions of the masses. Hope this helps. Cheers.
Revisiting after 10 years, it was nice reading through all those virtues again. Thank you for sharing the link.
After more than 2 decades of continuous building and problem solving, I see a lot of good things in that article and a handful of places where it goes wrong but was not apparent at the time.
But the content is solid as a general path to correctness and truth. In hindsight, a lot of those virtues were not just good, but correct. The nameless void at the end is the most potent of them all.
As someone who is weird, the reason I think these communities exist is because a lot of us are on the spectrum and it's honestly so exhausting trying to discuss ideas with people to just be called weird or toxic or whatever else. In a way the reason people see us as a group is because we can't do the groupthink thing, and so it's nice to talk to others who are not going mock you for liking Harry Potter and being interested in AI risk.
As a "group" we disagree on almost everything... It's why we talk to each other.
I can only speak for myself, but I definitely don't see my inability to socially align with others as a superiority thing. It can give me some predictive advantage when there's social stigma around a certain belief. But that's really about it. In most cases, especially professionally, it's very hard to navigate and I'd rather be naturally more aligned with people.
But yes, my profile is cringe and I'm sorry someone has downvoted you for sharing your view. There should be less of a taboo about telling people we dislike them if that is how we feel. It's better than to lie.
They they made the Animatrix as a companion piece.
The Second Renaissance [0] episode of the Animatrix talks about how the machines came to power in the Matrix universe. At the time, it made sense within the bounds of the movies.
Reading about a lot of this AI discussion makes me feel like it was a documentary sent back in time.
If you want to meet the sex part of that crowd, Slutcon is in a week.[1]
Tickets are $3,300 to $10,000. This is Aella's creation. It's in Berkeley, California, and there's a fair amount of overlap between the organizers and the Berkeley "rat" community mentioned in the parent article.
With a title like that, I was expecting a piece about society ending as a result of how too many people marrying chatbots affects the population pyramid.
I'm a born and raised the Southerner, so I don't really understand all this "Effective Altruism" and "Rationalist Movement" group stuff.
Seriously, I am not trying to be obnoxious or make some poor attempt at a joke. I used HN for years before I learned YCombinator was like Shark Tank for startups or something.
So, to my knowledgable folks in this thread, are these groups truly some efficacious, philanthropists, or are these movements just another one of those onanistic Bay Area beliefs I see people complain about a lot on the Internet?
(No hate here, I've never even been to the Bay, nor have I met any tech folks from there either).
I tried to read the article, but with all the proper nouns and acronyms, I might as well have been reading a foreign language.
That's a question without a good answer; compare asking the same question about e.g. Thomas Road Baptist Church (minus the "Bay Area" part).
Many members of the Effective Altruism movement have given away substantial fractions of their income to e.g. paying for mosquito nets in Africa. A non-trivial fraction have donated a kidney to a stranger. They're also an onanistic Bay Area group.
The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.
> Then the movement minted a fraud. Sam Bankman-Fried was not an accident of proximity; he was EA’s own product
TFA could have have mentioned that it's a fraud, MacAskill, who minted another fraud. MacAskill was SBF's guru and he was also involved in FTX and Alameda scams (but distanced himself before the scandal blew up) and MacAskill bought a 15 million GBP mansion in the UK for the Effective Altruism movement using funds stolen by SBF.
Last I checked the mansion hasn't been clawed back yet (but it could: funds from the Madoff ponzi have been clawed back more than 15 years after the fraud has been exposed).
Now of course people in charge of clawing back the stolen funds are the only hope of people who've been scammed for MacAskill didn't effectively altruistically sell the mansion to effectively altruistically hand the money to people who have been scammed.
That MacAskill and SBF maybe had groupsex with Caroline Ellison is one thing: they're proud to be polyamorous, whatever. We live in a free world.
But buying a $15 million mansion with stolen funds is thievery. Effective Scammers.
look up what "Agnotology" is. Ignorance is manufactured and can be weaponized-- it has a corporate lineage and silicon valley is very good at manufacturing it amongst it's workers-- Severance satirized this well. NYC will always beat SF because its harder to ignore voices from outside the bubble of an industry-town.
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
This sphere goes further back: There's direct connections to Extropianism and onto many more: Transhumanism, Julian Assange, Alcor, Timothy Leary, John Perry Barlow (EFF, Grateful Dead), another VP: Dick Cheney, Nicholas Negroponte, Marvin Minsky, Jeffrey Epstein...
Really depends on how broad of a brush you use. This article makes lots of connections but could connect with all of the above with only a few more.
> A sex worker from Idaho who became one of the subculture’s most-read writers and now runs an AI-doom propaganda residency with Grimes, Elon Musk’s ex-girlfriend.
I hate that I know who that is from this scant description.
> A splinter sect that fled to tugboats and whose members have been linked to six deaths. And a blogger who wanted to abolish democracy, found the ear of several billionaire patrons, and watched his ideas come to fruition in the vice president’s office.
I think I’m comfortable, even though the aspersion is on me, saying that anyone who can recognize Aella is too chronically online.
The zizians were a national headline and you only need to be plugged into US politics a little bit to know of Curtis yarvin given his influence on Peter thiel, musk, and Vance so I’d give knowledge of them a pass
Edit: I also know of the sanely diagram meme for her so that’s even more evidence. Maybe I should detox off the internet for a bit
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
>If worries about AI safety and the end of the world entered your life recently, it’s probably because of a resignation letter.
It's actually because of all the hoops I have to jump through to get Gemini to generate my kinky bara/furry pin-ups and videos.
I totally get that this is exceptionally weird to the vast majority of people. Even the other furries ostracize me for daring to engage with what they seem to think is a sort of digital Moloch.
I read a rant recently, where the main thrust was that "AI safety" was about controlling the user. Censorship. I find it easy to agree with that take. I suppose that there's some merit to not allowing these systems to teach users how to build a nuke, but this early in the game, with this many restrictions, what is "safety" in practice? "Protecting people from bad actors"? Or "defining a narrow idea of what's acceptable, according to the particular ethical and philosophical idiosyncrasies of the system admins and owners," which I guess I don't share?
My weird little art-adjacent initiatives are such small potatoes, a personal Mirror of Erised that has basically zero effect on the lives of anyone else. But even they're too much to abide. They want to make genAI an indispensable tool for the next generation or so of human expression, but the guardrails are up and tightening with every model release. No dicks in the contemporary art galleries of 2050, huh? (Maybe the other furries are right after all.) And what happens when AI is an arbiter in situations involving real blood or treasure? Situations that antagonize the platform's owners, or the existence or purpose of the platform itself?
And the public gets basically no input, let alone the individual. Sounds great. /s
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
If we all die because of AI, it will be because of the uses that people like Elon Musk and Peter Thiel and Palmer Luckey put it to, not because of a robot uprising.
If you are serious about this preference, would you also be okay with a compromise solution where only you (and anyone else who feels the same) die, rather than taking everyone else with you?
The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
9 replies →
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind.
The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
21 replies →
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi
28 replies →
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
Do you think they are household names in major Chinese labs?
I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
1 reply →
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
2 replies →
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
17 replies →
I would find it strange for anyone who's seriously interested in AI to have never heard of...
Those people you name are important only to the very recent micro-culture being discussed, not "AI".
Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only.
I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
1 reply →
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
I just heard about them today. The only other person mentioned in the story that I've heard about once was Aella.
I mean, why should I have heard about them if I'm not in SV?
Most people outside of SF have never heard of Y Combinator
I’ve only read his Harry Potter fanfic
[dead]
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
> Public got the last word, industry suffered a 40 year ice age.
This is the worst outcome.
It's like that, now imagine a couple billionaires started funding pro-nuclear groups as a means for them to promote their anti-democracy ideals.
> The framing of how we think about AI
I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.
They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.
But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.
We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.
Who are the actual scientists and engineers, what do they think, and are they not allowed to to blog?
As far as I'm aware, lots of them are actually deeply concerned about AI killing everyone.
1 reply →
So what you're saying is that we lack a reproduction study?
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons. When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it. The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural -stories were always a bad hack.
Nicely, though depressingly, put. Likely more responsible for the high levels of dementia and alzheimers etc than people want to believe.
this u?
> AI will replace all of them soon.
What is that spaghetti of words you tied together to form a sentence? Were you trying to be comprehensible?
Everyone can easily identify when people they don't like are captured in bubbles, but can rarely notice it when people they do like are trapped.
Probably all hot-button topics work like this. Probably we should downplay any time a small homogenous group of people sings a song of doom together. More dissonance please!
> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.
There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations.
AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.
This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.
Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that.
AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.
1. They can be more powerful than any one human.
2. They lack liability as they are not people.
3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.
That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.
That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.
This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.
Solve one, and you solve the other.
1 reply →
I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) than any folks in the groupthink bubble you are referring to.
That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.
When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."
I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.
It appears you're using a motte and bailey to broaden P(doom) to include all the other non-Skynet reasons regular people hate SV. Doom in p(doom) refers to X-risk not unemployment and upward transfer of wealth.
I don’t even tell people in my (remote work, non tech) neighborhood that I work in tech anymore unless they really probe. I don’t like getting my ass kicked.
1 reply →
Well it’s easy for these people. They are smarter than the plebs and so the plebs are barely if even human.
That’s like a running theme in these connected groups and even the article calls it out.
1 reply →
Do you have a model of AI that isn't in this groupthink? What does the discussion in your other universe look like?
I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.
Most people at least in the US who have any opinion on AI have a negative opinion. The one big exception to that is Silicon Valley and forums like HN.
But according to you, the negative opinion is the result "not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way" whereas the 1% or 2% of the US population that comprises Silicon Valley are not being rewarding for believing anything particular about AI, don't communicate with each other and form their beliefs about AI through a wholesome natural organic process. Do I have that right?
When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
Every time I see a post minimizing the dangers if AI I'm going to wonder whether it's from part of an AI swarm.
Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.
Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.
You are the entity he is referring to when he speaks of "groupthink".
5 replies →
Do you assume that every post criticizing anything is part of a sockpuppet disinformation campaign as well?
I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
As someone on the periphery looking in, it's because they make up a large web. Its an extended community that shares a lot of people between its many arms. There are also some shared values between all these people, even at the very extended edges. For example, they share a superiority complex. From my experience, most of these people believe that they see further or know more because they believe in certain things that are, on their face, unintuitive. I don't want to pass more judgement than to say, having members believe contradictory or outright falsehoods is a thing that cults do. I'm not saying this is a cult because I haven't seriously sat down and thought about it but that is reminiscent of something cults do.
Read HPMOR. Unlike what I thought it might be, the protagonist does not solve problems through pointing out logical fallacies or something. Harry Potter is a straight-up superhero who is infinitely smarter and more talented than anyone around him. It's not so much rationality as a narcissistic fantasy that goes on for hundreds (thousands?) of pages.
Even though I had been well-disposed to rationalism and read HPMOR to learn more, I could not take Yudkowsky seriously in any way after I gave up halfway through.
1 reply →
[flagged]
6 replies →
This is how the internet works now. There are only two clubs, and if you refuse to be associated with either one then someone will helpfully come along and assign a group for you.
It's a big tent community, in part because these people spend so much time entertaining wild, speculative hypotheticals, which in the process allows some of these particularly toxic ideas to spread.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's not just that they are "weirdos"
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
They're about as insular as the Unitarian church or Hacker News. It's basically an online community where anyone can check in, check out, and leave a comment or an essay. There's no formal structure or membership organization. The people who write or associate there could be edgy pre-teens or prominent math professors, it's really just a self-association thing. There are informal self-organized meetup groups in various cities, but not much different than groups who meet to discuss poetry or literature or video game speed-running. Some ideas have bubbled around more than others, like any memetic spread, but I don't think it's accurate to say that the people who associate there all agree on things, just like HN does not either. And nor is it any more of a clique than HN, or Twitter for that matter, even though you'll find some people on Twitter are more prominent and more vocal than others.
8 replies →
Very well stated
People in this comments page who are defending the crazy rationalist sects can be safely treated as though they’re arguing in bad faith. It matters little whether they are, or if they’ve ODed on their manbaby kool-aid and truly believe their nonsense; they’re just here to waste your time.
I think the point is not so much that we can just ignore the possibility of AI doing harm, but that if AI really is the existential threat that these people claim, then they are the last people we should want as stewards of the technology. These people like to talk about AI ethics, but if the ethics of the AI they build is their ethics, as demonstrated by their actions, I want no part of it.
Okay, so what should we do?
I for one would like nobody to build the AI, is that an option? How do we get it on the menu?
Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"
4 replies →
> when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.
Other people, sure, but AI? That's what this is all about, isn't it? An AI judge wouldn't have a personal life. No family to threaten, no sex/drugs/gambling addiction to be blackmailed or corrupted over. Who gets to feed training data into that AI would have outsized power, but then at least it would be transparent.
Judgement, wisdom, and resistance to corruption are the 3 things that Tom Kuegler analyzes out of the psychology of Dune as the qualities for the human beings who administer institutions/government. Even if you design a totall perfect system of governance, it still has to be administered by humans, and humans vary in those 3 areas. So the question is, can we construct a system of governance that has better judgement, better wisdom, and greater resistance to corruption han human administators, while not transferring sovereignty to whomever controls the AI?
1 reply →
The rationality of AI Optimists is at least rational in so far as making a huge pile of money is the rational thing to persue in capitalism.
1 reply →
I really don’t know how we went from “there are people bullish about AI, and others that see it as an existential threat” to the complete reversal/psyop in the past few months where AI doomer has become a synonym of booster (?) and grouped with rationalist and other pro-AI weirdos.
Like, I have never participated in the rationalist communities, I am not a neo-rx, I have always believed AI research to be an existential risk for humanity, and voicing this opinion now puts me in the same category as Yudkowsky and Altman/Amodei, the latter two being the total anathema to O.G. AI doomers like me because they’re the ones that are building the bloody thing
I really do not get what is going on, but this role reversal smells like a massive psyop campaign in tech circles to just silence any form of dissent against AI.
I feel your frustration, but it’s just normal human behavior, no need for a psyop. A lot of people are emotionally invested in AI being a fad, useless, easily controlled, a capitalist boondoggle, a circular financing scam, a bubble about to explode. The alternative is that we live in an incredibly dangerous time, because obviously if we’re building something smarter than ourselves, we’re building the kitchen where we’ll be cooked.
So you get the normal reactions:
1. Nothing that happens is ever surprising. Navier-Stokes was stolen, HuggingFace was pedestrian and normal, and coding agents that last year could barely write a function and now can do a day’s worth of work in ten minutes will never be able to do anything more, because this is the end of history.
2. The people that say otherwise are lying. Because it’ll make them rich (although they are already rich). Because they are a cult (although regular people asked if we should build machines that are better than us at everything have the same reaction). Because they are marketing geniuses (although suspiciously few companies advertise the lethality of their products).
3. The people who’ve been warning about this the longest are weird and in a sex cult. (Except Alan Turing.) (Except Geoffrey Hinton.) (Except Stephen Hawking.) (You are in this group. You must be, otherwise they’d have to listen to you.)
I do hope they’re right. But their song will never change, and when some misaligned AI kills thousands, they will say that of course AIs were always going to kill thousands, but did you know the guy who told us so also writes fanfic?
There's no connection between neo-rx and the Berkeley rationalists. A guy with a lot of expertise in Wikipedia shenanigans (David Gerard) manufactured a connection to smear the rationalists.
The argument was fairly specific: there is a groupthink involved, and so the enterprise is suspect.
It’s an excuse not to think, not an argument.
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
4 replies →
> draw a very straight line to catastrophe
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
Your mistake is framing this as "skynet vs unified human resistance led by john connor". In reality it's "human players in the existing class war suddenly curb stomped by software".
Think less "skynet" and more "a corporation without any humans in it". What defends it from physical attack? Same things that defend Google from physical attack now. Everything that AI will make worse, we already have. AI just supercharges it, lets it scale, lets it run without even the inconvenience of suppressed empathy (if it doesn't seem like that's constrained corporations much so far, let's see how deep the rabbit hole goes without any humans in the loop).
There won't be a human resistance. There may be people who resist, but the police will arrest them. Existing power structures will continue to be neatly and elegantly coopted. Extinction doesn't look like HKs roaming the landscape, it looks like human survivability simply being economically untenable in the economy of tomorrow.
There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
Yes, I wonder who the clowns are:
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
20 replies →
Those aren't the same are they?
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
Did the METR investigation not show you enough?
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
6 replies →
Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality and project models forward in time (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
> when you can just start from zero, look at what's known to be happening right now
Good idea, I wonder why none of the doomers have ever done it? They've never provided a good account of their prompts and environment, their models and training are completely closed too. All we have is "Trust me bro".
> and draw a very straight line to catastrophe.
Line from where? From a vaguely described and badly engineered experiment to "horror, doom, nukes, voodoo..."? The only straight line I can see is one of conflicts of interest.
It’s just nonsense man.
Like, Yudkowski is a high school dropout who wrote a bunch of Harry Potter fanfic I cannot believe that people take the arguments he makes seriously.
I don't think that comparison is quite apropos
> deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem before LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
> autonomous bots hacking sites.
They weren't even autonomous. They were tasked with solving a hacking task. "AI" doesn't actually do anything at all unless prompted to, then it takes millions and millions of dollars of computer to do those things.
>And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans
Ah yes - LLMs don't kill people, people kill people.
Cold comfort!
When the going gets weird, the weird turn pro. Few people have the ability/temperament to reason about things that have never happened before. Those that do have a weird psychological profile. Usually you can ignore them. Every once in a while, you can't.
Exactly. The vast majority of people never progress beyond naive extrapolation into the near future, even in the face of clear evidence that their previous extrapolations were wrong.
"Nobody will want to walk around with mobile phones. Who wants to be available any time and everywhere?"
People said similar things about Bluetooth headsets and are saying similar things about AR glasses. And these are very simple in terms of reasoning about how the technology evolves and the probability of adoption by people, yet troves of people get it ridiculously wrong; what they experience every day seems so normal that they can't imagine it ever changing.
Something as transformative and monumental as (potentially) an entirely new and powerful form of intelligence arising is far, far beyond the imagination of most. I don't mean that in that they are fundamentally incapable of it, but that it takes having been exposed to a bunch of counter-intuitive logic and reasoning about how the universe works, which most people just have not been.
The basic problem is that out-of-sample predictions don't work very well.
Depends on the person. Some people will say wild stuff like Neptune exists or black holes exist and they will be right.
Oh, man, not this again. I recommend listening to the "Behind the Bastards" podcast episodes on the Zizians if you want to really be confronted by the personal histories of some people who went really far down the rationalist rabbit hole. Yes, the harry potter fan fiction features prominently in it.
https://www.google.com/search?client=firefox-b-d&q=behind+th...
Probably best read with [Venkatesh Rao on EA Safety](https://substack.com/home/post/p-216194276) as a companion piece.
Rao's piece is much, much better. You should submit it as a top-level link!
I took this advice and posted at https://news.ycombinator.com/item?id=49757722. Thanks for the nudge! Just generally speaking, I find Rao's writing to be a really good, well-informed counterbalance to...well, darn near everyone else that writes in the same sectors these days.
Useless article. Doesn't matter whether Coxon has pure motives, it got attention. You can't put that back in a bottle.
It does matter whether AI is safe or not. I won't make a specific argument here, but I think the rationalist community would be overjoyed to see unaffiliated people sincerely trying to reason through these things.
It seems that the author has very weird and strong obsession over Yudkowsky and his HP fanfic and took the public hatred toward AI companies as an opportunity to ridicule him.
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election (that is almost certain a losing proposition for the party in power), they put the future of humanity at risk.
As Nixon did.
Well at least at that time, bi-partisan ethos was at a level that such acts would actually lead to an impeachment or forced resignation.
I remember reading back in 2014 (IIRC) Yudkowsky's explanation of the Bayes theorem. It was genuinely good. Some sort of first principles introduction.
HPMOR was entertaining as well. Death as the last enemy to be destroyed placed his priorities where his mouth is.
For a brief time back then I used to visit Lesswrong and many comments including Yudkowsky's were plain stupid (which was not hard to reason at all) but coming from folks who valued themselves as first principles thinkers was an oxymoron.
But his timeless equations (IIRC), Bayes theorem explanation and HPMOR were all remarkable and I will always appreciate those works.
I hate seeing all the blind negativity towards the rationalists, both because the general idea of improving structured thinking is good, and because there were some great essays on Lesswrong and adjacent blogs over the years. People who can’t see that are blinded by their own bias.
That said, the most interesting people and discussions seem to have moved on, and now it’s more of a social club. I haven’t read anything compelling from there for a long time. Adjacent bloggers like Scott still put out interesting writing, but their output has dropped significantly.
Good to know. Haven't been following them as I was busy with our work.
I empathize with you on the sentiment of adversity against rationality but disagree on the implications.
Usually sentences uttered by anyone contains compressed in it the state of the mind and the thought process leading upto it.
Thus people blindly hating rationality makes it easy for us to decipher their state. Those who follow the cult worship likewise reveals their own state. So weild it as a weapon. Everybody is human and susceptible to be fallible. Truth doesn't care that it is Yudkowsky and hence true.
So I wouldn't be bothered by the opinions of the masses. Hope this helps. Cheers.
My favorite writing of his is:
https://www.yudkowsky.net/rational/virtues
Revisiting after 10 years, it was nice reading through all those virtues again. Thank you for sharing the link.
After more than 2 decades of continuous building and problem solving, I see a lot of good things in that article and a handful of places where it goes wrong but was not apparent at the time.
But the content is solid as a general path to correctness and truth. In hindsight, a lot of those virtues were not just good, but correct. The nameless void at the end is the most potent of them all.
As someone who is weird, the reason I think these communities exist is because a lot of us are on the spectrum and it's honestly so exhausting trying to discuss ideas with people to just be called weird or toxic or whatever else. In a way the reason people see us as a group is because we can't do the groupthink thing, and so it's nice to talk to others who are not going mock you for liking Harry Potter and being interested in AI risk.
As a "group" we disagree on almost everything... It's why we talk to each other.
[flagged]
I can only speak for myself, but I definitely don't see my inability to socially align with others as a superiority thing. It can give me some predictive advantage when there's social stigma around a certain belief. But that's really about it. In most cases, especially professionally, it's very hard to navigate and I'd rather be naturally more aligned with people.
But yes, my profile is cringe and I'm sorry someone has downvoted you for sharing your view. There should be less of a taboo about telling people we dislike them if that is how we feel. It's better than to lie.
3 replies →
I remember first watching the Matrix.
They they made the Animatrix as a companion piece.
The Second Renaissance [0] episode of the Animatrix talks about how the machines came to power in the Matrix universe. At the time, it made sense within the bounds of the movies.
Reading about a lot of this AI discussion makes me feel like it was a documentary sent back in time.
https://www.youtube.com/playlist?list=PLUouZgbTp2o_7LK7zqubt...
The best piece of reporting on the subject I have ever seen.
If you want to meet the sex part of that crowd, Slutcon is in a week.[1] Tickets are $3,300 to $10,000. This is Aella's creation. It's in Berkeley, California, and there's a fair amount of overlap between the organizers and the Berkeley "rat" community mentioned in the parent article.
[1] https://www.slutcon.com/
Word of advice: don't take advice on sex from people who famously don't shower. I have heard from reputable sources that Aella smells like a latrine.
Looks like Aella showers 37 times a year (less than once a week) and goes to great lengths to justify this[1]. What is wrong with these people?
[1] https://x.com/Aella_Girl/status/1589690655965872128
> Tickets are $3,300 to $10,000.
Joining the Illuminati who rule the world is surprisingly cheap. Next time a similar article gets on HN front page, it will be about you!
Try joining Bildenberg group not a whore house
This seems just like a money grab of tech bros who didn't develop social skills.
Look at the schedule:
How to get the hot girl to like you
Flirting for the submissive man
What women want, taken from sex novels
80% of the schedule seems like self-help talks for men
It reads like the rich version of traditional pickup artist "seminars".. that space is now occupied by the likes of Andrew Tate I suppose..
With a title like that, I was expecting a piece about society ending as a result of how too many people marrying chatbots affects the population pyramid.
I'm a born and raised the Southerner, so I don't really understand all this "Effective Altruism" and "Rationalist Movement" group stuff.
Seriously, I am not trying to be obnoxious or make some poor attempt at a joke. I used HN for years before I learned YCombinator was like Shark Tank for startups or something.
So, to my knowledgable folks in this thread, are these groups truly some efficacious, philanthropists, or are these movements just another one of those onanistic Bay Area beliefs I see people complain about a lot on the Internet? (No hate here, I've never even been to the Bay, nor have I met any tech folks from there either).
I tried to read the article, but with all the proper nouns and acronyms, I might as well have been reading a foreign language.
That's a question without a good answer; compare asking the same question about e.g. Thomas Road Baptist Church (minus the "Bay Area" part).
Many members of the Effective Altruism movement have given away substantial fractions of their income to e.g. paying for mosquito nets in Africa. A non-trivial fraction have donated a kidney to a stranger. They're also an onanistic Bay Area group.
> If worries about AI safety and the end of the world entered your life recently, it’s probably because of a resignation letter.
No, it's because of an extensive and well coordinated FUD campaign. The resignation letter was just one salvo, and, like the rest, orchestrated.
I'm not a huge fan of calling LLMs "AI" and acting like they're our savior, but I'm even less of a fan of the brain-dead fearmongering.
The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.
> William MacAskill co-founded 80,000 Hour ...
> Then the movement minted a fraud. Sam Bankman-Fried was not an accident of proximity; he was EA’s own product
TFA could have have mentioned that it's a fraud, MacAskill, who minted another fraud. MacAskill was SBF's guru and he was also involved in FTX and Alameda scams (but distanced himself before the scandal blew up) and MacAskill bought a 15 million GBP mansion in the UK for the Effective Altruism movement using funds stolen by SBF.
Last I checked the mansion hasn't been clawed back yet (but it could: funds from the Madoff ponzi have been clawed back more than 15 years after the fraud has been exposed).
Now of course people in charge of clawing back the stolen funds are the only hope of people who've been scammed for MacAskill didn't effectively altruistically sell the mansion to effectively altruistically hand the money to people who have been scammed.
That MacAskill and SBF maybe had groupsex with Caroline Ellison is one thing: they're proud to be polyamorous, whatever. We live in a free world.
But buying a $15 million mansion with stolen funds is thievery. Effective Scammers.
This writing smells like the aggrandizement of subcultures seen in early Wired articles.
We need deep and rational discussion of the implications, impact and dangers of AI and how to mitigate them.
This is not that - although it is a summary of some of the social cliques / trends, and maybe a warning against groupthink.
Anyone know why this site brings Firefox to its knees (I'm on a Ryzen 9700X with 64GiB of RAM and scrolling has about a 2-second lag)?
dupe https://news.ycombinator.com/item?id=49738411
Really struck by some of the parallels with other "perfection cults", e.g. the Oneida Community circa 1850's https://en.wikipedia.org/wiki/Oneida_Community
The Oneida's also produced a violent assassin: Charles Guiteau https://en.wikipedia.org/wiki/Charles_J._Guiteau (I guess it is debatable they "made" him, same as with the Zizians)
Let's hope Thiel and company redirect that democracy-destroying energy into making quality flatware sometime soon...
[dead]
related / additional https://contraptions.venkateshrao.com/p/ea-safety
Thats certainly a title
I'd watch the movie for sure
It is pretty funny to me that the transhumanist movement is hoping that these LLMs can eventually move past a parody of human intelligence.
A lot of weird folks in that arena, a lot of strange books too that they wrote for the consumption of the public. It's always the same backstory too.
See also from earlier today [1].
More or less the same content but parent is much more palpable, IMHO.
[1] https://news.ycombinator.com/item?id=49737985
look up what "Agnotology" is. Ignorance is manufactured and can be weaponized-- it has a corporate lineage and silicon valley is very good at manufacturing it amongst it's workers-- Severance satirized this well. NYC will always beat SF because its harder to ignore voices from outside the bubble of an industry-town.
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
This sphere goes further back: There's direct connections to Extropianism and onto many more: Transhumanism, Julian Assange, Alcor, Timothy Leary, John Perry Barlow (EFF, Grateful Dead), another VP: Dick Cheney, Nicholas Negroponte, Marvin Minsky, Jeffrey Epstein...
Really depends on how broad of a brush you use. This article makes lots of connections but could connect with all of the above with only a few more.
> A sex worker from Idaho who became one of the subculture’s most-read writers and now runs an AI-doom propaganda residency with Grimes, Elon Musk’s ex-girlfriend.
I hate that I know who that is from this scant description.
> A splinter sect that fled to tugboats and whose members have been linked to six deaths. And a blogger who wanted to abolish democracy, found the ear of several billionaire patrons, and watched his ideas come to fruition in the vice president’s office.
Actually fuck I know who all of these people are.
Yeah, I did too. But it doesn’t mean we’re, like, hanging out with ‘em
Although perhaps we are excessively online
I think I’m comfortable, even though the aspersion is on me, saying that anyone who can recognize Aella is too chronically online.
The zizians were a national headline and you only need to be plugged into US politics a little bit to know of Curtis yarvin given his influence on Peter thiel, musk, and Vance so I’d give knowledge of them a pass
Edit: I also know of the sanely diagram meme for her so that’s even more evidence. Maybe I should detox off the internet for a bit
1 reply →
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
[dead]
>If worries about AI safety and the end of the world entered your life recently, it’s probably because of a resignation letter.
It's actually because of all the hoops I have to jump through to get Gemini to generate my kinky bara/furry pin-ups and videos.
I totally get that this is exceptionally weird to the vast majority of people. Even the other furries ostracize me for daring to engage with what they seem to think is a sort of digital Moloch.
I read a rant recently, where the main thrust was that "AI safety" was about controlling the user. Censorship. I find it easy to agree with that take. I suppose that there's some merit to not allowing these systems to teach users how to build a nuke, but this early in the game, with this many restrictions, what is "safety" in practice? "Protecting people from bad actors"? Or "defining a narrow idea of what's acceptable, according to the particular ethical and philosophical idiosyncrasies of the system admins and owners," which I guess I don't share?
My weird little art-adjacent initiatives are such small potatoes, a personal Mirror of Erised that has basically zero effect on the lives of anyone else. But even they're too much to abide. They want to make genAI an indispensable tool for the next generation or so of human expression, but the guardrails are up and tightening with every model release. No dicks in the contemporary art galleries of 2050, huh? (Maybe the other furries are right after all.) And what happens when AI is an arbiter in situations involving real blood or treasure? Situations that antagonize the platform's owners, or the existence or purpose of the platform itself?
And the public gets basically no input, let alone the individual. Sounds great. /s
[dead]
[dead]
[dead]
[dead]
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
If we all die because of AI, it will be because of the uses that people like Elon Musk and Peter Thiel and Palmer Luckey put it to, not because of a robot uprising.
Why not both? I honestly can’t fathom why people think — in an ultra-networked world — why LLMs can’t cause physical consequences.
I think it's more of a Chicken Little situation. Then the wolf ate us all up.
[dead]
I'd rather have the world end than live in a world where EAs are listened to or respected.
If you are serious about this preference, would you also be okay with a compromise solution where only you (and anyone else who feels the same) die, rather than taking everyone else with you?
I agree they’re kinda weird but are they really that bad?
10 replies →
I have great news then, that absolutely isn't going to happen. And the EA crowd deserves all of the potshots and more
Wow I’m sure glad that’s settled then.
1 reply →
ea would turn out to be the millennial's peak "gift" to humanity: the most spoiled, detached and deranged group that will either end or enslave all.