Comment by skybrian
1 month ago
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Talking to the incredible talent in the Chinese Labs is likely a good way to do that.
Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs.
Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
2 replies →
Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink.
I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
It’s kind of weird in that reading more widely is supposed to be broadening (you are more well-read) but in this case it’s supposed to contaminate you somehow. I suppose if you wanted a clean-room implementation of something, you’d want people who don’t know about what’s already been done, but normally one would prefer people who are more knowledgeable about the world.
It seems almost anti-intellectual: never trust someone who’s read the wrong books or been to the wrong parties.
> More out-of-distribution ways to think of things.
Openly admitting that it only caught fire once the Plagiarism-as-a-Service took hold is a good first step. It's the part that everyone knows and few will openly acknowledge.
"Outside the box" is out.
I blame the play Rossum's Universal Robots, published in 1920, in which robots launch a global rebellion and exterminate the human race.
If that poisonous piece of propaganda had never been published, then today everyone would probably perceive AI correctly as a benign and beneficial.
The Terminator didn't help either, but Cameron probably got the idea of robots hostile to humanity from Rossum's Universal Robots. How else would he have arrived at such an absurd conclusion?
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind.
The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age..
Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
1 reply →
I would second this. I only saw it in passing where someone was rightfully making fun of yud somewhere, but having been in and out of academia in tech for two decades now, I've never heard anyone mention fanfic or yud other than randos on HN threads, and most of the physics people I know, at least the ones familiar with Jaynes and his contemporaries, would almost certainly laugh at a lot of fumbles I occasionally see with people trying to apply "baysenianism".
Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.
I spend a lot of time on the Internet and other then sounding vaguely familiar I have no idea who Yudkowsky is and have never read any of their output.
The assumption "spent a lot of time on the Internet" is still through a framing you're assuming is more general even then.
HackerNews straight up has sub demographics who likely barely interact based on article interests.
You have to be quite unaware of your bubble if you think knowing about a niche Harry Potter fanfic is "standard" for people on the Internet.
"niche"? It's literally one of the most read pieces of fanfiction of all time.
13 replies →
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi
One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
I love the assertion that they were right and that you're familiar with what they were right about, but never actually stating outright what they were right about. Super convincing.
1 reply →
> what they were predicting 20 years ago
Well one thing that they got wrong was how easy AI alignment ended up being in the end. I can fully understand have a basic view of alignment even 10 years ago, about how its difficult to explain human values to a computer.
But, looking around at whats out there today, it seems that computers are able to do a pretty good job of understanding human values without accidently believing its a good idea to turn the world into paperclips/computronium because you told it to make your computer run faster.
12 replies →
I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
8 replies →
Bold of you to assume that I've ever been wrong about anything. But are you unfamiliar with the "Baltimore Stockbroker" confidence scam? The same basic issue applies here.
[dead]
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
Do you think they are household names in major Chinese labs?
I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority.
China probably looks at these people as a good reason why they have to keep going no matter what.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
2 replies →
A while back I saw LessWrong recommended on HN in a list of similar communities that included lobsters.rs, etc. I thought the design of the site looked really nice so browsed with a open mind (I didn't make the rationalism/LW/Eliezer connection).
At the time, one of the recommended posts that was soft-stickied at the top for 1 week + was:
"How to Make Superbabies"
https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-mak...
Which, when I finally ended up clicking, was pretty much exactly what if sounds like from the title:
A long, meandering, manifesto full of selectively chosen charts and research findings with 38 mentions of "IQ", from a guy who has been unsuccessfully pitching investors on genetically modifying a grab bag of (dubiously) intelligence-associated genes in human babies.
Apart from the overall unhingedness, the part I found most striking while reading it was the off-the-chart level of self-confidence that flipping a bunch of genes on/off with CRISPR would have more or less zero risk of complications or unpredictable interactions. In his view, all that was holding back humanity's inevitable next step up the evolutionary ladder were the narrow-minded gatekeepers in the scientific establishment clinging to their so-called "ethics" and regressive "precautionary principle".
Without anything resembling experimentation to test his plan in animal models, yet alone early stage clinical trials for safety, he was willing to roll the dice on some unlucky baby dying in childbirth or living a life of agony from chronic disease based on some cherry picked papers he found on PubMed.
Because of the baffling hubris and my overall low impression of his intellectual honesty/rigor in the write up (clearly coming from a place of seeking to justify his own pre-existing beliefs), I opened the comments section expecting/hoping this was just some local nut who would be torn apart in the comment section.
But lo and behold, the very first comment at the top:
... and thus ended my brief time trying out LW.
My extremely uncharitable view of of these kinds of people, who now infest the American AI scene, is that they got cyber bullied by domain experts for saying the most ridiculous crap on repeat, and have therefore decided to spend their life on eliminating domain expertise.
9 replies →
The connection is that by making humans smarter, we could sidestep the need for powerful AI or be better equipped at dealing with AI that's smarter than us.
I find the proposal in the post ridiculous too and human gene editing given current technology is a bad idea, but I think human augmentation in general is an interesting path and something that will very likely be taken more seriously in the future.
1 reply →
I would find it strange for anyone who's seriously interested in AI to have never heard of...
Those people you name are important only to the very recent micro-culture being discussed, not "AI".
Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only.
I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche.
If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
I just heard about them today. The only other person mentioned in the story that I've heard about once was Aella.
I mean, why should I have heard about them if I'm not in SV?
Most people outside of SF have never heard of Y Combinator
I’ve only read his Harry Potter fanfic
[dead]