Comment by chubot

Totally agree. There do seem to be a bunch of accomplished people that have acquired "mind bugs" from people whose job it is to write and be popular, and not produce very much.

Yudkowsky is a prime example of the latter group. I think I became aware of him through the Roko's Basilisk meme many years ago, and I thought "wow so this is what happens when you have a lot to think about and nothing to do". It's basically nonsense.

Two other people like that are Nick Bostrom and Will MacAskill.

I heard of "Superintelligence" by Bostrom probably through an Elon Musk recommendation in 2015 or so. That's when he started OpenAI, and I was working on AI at the time. (Remember Elon had a completely different reputation then; he was respected as a person who built things, and wasn't associated with any political ideas.)

So I got the book, and to its credit, the beginning of the book acknowledges that the rest of the book may be complete bullshit. It's extremely low confidence extrapolation. And I pretty much agreed with that preface -- most of the book was nonsense. However many people seemed to take it seriously.

(I'm not saying AI is going to be great, or catastrophic either. I was just looking for some kind of high quality analysis from someone with domain knowledge, and found none of it in that book.)

----

I think I became aware of MacAskill through a NYTimes article puffing his book last year. There was something "off" about it.

And few months later, it turned out that he was conveniently unaware that the only reason that his projects were funded was because his friend was committing a huge crime:

https://time.com/6262810/sam-bankman-fried-effective-altruis...

So yeah, obviously I can't take seriously the moral opinions of somebody whose morals fall down immediately and spectacularly when faced with a whiff of the real world.

It seems like his main job is to create and advocate ideas that are intellectually appealing to and in the interest of billionaires. It came out in the FTX aftermath that he was telling Elon that SBF could help him buy Twitter.

BTW Elon also recommended the MacAskill book, which another thing that has lowered my opinion of Elon. People are people are flattering him with ideas, and he's taking the bait.

----

Anyway, normally I wouldn't have any awareness of the kind of writing that these 3 people produce. Much of it is low quality, which I would expect given their lack of experience and expertise in the domain of AI. And 75% of it couldn't have any effect on the world -- even in principle.

But like you, I'm puzzled that some people I respect seem to take them seriously. I mean I do respect Fridman, since I've learned a bunch of things from his podcasts with other guests. (I understand why a lot of people don't like him, but I prefer to focus on a person's best work and not their worst.)

But yeah the best work of Yudovsky, Bostrom, and MacAskill doesn't seem noteworthy.

It seems designed to generate attention, and that's mostly it.

The extremely, extremely obvious thing they're missing (or just fail to emphasize because it won't get them attention) is that you should be scared of corporations and governments with AI (first), not scared of autonomous AI with its own will.

That is, the "superintelligence" and "longterm-ism" ideas are basically deflections from the real issues with AI.

----

So yeah I was puzzled about this whole thing, and someone on HN recommended this substack, and I recommend it too. It's pretty interesting and it draws a line between the 90's extropians@ mailing list, Yudowsky, Bostrom, and multiple founders of Bitcoin.

Also a lot of dark stuff like suicides, mental illness, and abuse in the rationalist movement.

https://aiascendant.substack.com/p/extropias-children-chapte...

What also seems to have happened in this era is that Yudkowsky and MIRI, flush with newly donated millions, decided to try to evolve from theory and community-building to actual practical AI research. It did not go well.

This was quite a claim: essentially, that this research would, if published, meaningfully accelerate perhaps the most extraordinarily difficult challenge in the history of human innovation. It is a claim which remains 100% unsubstantiated.