Comment by gjm11

4 days ago

I see this said all the time and I think it's almost entirely wrong.

(Short version: those parallels aren't terribly close and one can find plenty similar parallels in things that no one thinks are religion-like. Long version: read on.)

"Roko's Basilisk is just rewarmed Pascal's Wager". I don't think it really is, but in any case it's something that so far as I can make out rationalists generally don't agree with and don't give any thought to except when it gets trotted out yet again as an illustration of how crazy rationalists are.

"The singularity is just the apocalypse / rapture". Both are hypothetical future events that would change the world a lot. Beyond that the match doesn't seem very close. The Christian apocalypse narrative is, above all else, about Good Versus Evil. A battle between God and his enemies! A grand judgement in which the living and the dead are all consigned to eternal bliss or eternal destruction! Us against Them! And there's none of that in what people say about technological singularities. (Well, except for Roko's basilisk, I guess. But, again, that isn't actually a thing anyone believes in or gives any thought to.)

"High priests (tech CEOs)". I don't think I've ever seen anything written by rationalists that treats tech CEOs as high priests. Here are a few quotations from recent highly-upvoted Less Wrong comments from high-karma LW users; each was the first thing I found in a search that seemed like it was expressing any sort of opinion on the CEO in question: "Sam Altman has systematically dismantled all of OpenAI's oversight, and betrayed the promises OpenAI made to humanity." "I would add that Elon Musk is not trustworthy and has been strongly suspected of manipulating the recommendation of X ..." "The best executives are very strong generalists, and the best managers are chosen to be strong performers in the task that they are managing other people to do. Elon is widely known to be a strong engineer, as well as a strong designer, and spends much of his time arguing details of that kind of work with his reports. Mark Zuckerberg is known to do similarly." First two are very strongly negative. Third is positive but a long way from setting either of the people involved up as high priests. I didn't find anything very specifically pro or con Satya Nadella, Sundar Pichai, or Jeff Bezos.

(A lot of internet-rationalists were pretty keen on Elon Musk back in the day, but so were a lot of other people. And there was an infamous post in which Yudkowsky wrote about how major-company CEOs and hedge fund managers and eminent professors seemed really impressive when he met them, which is cringey but not obviously wrong and in any case falls well short of calling anyone high priests -- and the comments below it are full of people saying things like "have you considered that maybe one way you get to be a CEO is by being good at looking impressive?".)

"theologians (Yud, SA)": if "theologian" just means "person who thinks about things and whose writings are respected" then, sure, Yudkowsky and Alexander are "theologians", but then literally every field will have its "theologians". Donald Knuth wrote some big books that everyone respects, and sometimes the word "bible" is attached to them! Clearly he's a theologian and computer science is just another religion! (Actually, Knuth is a theologian, a little bit; see https://archive.org/details/316bibletextsill0000knut.)

"sacred texts": see above.

"heresies (logical fallacies)": heresies are doctrines that arise within a particular religion and are officially rejected by it; to anyone outside the religion they're an irrelevance. Logical fallacies are definitely not that. If someone mixes up "A implies B" and "B implies A" or something, you don't need to be an internet-rationalist to think they're wrong or to criticize their argument.

You might be able to make an argument that Roko's basilisk is an example of a heresy within rationalism? But I don't think the existence of inside-baseball debates where some positions are officially deprecated is terribly specifically religious. The idea that goto statements are a great way of structuring a piece of computer software would be a heresy, for instance. Belief in "cold fusion" would be another. But computing and physics aren't religion-like in any useful sense on that account.

I should maybe clarify that when I say "I see this said all the time" I don't mean specifically the version targeted at internet-rationalists. There's the version aimed at singularitarians (not all internet-rationalists are singularitarians nor vice versa) and a slightly different version claiming that atheism is just another religion, and so forth. The general template is: Thing X is just another religion / just warmed-over Christianity, because here are half a dozen things about it that have some loose resemblance to some religious thing. I find such things uniformly unconvincing; it seems to me that one can draw loose parallels between almost anything and almost anything else.

And: where are the rationalists', or the singularitarians', or the atheists', sacred places? what are their regular religious rituals? what organization corresponds for them to churches and the like? what things are they obliged to believe, as opposed to believing by definition as a member of whatever group? what ancient traditions do they preserve or claim to preserve? Of course you can have a religion without any single one of those things, but the more common features of religion a thing doesn't have the less plausible it is to call it a religion.

I've never seen an argument like this that if applied consistently wouldn't equally assert that many academic disciplines, large companies, political movements, etc., are "religions".

I'm not going to get drawn into a thousands paragraph debate about rationalists belief, so this will be my only reply and I'll welcome your rebottle.

Your evidence isn't very convincing. I think the singularity is actually framed as a battle between good and evil. How else could Sam Altman be accused of betraying humanity if what his company does isn't framed as that? Moreover, that quote pretty much proves what I'm saying about tech CEOs, how the rationalists view them. There's no "betrayal" here, Sam is and always has been a craven capitalist. One would only feel betrayed by him if they believe in him as someone who violated an implicitly shared dogma.

And that's the big reason for the fallout you're identifying. Early on people like Sam Altman and Elon Musk pretended to be saving humanity through their investments and corporations. It was a marketing ploy and rationalists bought into it hook line and sinker. Of course it was always a ruse, of course these guys were never going to deliver, because at the end of the day their goals were to become rich. Which they achieved. So now the world-saving facade is dropped, and the rationalists who bought into it are none too pleased. Even this line:

"Elon is widely known to be a strong engineer, as well as a strong designer, and spends much of his time arguing details of that kind of work with his reports. Mark Zuckerberg is known to do similarly."

Is very flattering of Musk's and Zuckerberg's supposed engineering prowess. One only need look at the Cyber Truck to understand what kind of designer and engineer Musk is.

> Clearly he's a theologian and computer science is just another religion!

I mean you could take it in that direction. Various cults like the cult of Apple that flourished around Steve Jobs, who himself was a bit of a cult leader. There's a cult around Elon Musk. The thing that's missing is the deity of AI, which is unique to the rationalist community. You might even be able to say that academia is a religion that's formed around to worship science, and there's a lot of parallels between religion there too, down to the dresses and funny hats.

> every field will have its "theologians"

They're theologians when they talk of theology rather than theory. Knuth is a practician. People talking about the singularity are like the guy on the street saying the end is nigh, and people writing blogs about it are essentially priests saying the same thing from a pulpit, the equivalent of theologians are the ones writing scientific papers about it.

> If someone mixes up "A implies B" and "B implies A" or something, you don't need to be an internet-rationalist to think they're wrong or to criticize their argument.

That's not exactly what I'm talking about, logical fallacies are fine and you should point them out when it's relevent. What I mean is for example if you're talking to a rationalist, and you make an appeal to emotion, that's regarded as an invalid argument. Or an even better example is when you make an appeal to authority when it matters, and you get into an argument about whether it's valid or invalid ad hominem. In normal discourse, people make emotional and authoritative arguments and those are valid to consider. In rationalist communities, that's considered a logical fallacy, and the discussion is over. So more often than not when you're talking to a rationalist, the conversation devolves into a debate about which fallacies are being applied and whether they're valid or invalid, because the only thing they'll consider is logos. The doctrine here is that pathos and ethos are invalid forms of persuasion. This is why they're so comfortable rejecting expertise, and why it was so easy to be fooled by people like Musk who used good arguments but lacked actual expertise.

  • I'll attempt to be brief.

    If you think it's likely that there will be a singularity and it will be bad, then you'll regard someone working toward it as bad. That doesn't mean that there's any good-versus-evil-ness about the singularity itself. Whereas Armageddon is specifically a battle of good against evil, the Last Judgement is specifically a separation of people into (something like) good versus evil, etc.

    The "betrayal" line is saying that OpenAI was founded as, supposedly, an ethical company trying to make sure that AI worked out for the good of humanity, and Altman is failing to live up to that. It doesn't have anything to do with any high view of Altman personally.

    I agree that that line is more positive about both Musk and Zuckerberg than either deserves. But the claim wasn't "some rationalists think too highly of tech CEOs", it was "rationalists think about tech CEOs like religious people think about their high priests", and the point is that nothing I've found suggests that that's so. Sometimes people say nice things about (say) Mark Zuckerberg, sometimes not so nice (there was one comment where someone suggested that a different approach to AI safety might have resulted in "Mark Zuckerberg stamping on a face for ever") but no more of it than in the world at large is looking reverently to them as authorities.

    If you're happy also saying that computer science and academia in general are religions, then fair enough, but at this point I think saying "X is a religion" is no longer really saying anything.

    I don't know whether you're right or wrong in saying that rationalists are too unwilling to accept arguments that are logically invalid but persuasive in other ways, but this seems a long way from saying that logical fallacies are to rationalists as heresies are to (say) Christians.

    • > but this seems a long way from saying that logical fallacies are to rationalists as heresies are to (say) Christians.

      Logical fallacies do nothing to show an argument is wrong.

      All they do is demonstrate that a chain of reasoning cannot be a logical proof that the conclusion must be true, given its premises.

      Dismissing arguments because "they're fallacious" is actually not so far removed from dismissing ideas because they're heretical.

      In each case, what matters is whether the ideas are true, not whether your interlocutor proved rigorously it is true.