Comment by cma
5 hours ago
I would also recommend God and Golem, Inc., A Comment on Certain Points Where Cybernetics Impinges on Religion:
https://monoskop.org/images/1/1f/Wiener_Norbert_God_and_Gole...
Norbert Weiner was an atheist but he talks about three areas religion is the only thing to have really examined that relate to capable AI: omniscience, omnipotence, and worship (gadget worship). It has very prescient stuff on blackbox learning/distillation, reinforcement learning/reward hacking, alignment through human feedback.
His The Human Use of Human beings and Cybernetics are extremely good too and have more of a mash of the themes between Rerum Novarum and Magnifca Humanitas, and more near-term automation.
Awesome, thx for posting. I now have my new next book to read. Been wanting to read more of the original cybernetics stuff.
Would you also recommend "The Human Use of Human Beings"?
Yes, it starts slow with a lot of history of Cybernetics stuff through Leibniz and stuff (kind of prescient given chain rule -> back propagation, and control theory's relevance to optimizers). It is about twice as long as God & Golem and covers a lot of the same plus more examination of automation and human augmentation. I think either it or Cybernetics also goes into mass communication with some relevance to how social media played out.
God & Golem is the most succinct and up to date though, probably a 2hr read.
The book Cybernetics is a lot of math and ergodic theory stuff that went beyond me, but is the longest and you still get a lot out of it skimming over that stuff if you don't have the background for it. The last revision of it in the 50s added some of the same blackbox function copying/imitation learning/distillation stuff, reinforcement learning with reward hacking concerns, and superintelligence as genie/monkey's paw.
I would read the three in reverse order of publication.
He also foresaw another big area of potential existential danger, Wiener filter for guidance and control of missiles (later superceded with Kalman filter bringing the nuclear hard targets era with 15min retaliation windows) and refused to work on it or share prior work, and he also had bioweapons delivery concerns before the bioweapons treaties, publishing this open letter in The Atlantic in 1947:
https://archive.ph/D7BPt