Comment by hackandthink
5 days ago
Scary stuff from one of the winners:
"A Taxonomy of Omnicidal Futures Involving Artificial Intelligence"
(Jacob Tsimerman, Andrew Critch)
5 days ago
Scary stuff from one of the winners:
"A Taxonomy of Omnicidal Futures Involving Artificial Intelligence"
(Jacob Tsimerman, Andrew Critch)
Jacob Tsimerman believes that AI will be better than human mathematicians within 2 years.
We should not be surprised if AI solves the Millenium problems very soon and advances to a level that is barely comprehensible, or even incomprehensible, to the best humans.
We can expect this not by curve fitting to recent progress but by reasoning from first principles about where the progress has come from (synthetic data, non-human corpus) that has no obvious upper bound on capabilities.
The obvious and primary application will not be on showy millennium problems or long sought conjectures, but on optimizations and breakthroughs surrounding transformers.
We are doing fine so far with just Transformers, we have just crossed Trillion parameter models, we don't know if 100s of Trillion param model wont be better.
1 reply →
[dead]
Does anyone know if Tsimerman talked about AI extinction risk at other places? Critch has worked long-term in the field (MIRI, CHAI) but I was surprised to see this colab.
they are friends according to https://www.ams.org/journals/notices/202607/noti3372/noti337...
May all world-class mathematicians take a look at the AI alignment problem such that humanity can have a better chance of passing through, and may some of them decide to not focus just on sexy parts like coming up with ways to kill us all, and may those who do so anyway at least come up with more interesting or plausible stories than the ones presented in this paper.
This is a piece of sci-fi formatted with LaTeX so it looks like philosophy.
I think this says more that being very good at theoretical math does not at all translate into intelligence or subject matter experience about how humans will realistically handle potential armed conflict and potential risks of mass casualties, at the political/nation-state level.
Just by reading the abstract it's terrifying.
Ed Zitron should write his newsletter with LaTeX and publish them as PDFs in arxiv. Seems to make the techbros automatically take it seriously. Maybe that's what it takes to pop the bubble.
Whether or not Ed is right, he doesn't take any criticism, but peer review is a fundamental tenet of actual science. Vs being a blowhard on the Internet and getting paid for it.
Peer review is actually a surprisingly modern concept. It is not actually a fundamental tenant of science. The scientific method notably doesn't actually include peer review in it's core.
This would work better as a lesswrong post