Comment by keeda

I can't understand the stance that we do NOT have extraordinary evidence. I cannot stress enough that just ~4 years ago the concept of general purpose AI models that can do everything they are doing today was pure sci-fi.

And since then they have grown even more powerful than they were predicted to be, which, note, also faced a lot of skepticism at the time. The Hugging Face hacks and recent steamrolling of longstanding Math problems are just two recent pieces of extraordinary evidence.

And worse, people trust this technology because it behaves like people, but it actually works in ways nobody really understands, even exhibiting deeply weird and even disturbing characteristics (https://news.ycombinator.com/item?id=49635518) -- each of those quirks is extraordinary in itself.

And now we're rushing to give it control over the real world while deploying this powerful, quasi-chaotic technology in an infinite variety of ways everywhere in this highly vulnerable society.

I don't know what the standards for "extraordinary evidence" should be, but given such extreme unpredictability and rapid change, I fear it may end up being "an actual catastrophe".