Comment by kalkin
3 years ago
Thanks for this clarification.
I think the post conflates "fast takeoff" and "any existential risk to worry about from AI" a bit, which is fair enough since Bostrom does the same. Some of the arguments apply just to the former, some to both.
But especially if it turns out that LLMs are a meaningful piece of the puzzle to AGI, we might be living in a slow-takeoff world. And yet that doesn't mean there's nothing to worry about, IMO. We have a bit more time to figure out how to align AIs with a slow takeoff, but we still have to do it. Even in our current world, deep learning capabilities seem to be advancing a lot faster than our ability to understand how deep learning models make decisions. And even if we did develop the theoretical capability to align models, we have to actually use it. Seems unfortunately plausible that by default we instead give the first superintelligent models directives like "just make Facebook market cap number go up" - or maybe we make the first corporate models very conservative but then someone leaks the weights and open sourcers tell a superintelligent model "please destroy humanity" just for the lulz. If a misaligned model is only a little bit smarter than us (because we're assuming slow takeoff), we probably still have a shot at beating it and saving ourselves - but I'm not sure how much to count on that, given our inability to control even complex institutions actually made of people, and the advantages that an AI with otherwise-human-equivalent reasoning capability gets by default (ability to save & restore, copy/parallelize, speed up from hardware improvements, etc).
Even if AGI is never achieved, it could still be an existential risk.
Something significantly stupider than an average human, but that was 100% focused and 100% loyal could potentially be used by a very smart human in a way that effectively made them super-intelligent to compared to an unaugmented human.
Computers have approximately 100% perfect memory recall, vastly increased factual and numeric memory compared to humans, much better calculation, 100% focus and "loyalty". I've been wondering whether a computer recognising a face as someone from your contacts counts as you being super-intelligent (I don't think it does), or a spreadsheet adding up thouands of numbers (also no), an infinite ToDo list and calendar reminders (maybe?), spaced repetition learning (possibly?) and from there - what would count? What would it mean for you to be super-intelligent by machine augmentation? What would computer software which effectively increased human intelligence look and behave like? Surely not like a window with text input and clicky buttons...?
We'll be using our own AI to fight AI, and it will also be able to save, restore, and parallelise. I expect in the future security will be an important concern. Just like biology, it will be an ever shifting game.