Comment by olalonde
3 hours ago
The more frontier models progress, the less concerned I am about this. Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power. Ego evolved alongside intelligence in nature because of natural selection, but it doesn't appear to be an inevitable emergent property of intelligence.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
That’s because these aren’t intelligent like you, I, or even a dog. They are mathematical models formulating likely responses with no agency or drive. They appear to be intelligent, but they are the epitome of “fake it till you make it”.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.