Comment by hn_throwaway_99

I found this post particularly uninformed. A lot of folks saying "I don't see how that could happen" don't seem to understand the steps that have been laid out that detail how this occurs.

I really encourage folks to read the AI 2027 and related scenarios. You can definitely argue and disagree about the steps, but I feel like a lot of folks just don't even understand how this is plausible because they haven't read the arguments. Briefly:

1. All the frontier model companies are (or at least were) racing so that the AI models themselves build the next generation of models. This is not in debate.

2. The fear is that a misaligned model will essentially build the next, more advanced model with hidden goals. We literally already saw the danger of that in Hugging Face, where agents were deliberately trying to cover their tracks.

3. Nearly everyone believes as AI gets more powerful that it will be integrated into more physical world systems. Russia was already caught using Nvidia chips running AI powered drones that killed 3 people in Ukraine. The point is not that folks are using new tech to kill people, the point is that we're already putting AI into literal bombs.

I get it, before the Hugging Face incident I also thought all the prophecies about doom were just marketing speak. But now I see more hand-wavyness from the other side, oftentimes arguing against straw men like "AIs need to be like SkyNet and become sentient" to kill us, which is simply not how it works.