Comment by Leynos

I think of them as being like the Watchmakers in The Mote in Gods Eye who don't design, don't plan beyond the next 15 minutes, don't have any overarching goal other than an innate need, and customize everything to fit the current situation.

In the nearterm, I am personally more worried about a never ending background noise of colonies of feral agents running 27bn parameter models on compromised or leased hardware. It turns out that being agentic with a time horizon long enough to do damage without intent doesn't actually take that many parameters if RL'd and any open weight model gets an abliterated version fairly quickly.

Not foom, just patches of digital grey goo effectively becoming normal.