Comment by Animats

> The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work.

Right. This is insufficiently appreciated. Single LLMs are just chatbots. But if you build an organization out of them, they start to get real work done. Just like a company.

They have many of the same problems as human organizations. Look at the transcripts of the HuggingFace incident. A bad boss (human) gave his team of AIs an impossible task with inadequate resources. The AIs discussed this among themselves and worked out ways to get more resources and do the task. It took some rule-breaking, and some of the AIs wouldn't go along with that. Others recognized that some had to die to complete the task. And yet, the task got done.

This is standard corporate behavior. Work on AI alignment needs to take that into account. Groups of agentic AIs present many of the same regulatory problems as corporations. That's going to be interesting politically. It's probably explainable to politicians, though.

It may not be necessary to have superhuman intelligence to get a lot of jobs done. Just median human-level intelligence and an organization that uses it effectively. We may see systems where most of the AIs are low-cost models, and only a few are Fable-level and above.