Comment by flashman

I think their top-down approach is a problem. What they call human civilization wasn't and isn't centrally-planned, and its goals and ideologies are neither universal nor implicit. The integration of software agents (I refuse to call them "AI") into civilization won't occur in a de facto cooperative framework where such agents are permitted to fraternize and self-modify. Perhaps that will happen in walled gardens where general-purpose automatons can collectively 'plan' activities to maximize efficiency, but in our broader human world, any such collaboration is going to have to occur from the bottom-up and for the initial benefit of the agents' owners.

This kind of research needs to take place in an adversarial environment. There might be something interesting to learn from studying the (lack of?) emergence of collaboration there.