Comment by AnthonyMouse

> I also disagree it would only take revising incentives. Such an AI would be shut down before it gets anywhere.

That's not how it works. The theory is that the thing is good at what it does. (The ones we have aren't very good, but then it doesn't matter either way.)

If it's good at what it does then it takes that into account. It says, propose a law to adopt score voting in all the states where it would pass. It passes in states representing a third of the population. Half the Republican seats in California go to the libertarians instead, the Democrats lose some seats in Pennsylvania to a new party that wants more anti-trust enforcement because the farmers are pissed off about not being able to fix their tractors, etc.

None of the entrenched interests strongly opposed the change because it had no obvious direct effect on them and some of them even benefited from it, e.g. the tech companies have more influence in California and prefer libertarians to Republicans. But now you have a bunch of libertarians in Congress that the Republicans need for a majority, and they want to actually get rid of anti-competitive healthcare regulations instead of just paying lip service. Now the Democrats need the party demanding real anti-trust enforcement.

By the time they figure out what the change is going to do, it's already done. And it could do multiple things like that at once.