Comment by JoshTriplett

There are already such "APIs", which can be operated by a combination of textual communication and money. Or by illicit security vulnerabilities. You might notice that LLMs are pretty good at that now.

We're building something that has the capabilities of humans. There is no X for which it's persistently safe to assume humans can X and AI cannot X.