Comment by hosh
2 years ago
Related to this is Promise Theory - https://en.wikipedia.org/wiki/Promise_theory
Promise Theory goes well beyond trust, by first understanding that promises are not obligations (promises in Promise Theory are intentions made known to an audience, and carries no implied notions of compliance), and that it is easier to reason systems of autonomous agents that can determine their own trust locally, with the limited information that they have. Promise theory applies to any set of autonomous agents, both machines and humans.
Back when this was formulated, we didn't have machines capable of being fully autonomous. In promise theory, "autonomous agent" had a specific definition, that is, something that is capable of making its own promises as well as determine for itself, the trustworthyness of promises from other agents. How such an agent goes about this is private, and not known to any other agent. Machines were actually proxies for human agents, with the engineers, designers, and owners making the promises to other humans.
AI that is fully capable of being autonomous agents under the definition of promise theory would be making its own intention known, and not necessarily as a proxy for humans. Yet, because we live in a world of obligations, promises, and we haven't wrapped our heads around the idea of fully autonomous AIs, we will still try to understand issues of trust and safety in terms of the promises humans make to each other. Sometimes that means not being clear in that something is meant as a proxy.
For example, in Schneier's essay: "... we need trustworthy AI. AI whose behavior, limitations, and training are understood. AI whose biases are understood, and corrected for. AI whose goals are understood. That won’t secretly betray your trust to someone else."
Analyzing that with Promise Theory reveals some insights. He is suggesting AIs are created in a way that should be understood, but that would mean they are proxies for human agents, not as an autonomous agents capable of making promises in its own right.
Tangentially, I am reminded of the essay "The Problem with Music" by Steve Albini [1]. There is a narrative in this essay about A&R scouts and how they make promises to bands. The whole point is that the A&R scout comes across as trustworthy because they believe what they are promising. However, once the deal is signed and money starts rolling in, the A&R guy disappears and a whole new set of people show up. They claim to have no knowledge of any promises made and point to the contracts that were signed.
In some sense, corporations are already a kind of entity that has the properties we are considering. They are proxies for human agents. What is interesting is that they are a many-to-one kind of abstraction where they are proxying many human agents.
It is curious to consider that most powerful AIs will be under the control of corporations (for the foreseeable future). That means they will be proxying multiple human agents. In some sense, AI will become the most effective A&R man that ever was for every conceivable contract.
1. https://thebaffler.com/salvos/the-problem-with-music