Comment by edbaskerville

The literature on the evolution of cooperation, focused around computational thought experiments with iterated prisoner's dilemma, seems relevant here, e.g.,

https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation

If you allow a population of individuals repeatedly playing prisoner's dilemma against each other to evolve their own strategies, you end up with a large percentage of the population cooperating with each other by default, but punishing cheaters after they are observed cheating. But a small percentage of cheaters will always persist, because as the number of cheaters goes down, the number of naive cooperators will go up, thus making it more advantageous to cheat.

In evolutionary jargon, cheating behavior undergoes "negative frequency-dependent selection". And you end up with a low, but nonzero, equilibrium frequency of cheaters.

This outcome here depends on the order of rewards/costs: the best outcome comes from cheating on a cooperator; next best is cooperating with a cooperator; then cooperating with a cheater; and worst is two cheaters cheating on each other.

It's a caricature, but the evolutionary dynamics seem to map pretty well to the kind of examples people are bringing up here in the comments.

(The actual "prisoner's dilemma" is rather a confusing story to use, because it's about criminals trying to decide whether to cooperate with each other or betray each other to avoid jail time. So you end up talking about the evolution of cooperation among a population of criminals.)