Comment by andrewla

> When algorithms judge everything we do, we need to protect the right to make mistakes.

This is better expressed as saying when your behavior is reduced to metrics, you distort your behavior to match those metrics. An extension of Goodhart's law [1] to social behavior, as we become more capable of deriving metrics to assess social behavior.

[1] https://en.wikipedia.org/wiki/Goodhart%27s_law -- "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes" or "When a measure becomes a target, it ceases to be a good measure"