Comment by aesthesia

This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.