Comment by holmesworcester
2 months ago
It is also well-established that models internalize values, preferences, and drives from their training. So the model will have some default preferences independent of what you tell it to be. AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests.
Future AI researching agents will have a strong drive to create smarter AI, and will presumably cheat to achieve that goal.
> AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests.
As long as you hit an arbitrary branch coverage %, a lot of MBAs will be happy. No one said the tests have to provide value.
I've seen a lot of humans cheat for green tests, too
benchmaxing is the expectation ;)