← Back to context

Comment by bhaak

18 hours ago

It passed all the tests.

If you can't trust your test suite to catch an automatic language translation you shouldn't trust it at all. :)

12 comments

bhaak

Reply
user142  17 hours ago

Tests can only prove the presence of bugs, but not their absence. If the AI can access the tests, it can easily make them pass by just adding additional if statements. It doesn't mean the code is actually correct.

andrewflnr  17 hours ago

What if we only trusted the test suite a reasonable amount, instead of pretending trust must either be blindly total or nonexistent?

solid_fuel  8 hours ago

The entire underlying system has been replaced. The test suite is written around the current fuzzy edges and past problem areas, not every single behavior of the existing platform.

"If you can't trust your test suite to catch a hardware floating point arithmetic bug, you shouldn't trust it at all."

"If you can't trust your test suite to catch a JVM bug, you shouldn't trust it at all."

"If you can't trust your test suite to catch a recurring memory error, you shouldn't trust it at all."

debugnik  18 hours ago

It also modified many of the tests to make them pass in mischievous ways. You can't trust a test suite to catch regressions if the new version doesn't use the same test suite.