Comment by skylurk

But Bun also implements the interpreter to those tests— doesn't that mean it has a lot of leeway to over-fit to them? It's not as clean as "don't change these files" in this case.

In general I would agree though, and would take it one step further: the agent writing the source code should not even be able to see the tests, nor the agent writing the tests see the source code.