Comment by cookiengineer

Unit tests don't test for branch coverage.

That's the culprit, because LLMs tend to forget and remove a lot of branch logic in these kinds of tasks. If unit tests don't cover these specific if/elseif/else cases, then they'll just disappear.

They'll also disappear if the LLM is allowed to modify the unit tests, because they sure like to cheat their way around into greenlit test suites. The agentic environment must disallow write access to the unit test files for the agent that writes the code.

If you implement that in your tools, you'll see quickly how the models will try to rewrite the unit tests at all cost, no matter what kind of prompting you've done. Tool policies are the only boundary to successfully guarantee this.

Source: Am building my own agentic environment because of that behavior