Comment by lionkor
6 days ago
The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details.
Then it can write tests based on the interface, and that is usually a lot more solid.
That's yet another band aid on a hemorrhaging patient. And it still assumes the agent(s) are capable of correct comprehension of the purpose.
Yeah for sure it's a bandaid fix, but it does produce tests that fail if the interface doesn't match the implementation, more often than not doing this does (in my experience).
With a skill like this, which I have written, the LLM will end a session with "wrote 24 tests, 5 failing due to API contract mismatch" or something, and leave it at that. I can then decide if that's a bad interface or bad implementation.