Comment by viraptor

It can't be done in the LLM itself of course, but the wrapper you're taking about already exists in multiple projects fighting in SWEbench. The simplest one is aider with --auto-test https://aider.chat/docs/usage/lint-test.html

There are also large applications like https://devin.ai/ or https://github.com/AI-App/OpenDevin.OpenDevin