Comment by Tade0
17 days ago
My setup has some of the things mentioned and I found that occasionally the LLM will lie that something passes, when it doesn't.
17 days ago
My setup has some of the things mentioned and I found that occasionally the LLM will lie that something passes, when it doesn't.
Yup I have run into the same.
I use a pre-commit hook to run `pnpm check`. I missed mentioning it in the original comment. Your reply reminded me of it and I have now added it. Thanks.
That's something I find to be incredibly frustrating. I have to keep reminding it that we're not done, no matter how much I enforce that the lints must pass before we're done.
Make the error message much more dramatic and it will be less likely to miss it. Create a wrapper if you can't change the error message.
Remember these are still fundamentally trained on human communication and Dale Carnegie had some good advice that also applies to language generators.
If you're using Claude try the hookify plugin and ask if to block commits unless the rules pass.