Comment by achou
8 hours ago
Y'all are sleeping on custom lint rules.
Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you. Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that explains entire architectural principles, concurrency rules, etc. Stuff that is very tailored to your codebase and are far more precise than a generic compiler/lint error.
Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it".
To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if "is variable XYZ unused?" or "does variable X oveshadow Y" or other more "esoteric" lint rules will not catch it. Likelihood is high that the LLM just hallucinated some false positive lint rule anyways giving you a false sense of security.
> static code analysis which by definition does not find runtime bugs
I'm not sure if there's some subtlety of language here, but from my experience of javascript linting, it can often prevent runtime problems caused by things like variable scoping, unhandled exceptions in promises, misuse of functions etc.
I've also caught security issues in Java with static analysis.
The usefulness of using static code analysis (strict type systems, linting) versus not using static code analysis is out of the question. Specifically JavaScript which does not have a strict type system benefits greatly from using static code analysis.
But the author claims that you can catch runtime bugs by letting the LLM create custom lint rules, which is hyperbole at least and wrong at most and giving developers a false sense of security at worst.
1 reply →
Anthropic ships an official plugin to create linters for you based on your Claude Code history or instructions, it’s great. You can vibe code your lint rules per repo.
https://github.com/anthropics/claude-code/tree/main/plugins/...
Yeah, I have written multiple almost completely-vibecoded linters since Claude Code came out, and they provide very high value.
It’s kind of a best case scenario use-case - linters are generally small and easy to test.
It’s also worth noting that linters now effectively have automagical autofix - just run an agent with “fix the lints”. Again, one of the best case scenarios, with a very tight feedback loop for the agent, sparing you a large amount of boring work.
Clever! Sharing my lightning test of this approach.
Context - I have a 200k+ LOC Python+React hobby project with a directory full of project-specific "guidelines for doing a good job" agent rules + skills.
Of course, agent rules are often ignored in whole or in part. So in practice those rules are often triggered in a review step pre-commit as a failsafe, rather than pulled in as context when the agent initially drafts the work.
I've only played for a few minutes, but converting some of these to custom lint rules looks quite promising!
Things like using my project's wrappers instead of direct calls to libs, preferences for logging/observability/testing, indicators of failure to follow optimistic update patterns, double-checking that frontend interface to specific capabilities are correctly guarded by owner/SKU access control…
Lots of use cases that aren't hard for an agent to accurately fix if pointed at directly, and now that pointing can happen inline to the agent work loop without intervention through normal lint cleanup, occurring earlier in the process (and faster) than is caught by tests. This doesn't replace testing or other best practices. It feels like an additive layer that speeds up agent iteration and improves implementation consistency.
Thanks for the tip!
I disagree. A ton of linting is cosmetic stuff, you need more specific guardrails, even though linting in general is helpful.
I realized this recently and I've been creating a RuboCop plug-in[1] to automatically have the LLM code better match my personal style. I don't think it'll ever be perfect, but if it saves me from moving a few bits around or adding spacing I'd rather see then it's wroth. The fun part is I'm vibe coding it, since as long as the tests verify the rules then it doesn't really matter much how they work. As a result adding a new rule is pasting in LLM generated code followed by what I'd prefer it look like and asking it to add a rule.
[1]: https://github.com/tristandunn/rubocop-vibe/
I just discovered https://megalinter.io/
Ha, I just had the LLM create my first custom eslint rule yesterday and was thinking that I should make more.
could you explain what this does ? like i ask it to write a testcase (pytest, etc). why is a linter better ?
I like this idea but I can’t think of a concrete example to ground it. Can anybody share a real example?
I got tired of brittle literals like `http://localhost:3000` and `postgres://…@127.0.0.1/...` creeping into my code, so I wrote a few ESLint rules that detect “hardcoded infrastructure” strings and ask the agent to find the constant in the codebase — not by guessing its name but by `grep`-ing for its value.
The detection is based on dumb string literal heuristics, but has proven rather effective. Example patterns:
const hardcodedInfrastructure = { url: /^https?:\/\/(localhost|127\.0\.0\.1|192\.168\.\d+\.\d+|10\.\d+\.\d+\.\d+|172\.(1[6-9]|2\d|3[01])\.\d+\.\d+)(:\d+)?/i, dbUrl: /^(postgresql|postgres|mysql|mongodb):\/\/.*@(localhost|127\.0\.0\.1|192\.168\.\d+\.\d+|10\.\d+\.\d+\.\d+|172\.(1[6-9]|2\d|3[01])\.\d+\.\d+)/i, localhost: /^localhost$/i, localhostPort: /^localhost:\d+$/i, };
Claude Code is obsessed with using single letter names for inline function parameters and as loop control variables. I don't like it and I think it is sloppy, so I told it to stop in CLAUDE.md. In my experience, Claude Code will respect CLAUDE.md around 70 % of the time, it seems to cherry pick areas that it will respect more and less often and of course it kept ignoring this instruction. So I told it to add a pre-commit hook and invoke the TypeScript compiler and analyze the AST for single-letter variable names and tank the pre-commit check when it detects one with an error message indicating the offending symbols' locations. Now it can be non-deterministic as much as it wants, but it will never commit this particular flair of slop again as the adherence is verified deterministically. I already have a few more rules in mind I want to codify this way to prevent it from reproducing patterns it was trained on that I don't like and consider low quality.
Max 200/300 LOC per file is pretty popular.