Comment by rogerrogerr
3 months ago
I’m convinced the magic bullet is deterministic checks. Linters, static analyzers, etc. Whatever you can do to create deterministic gates that the LLM simply must overcome to reach a “done” state, do it. Has been making a huge difference for my team, but sister teams are so invested in writing the perfect Make No Mistakes prompt that they just can’t see it.
Basically I treat it like a junior dev. We don’t get junior devs to write code correctly by cajoling them just right, we add CI gates. It still works.
> Whatever you can do to create deterministic gates that the LLM simply must overcome to reach a “done” state, do it.
First thing Gemini did when I tried that was turn off all the rules in eslint.config.mjs claiming they were "overly stylistic"
Yes, it got better once I explicitly told it not to disable any rules, so I accept I was holding it wrong but I do worry just how many footguns it puts into other things because I didn't know the right guardrails to give it.
Add hooks to block it from being able to edit the files you don’t want changed.
Why does your harness allow the LLM to generate output that can disable rules & checks?
I have no idea because Google Antigravity is closed source.
1 reply →
Wouldn't have helped, sibling comment: https://news.ycombinator.com/item?id=48797883
Architectural decisions are not lintable.
> Architectural decisions are not lintable.
I made a tool called ProjectLint to lint architectural decisions and anything that ESLint and ArchUnit and others like that wouldn't be a good fit for. It's not public yet, but is fairly rudimentary and based on Go + goja: https://github.com/dop251/goja
You write rules in ESLint, it makes sure that they're followed. If it's a file in your repo, it can obviously be checked. Now whether you can describe your architecture well enough to lint it (at least the stuff you care about enforcing), that's a different question.
In my case it's more like: "Oh, the AI messed up this pattern, but it will need to be followed N more times in the future, better write another projectlint rule." After a while, you can even copy them over between projects, as long as they're following the same conventions.
In the same spirit of https://www.archunit.org/
You can have a separate LLM as a brutal enforcer of an architecture decision. Works pretty well in the right harness.
Wider architectural guidelines, yes. "How a senior developer in this company would do it" — not so much.
1 reply →
It will burn up the tokens to get through the deterministic gates, more so when n order dependencies are involved in the mix. Enough typewritters and monkeys could get it done too.
I found I used fewer tokens by having many short prompts than long ones — because it spent many fewer tokens thinking and narrating.
Why aren't the teams using shared checks? Are the codes in different repos?
They’re very, very different projects.