Comment by andrewljohnson
15 days ago
LLM editing should be done to produce deterministic output.
That is, the LLM should produce a diff, and the user should accept the diff. It seems like a bad pattern to just tell the LLM to edit any long document without that sort of visibility. Same goes for prose as for code.
I always thought it was a little weird that LLMs aren't sophisticated enough to surgically edit files as needed.
For example, if there is a code block that needs to be wrapped within another function call, it'll rewrite the entire function call and you'll just have to pray that the re-written code block wasn't subtly changed.
I _think_ so far it hasn't introduced any changes....
You can just look at the diff when you do a pull request, no prayer needed, and if you want it to be “surgical” in that way, your prompt (and agents.md) can be specific.
You can also unit test the function to better assure behavior didn’t change.
Indeed, that's what I do. I inspect the diff, though if it's an indentation change the entire block will be marked changed.
Still not an excuse to not read every line of course...
Unit tests give me the confidence that at least those tested logic paths are unaffected.
Sometimes with older codebases one cannot assume the paths have adequate test coverage.
1 reply →
This gets skipped because continual approvals break up user flow so we let LLMs make a few hundred line diffs then a user does a bulk review, and can just revert all/partially. It's naieve to assume user will review every LOC in every instance.
I’m fine with bulk review, it just has to get reviewed before a merge. You don’t need to review the LLM output as you work except as it aids you to work.