In my experience telling it what I want is not a reliable process at all whatsoever if what I'm asking for is sufficiently complex, no matter what context I provide. So instead I break tasks down into very small parts, ask for solutions to those that I can reasonably quickly assess and then put them together myself. Asking it to do the architectural or deep algorithmic legwork IME wastes so much time and is often just wrong.
There's a balance to be found here, that's unfortunately very hard to find at times.
In my experience, there are two classes of tasks: some are very "in-distribution", and for those LLMs can near-flawlessly perform the "architectural or deep algorithmic legwork", with maybe a single second round to fix the mistakes. For others, I have to break the tasks down myself, and often it's a "death through thousand papercuts", because the size of a task that I can quickly verify and the LLM will not screw up with > 50% probability is small enough that it's sometimes net negative time spent relative to doing it myself (and using LLMs only as glorified search engine and article summarizer).
I like to tell myself that I'm getting better at recognizing these two classes up front, but I'm still frequently surprised when "type 1" turns out to be "type 2".
But circling back to the main topic: with "type 2", agent instructions are paramount, if only to enforce the "small steps, pre-commit to scope and methodology, verification at the end, user doesn't even want to know about anything in between" rules, as agents naturally want to run ahead faster than I can keep up with.
A good practice to use (ime) is having it do research for the larger task, propose alternatives, and write that in a file. You can then review and comment that up, go through another iteration.
Then when it comes to implementation time, things typically go much smoother for larger changesets. Be wary to not overplan, as we all know how often we realized we missed something once we get into the details. Here, I stop the session and go back to iterating on the design/plan doc. Not a step-by-step guide, if you don't instruct them to the difference, they will just pseudo-implement in the plan like they do in their thinking traces, need to be be explicit about the level of detail.
this is probably outdated, attention typically stays fine up to ~200k tokens these days
you end up clouding that more with an agent having to re-understand concepts or conventions
AGENTS.md is good when it is a nested sparknotes for the project, you save context and turns overall, but keep them minimal and largely gotchyas or unusual workflows in your repo
You still need to define assumptions somehow. What you want and what the model wants will not match up by default.
In my experience telling it what I want is not a reliable process at all whatsoever if what I'm asking for is sufficiently complex, no matter what context I provide. So instead I break tasks down into very small parts, ask for solutions to those that I can reasonably quickly assess and then put them together myself. Asking it to do the architectural or deep algorithmic legwork IME wastes so much time and is often just wrong.
There's a balance to be found here, that's unfortunately very hard to find at times.
In my experience, there are two classes of tasks: some are very "in-distribution", and for those LLMs can near-flawlessly perform the "architectural or deep algorithmic legwork", with maybe a single second round to fix the mistakes. For others, I have to break the tasks down myself, and often it's a "death through thousand papercuts", because the size of a task that I can quickly verify and the LLM will not screw up with > 50% probability is small enough that it's sometimes net negative time spent relative to doing it myself (and using LLMs only as glorified search engine and article summarizer).
I like to tell myself that I'm getting better at recognizing these two classes up front, but I'm still frequently surprised when "type 1" turns out to be "type 2".
But circling back to the main topic: with "type 2", agent instructions are paramount, if only to enforce the "small steps, pre-commit to scope and methodology, verification at the end, user doesn't even want to know about anything in between" rules, as agents naturally want to run ahead faster than I can keep up with.
A good practice to use (ime) is having it do research for the larger task, propose alternatives, and write that in a file. You can then review and comment that up, go through another iteration.
Then when it comes to implementation time, things typically go much smoother for larger changesets. Be wary to not overplan, as we all know how often we realized we missed something once we get into the details. Here, I stop the session and go back to iterating on the design/plan doc. Not a step-by-step guide, if you don't instruct them to the difference, they will just pseudo-implement in the plan like they do in their thinking traces, need to be be explicit about the level of detail.
do you just manually type out your important instructions every time instead of being smart and putting a few lines in a text file?
this is probably outdated, attention typically stays fine up to ~200k tokens these days
you end up clouding that more with an agent having to re-understand concepts or conventions
AGENTS.md is good when it is a nested sparknotes for the project, you save context and turns overall, but keep them minimal and largely gotchyas or unusual workflows in your repo