Comment by samrus
12 hours ago
ive found that the shorter the "task horizon" the more time saved
essentially, a longer horizon increases chances of mistakes, increasing time needed to find and fix them. so at one point that becomes greater than the time saved in not having to do it myself
this is why im not bullish on AI agents. task horizon is too long and dynamical
So here's my problem, ultimately
If the task horizon for the LLM is shorter than writing it yourself, this likely means that the task is well defined and has an easy to access answer
For this type of common, well defined task we shouldn't be comparing "how long it takes for the LLM" against "how long it takes to write"
We should be comparing against "how long it takes to find the right answer on SO"
If you use this metric, I bet you the best SO answer, which is also likely the first google result, is just as fast as the LLM. Maybe faster