Comment by vachina
19 days ago
> This seems like it's fixing the symptom rather than the underlying issue?
Exactly. LLM does not know how to use a debugger. LLM does not have runtime contexts.
For all we know, the LLM could’ve fixed the issue simply by commenting out the assertions or sanity checks and everything seemed fine and dandy until every client’s device catches on fire.
And if you were to attach a debugger to a SOTA LLM, give it a compute environment, have it constantly redo work when CI fails, I can easily imagine each of these PRs burning hundreds of dollars and still have a good chance at failing the task.
This was my latest experience of using agents. It created code with hard coded values from the tests.