Comment by stephc_int13
2 days ago
I tend to share the sentiment of the author.
I think that coding assistants tend to quite good as long as what you ask is close to the training data.
Anything novel and the quality if falling off rapidly.
So, if you are like Antirez and ask for a Linenoize improvement that has already be seen many times by the LLM at training time, the result will seem magical, but that is largely an illusion, IMO.
Heads up that this is "more true" for non-reasoning LLMs. Reasoning gives an LLM a lot more runway to respond to out of distribution inputs by devoting more compute on understanding the code it's changing, and to play with ideas on how to change it before it commits.
But how many times a week are you truely doing something novel? A thing that nobody in the world (or the LLM training data) has done before?