Comment by rvz
6 months ago
After all of that, every PR that Copilot opened still has failing tests and it failed to fix the issue (because it fundamentally cannot reason).
No surprises here.
It always struggles on non-web projects or on software where it really matters that correctness is first and foremost above everything, such as the dotnet runtime.
Either way, a complete disastrous start and what a mess that Copilot has caused.
Part of why it works better on web projects is the sheer volume of training data. There is probably more JS written than any other language by orders of magnitude. Its quality is pretty dubious though.
I have so far only found LlMs useful as a way of researching, an alternative to web search, and doing very basic rote tasks like implementing unit tests or doing a first pass explanation of some code. Tried actually writing code and it’s not usable.
> There is probably more JS written than any other language by orders of magnitude.
And the quantity of js code available/discoverable when scrapping the web is larger by an order of magnitude than every other language.
> Part of why it works better on web projects is the sheer volume of training data.
OTOH webdev is known for rapid framework/library churn, so before too long there will be a crossroads where the pre-AI training data is too old and the fresh training data is contaminated by the firehose of vibe coded slop.