Comment by 8-prime
6 days ago
This is something that I have noticed as well. As soon as you venture into somewhat obscure fields, the output quality of LLMs drastically drops in my experience.
Side note, reverse engineering SAP ABAP sounds torturous.
The worst part isn't even that the quality drops off, it's that the quality drops off but the tone of the responses don't. So hallucinations can start and it's just confidently wrong or even dangerous code and the only way to know better is to be better than the LLM in the first place.
They might surpass us someday, but we aren't there yet.