Comment by amelius
1 year ago
Maybe we can use LLMs to automatically bring these third party libs up to par?
Could be a nice showcase project for Copilot.
1 year ago
Maybe we can use LLMs to automatically bring these third party libs up to par?
Could be a nice showcase project for Copilot.
> Maybe we can use LLMs to automatically bring these third party libs up to par?
So, I actually tried this. I tried to use copilot to help generate type stubs for a third party library, hoping to be pleasantly surprised.
Copilot generated reasonable-looking type stubs that were not close enough to correct to be of any value. Even with the full source code in context, it failed to "reason" correctly about any of the hard stuff (unions, generics, overloads, variadics, quasi-structured mappings, weird internal proxy types, state-dependent responses, etc. etc.).
In my experience, bolting types onto a duck-typed API always produces somewhat kludgy results that won't be as nice as a system designed around static typing. So _of course_ an LLM can't solve that problem any more than adding type stubs can.
But really, the answer to "will LLMs fix $hard_problem for us?" is almost always "no", because $hard_problem can rarely be solved by just writing some code.
What about GPT-4? I think that coupled with a decent RAG and agent systems, it would do a good job.
Ok, how does one set up "GPT-4 coupled with a decent RAG and agent systems"?