Comment by camjw
3 years ago
Can you give a concrete example of this problem that you expect an LLM to not be able to solve? It's fine saying "abstract concept" and "problem it has not encountered before at all" but these seem to me quite fuzzy concepts.
Sure. Ask it how to replicate the payoff of a financial derivative. It can explain the concept but it cannot use it on a specific payoff to arrive at a correct replication (beyond the odd widely published stuff). Taking ChatGPT, it will, however, talk about generic stuff, some incorrect stuff and some unrelated things when probed.
Maybe also what I wrote a bit above: describe some greater than 3 dimensional objects and get it to stack them for some purpose could be another thing to try (I think, I will actually).