Comment by magicmicah85
15 hours ago
> In fact, we explicitly prompt against this behavior in no uncertain terms, but the instructions – and the entire spirit of the task – are lost in the interest of making forward progress
LLMs and humans are quite alike. :) I notice that a few models will give up instead of ignoring their instructions and that's the model I would want working on tasks like this. An LLM should be able to categorize and reconcile transactions, but if it's not sure, it should quit and give it back to the humans.
> but if it's not sure, it should quit
Can it be sure or not? I've never been able to get LLMs to give confidence measures that match their actual outputs. I'll ask an LLM "Are you sure?" and it'll reply "Absolutely" when it's output is completely wrong, or it'll backtrack on a correct output with "I should not have provided an answer when I was unsure. Here is an answer I am sure of..." and then provide something completely wrong.
If they can't properly and consistently score their confidence, how do they "know" when to quit and give it back to the human?