Comment by lordnacho
7 months ago
Interestingly, one of my two big observations of LLM failure was also on an accounting task.
I thought it would be easy to do this, which is why I was surprised:
I had a folder full of bills, each of them with the VAT amount. Some were pictures, and some were PDFs. I asked for the total VAT for all 19 bills.
It took an immense number of prompts to get it to find the numbers correctly. It would get confused about reading the images as binary, that kind of thing. Or it would forget that it had to continue once it had found a few numbers. I got a total out in the end, but it took far too many prompts.
This is the only time I've come across a task a child could do that LLM failed at.
“ This is the only time I've come across a task a child could do that LLM failed at.”
Consider yourself lucky. It’s the people who haven’t run into something like this that will end up placing too much trust in these tools.