Comment by malfist
2 days ago
> nearly as accurate as a human book keeper
Anything to avoid using the metric system.
Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.
Hey, author of the blog post here.
I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.
It’s a service that provides an AI bookkeeper so it’s a pretty relevant metric.
You'd care if you had a human bookkeeper and you were considering replacing them with this company's AI bookkeeper.
Why would I want both a less accurate book keeper and to incur all of the liability of doing the books myself?!
It's not less accurate. As commented above, the control knew they were being tested against the machine, so made sure to be super careful.
In everyday life the human is less careful, and the machine costs 1% of the human.
Presumably your current book keeper is not your slave, and you have to pay them...
7 replies →
Because it's way cheaper. Not saying it's a good tradeoff, but that appears to be their pitch.