Comment by smallstepforman
3 hours ago
Today I asked Gemini to extract a table from an PDF appendix and create C++ data table with its contents. After 15 or so iterations with corrections and new mistakes, it eventually gave up. I was floored when it said “I’m sorry, I cannot do this simple task, I’ve exceeded my error threshold and cannot do this task for you. My LLM prediction engine invents data instead of doing a simple data copy/reformat”.
Stunned to see that Gemini threw its digital arms in the air and gave up.
That's better than the loop grok got stuck in trying to use git and push the work it did leading to a $15 api credit deduction.
Getting AI/ML to acknowledge "I don't know" is such a challenge.
Not true regarding ML, most ML methods support RMSE even if they are non parametric methods.
That's interesting because my experience has been almost the opposite. A few months ago I tested Gemini on converting screenshots of tables from PDF files into CSV. I tried it on several different tables and it got every one right. It consistently outperformed ChatGPT.
anyone who has used both knows this is inaccurate or dishonestly stated (ie. you were using gpt nano or some nonsense)
You didn't say whether you were using the App but the App's performance seems to be severely throttled compared to API.
You should just have it OCR a screenshot of the PDF that would probably work better
We've been quite impressed with GCP Document AI. Not sure if it has a free tier but perhaps that's where Google is putting all the good OCR.
I haven't heard any accounts of it doing that since Gemini 2.5, but it was pretty easy to get it to do it with a programming task back then after a few failed attempts. Very interesting to hear it'll still do it.
I envy you that it admitted that rather than simply making up data and lying about it.
[dead]