Comment by smallstepforman
4 hours ago
Today I asked Gemini to extract a table from an PDF appendix and create C++ data table with its contents. After 15 or so iterations with corrections and new mistakes, it eventually gave up. I was floored when it said “I’m sorry, I cannot do this simple task, I’ve exceeded my error threshold and cannot do this task for you. My LLM prediction engine invents data instead of doing a simple data copy/reformat”.
Stunned to see that Gemini threw its digital arms in the air and gave up.
It's extremely hit or miss. I've had it one-shot a pretty decent analytic prototype from a brief description, but also had it get trapped in hour-long back and forth regression hell over incredibly simple things like adding a static favicon (ie it would add it, then keep taking it away with every subsequent iteration, breaking something else every time it was asked to put the favicon back etc.).
That's better than the loop grok got stuck in trying to use git and push the work it did leading to a $15 api credit deduction.
Getting AI/ML to acknowledge "I don't know" is such a challenge.
This is why the world model approach is so important. It allows you to feed back the prediction accuracy of the model to itself at training time, enabling it to predict (to some degree) its own uncertainty. If you jump through a couple of hoops you can also do this at run time to give it “spidey sense” that something’s not right with current inference.
Not true regarding ML, most ML methods support RMSE even if they are non parametric methods.
1 reply →
Tabula + Excel could probably do it quicker.
My go-to for this is to screenshot and use the built-in text extraction in the screenshot tool (I'm on a mac), then pass on that text data to whatever processing. It's a pretty good tool so long as the PDF is in OK shape (I've had errors in scanned images).
It's so horrible that in 2026 people are still publishing important data and specifications in a format like PDF that's difficult for LLMs to consume. We need to drag them kicking and screaming to HTML or Markdown. Heck, even Microsoft Word DOCX is superior for reliable parsing and content extraction.
That's interesting because my experience has been almost the opposite. A few months ago I tested Gemini on converting screenshots of tables from PDF files into CSV. I tried it on several different tables and it got every one right. It consistently outperformed ChatGPT.
anyone who has used both knows this is inaccurate or dishonestly stated (ie. you were using gpt nano or some nonsense)
Years ago, I used Acrobat to extract tables from a PDF. Had to do it manually, but it pasted nicely into Excel.
You didn't say whether you were using the App but the App's performance seems to be severely throttled compared to API.
You should just have it OCR a screenshot of the PDF that would probably work better
I haven't heard any accounts of it doing that since Gemini 2.5, but it was pretty easy to get it to do it with a programming task back then after a few failed attempts. Very interesting to hear it'll still do it.
We've been quite impressed with GCP Document AI. Not sure if it has a free tier but perhaps that's where Google is putting all the good OCR.
I envy you that it admitted that rather than simply making up data and lying about it.
[dead]