Comment by refurb
9 hours ago
I’m curious how this increased throughput happens.
You’ve accurately stated that AI isn’t as rigorous as a trained attorney. Doesn’t that mean that every single datapoint must be confirmed by a human?
How is that quicker than just using a human to read the content and make the call? Data entry savings?
Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd.
It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).
Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.
Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
Does the human find enough bugs that they stay on guard, or just rubber stamp everything without really looking at it? It’s hard to stay vigilant when stuff looks plausible.
Complete speculation: you could instruct Claude to hide one random mistake in every document.
And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database.
A review that doesn't find the mistake counts as invalid.
1 reply →
Don't forget that humans have a not insignificant error rate when copy/pasting or copy/typing data.
And it's possible to run each document through the LLM pipeline multiple times, using different models and/or prompts each time, to check for errors and inconsistencies. That will take more time and cost more, but it can reduce the error and hallucination rate significantly.
I'm doing some public court records processing for bankruptcy cases (interested mostly to seek out corruption in big national cases), and yes, the "variousness" of random PDFs is exactly the issue. Trying to get the cost for a whole case down to a minimum.
Sample is around 300 court dates, shy under 1k files.
At best I'm building a claude skills file.
Thanks for sharing the details. Does the attorney check that the AI copied the data accurately? Or is it just assumed to be correct?
Your experience mirrors my own. AI is great for parsing data that can take up a huge amount of time. My only concern is whether or not it’s done accurately. I wouldn’t use it for anything where mistakes cause serious consequences.
> Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.
Nobody working there has the requisite experience to do this in a reasonable amount of time. These are not particularly tech-savvy folks, Claude use aside.
1 reply →
You don’t need a trained attorney to schematize data. The LLMs are used to make the data easier to understand and manipulate.
They'll also hallucinate and change meaning in the process of extraction and "schematization"
Not necessarily. Depends how you use it.
"Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.
11 replies →
Doing similar-ish things with Claude, it's helpful to have something to ground it.
For instance, if you can say:
"Refer to the database schema in x.sql as your source of truth for the database structure we want to import into*. Do not invent data, tables or columns that do not exist. Carefully match all output against this database schema and do not create output that doesn't exist if it does not match the schema, simply skip it."
You will end up with a far better result in my experience.
Gotta treat it like a child.
1 reply →
They do, sometimes. That's why the review still has to happen.
But now it's comparing already filled columns on a spreadsheet, not copy-pasting every single thing from an (often uncopyable) PDF.
1 reply →
Reviewing something takes less time than producing it.
I...don't think that is universally true.
Few things are. It also doesn’t require standards. “Stamp this diff” culture is everywhere even before AI. A stamp is literally easier than anything else.
Whether that is useful measurement I suppose depends on the circumstances.
Yes, but empirically in this case, it seems to be true.