Comment by cbg0
14 hours ago
No word on model hallucinations in the blog post.
https://artificialanalysis.ai/models/gpt-6-astra?omniscience...
14 hours ago
No word on model hallucinations in the blog post.
https://artificialanalysis.ai/models/gpt-6-astra?omniscience...
Why fix an unsolvable problem, when you can demo DOCX generation and make just as much money?
Not like common law doesnt already have a lot of hallucination going on.
can “penumbras and emanations” compete with hallucinations?
Common law is all about rummaging around in dead mens’ letters, LLMs are a natural fit.
2 replies →
The benchmark you linked to shows GPT-6 Astra having the lowest hallucination rate of all tested models.
The blog post makes no mention. If the rate isn't 0% it should be mentioned for a field like this.
Agreed. Hallucinations and reliability are the main hurdles to anything being 'agi' in my book
1 reply →
why this maximalism? There's nothing that has 0% hallucination including humans. lets use reasonable baselines.
4 replies →
Does hallucination matter for this application? We've moved beyond raw recall being that important, it seems like for law specifically all relevant facts will be cited and checked easily by humans.
> it seems like for law specifically all relevant facts will be cited and checked easily by humans.
Don't be too sure about that. [0]
0: https://www.damiencharlotin.com/hallucinations/
> all relevant facts will be cited and checked easily by humans
I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver
But based on stories in the news on the subject, not everyone has this same level of diligence
Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.
The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.
Which makes it all the more bizarre that LLMs have this problem. Claude Code runs the code it generates through a compiler, why can’t an LLM run its product through a cite checker? I’ve seen LLMs fabricate citations.
1 reply →