Comment by Isamu
2 hours ago
To me the dream of AI is: an agent can do the drudge work of tracking every statement down to the supporting reference, documentation and ultimately the work involved in verifying.
It is NOT being used for this, instead we want faster “answers” that avoid due diligence. Same as always but lazier.
Agreed, and there is already a solution that’s better to support anyway. It may not be perfect, but Wikipedia operates on this model at scale at a much smaller annual budget than any of the AI companies and has produced much higher quality content overtime for all of humanities benefit than the AI teams which are most just scraping the work of others while spending most of their original work to replace human labor for their own financial benefit.
Wikipedia community bans AI-generated entries because verifying and removing AI hallucinations imposes even more workload on editors than writing articles from scratch [1, 2].
If you don’t agree that Wikipedia’s approach, one can always fork it like was done with SpaceX / Twitter did with its launch of Grokipedia on October 27, 2025, which hasn’t been really incorporating much improvement well with lots of digressions in its quality and reliability [3].
[1] https://en.wikipedia.org/wiki/Wikipedia:Writing_articles_wit...
[2] https://en.wikipedia.org/wiki/Wikipedia:Editing_policy#Artif...
[3] https://en.wikipedia.org/wiki/Grokipedia#Reliability
[Edited for better formatting]
I've had the same thought as probably have many others - what solutions exist in this space?
To put it into words: a layer over an LLM that tracks provenance: the documents it has access to, the facts it has extracted, the mathematical expressions and queries it's run and finally the artifacts and results it's yielded complete with receipts for exactly how it arrived at it's findings and conclusions.
I assume legaltech products have to do it like this. Is the general pattern implemented somewhere on github?
Not sure why a solution is even needed. If I had to compile all the resources I had consulted for something like C programming, they wouldn't amount to a single shelf. I would say the same thing for research where the amount of sources is still very low. It's kinda reminded me of the big paper manuals and books before the internet. I don't think the whole library of congress was needed to write UNIX.
So AI being a source of truth and information is not a something that important. It's a generator and we should use it as such, getting it to map domains and repeating patterns. It should not be used as an encyclopedia or oracle.
But soon, the majority of those references, documentation and work will be AI generated. We are peeing and likely defecating in our own water well.
I mean you can definitely do that - just set it up do to that.
What you are describing is that a majority of humans are inherently not interested in that. There are are a subset of people who would like to do that - and certainly can now that they have the tooling.
Its just a vastly strong tool. If you want it, build it.
I think it's more that the majority of humans DON'T REALIZE it's NOT doing that.
Most people think AI answers are carefully sourced, triple vetted, over-researched to death and can trust it with their life, despite the warnings, until they run into nonsense and then dismiss it as a one-off.
If there was a "always give the best possible, most researched and trusted answer" flag that was easily visible, or maybe "chatGpt" and "bestAnswersGpt", where the second one did all the verification and was considered much better overall, guess which one people would start using after a while and/or quote, because of improved accuracy?
It's not that people don't WANT honest information (well, most people) - they just don't KNOW they're not getting it right now. That's the problem with tech - they're trading on "computers were always right in the past" because there has been no generally available "fuzzy search" type application that needs handholding to give "good" results. Most software either gave the result of computations that are always the same, or enforced static rules (like spellcheck) which were perceived as "always right".
‘AI’ isn’t giving unsourced answers. Hastily cobbled together chatbots based on LLMs, inadequately prompted and without adequate research tools, are being put on the internet by humans and giving unsourced answers.
Well prompted LLMs with appropriate tools can do exactly what OP asked for. So AI can do this. That the AI-based products people are being given don’t is not a limitation of AI, it is a failure of humans.