Comment by golol
2 years ago
If you build a complex Chain if Thought style Agent and then train/finetune further by reinforcement learning with this architecture then it is not a band-aid anymore, it is an integral part of the model and the weights will optimize to make use of this CoT ability.
It's been 3.5 years since GPT-3 was released, and just over a year since ChatGPT was released to the public.
If it was possible to solve LLM hallucinations with simple Chain-of-Thought style agents, someone would have done that and released a product by now.
The fact that nobody has released such a product, is pretty strong evidence that you can't fix hallucinations via Chain-of-Thought or Retrieval-Augmented Generation, or any other band-aid approaches.
I agree: but I just wanted to say that there are specific subdomains where you can mitigate some of these issues.
For example, generating json.
You can explicitly follow a defined grammar to get what will always be a valid json output.
Similarly, structured output such as code can be passed to other tools such as compilers, type checkers and test suites to ensure that at a minimum the output you selected passes some minimum threshold of “isn’t total rubbish”.
For unstructured output this a much harder problem, and bluntly, it doesn’t seem like there’s any kind of meaningful solution to it.
…but the current generation of LLMs are driven by probabilistic sampling functions.
Over the probability curve you’ll always get some rubbish, but if you sample many times for structure and verifiable output you can, to a reasonable degree, mitigate the impact that hallucinations have.
Currently that’s computationally expensive, to drive the chance of error down to a useful level, but compute scales.
We may seem some quite reasonable outputs from similar architectures wrapped in validation frameworks in the future, I guess.
…for, a very specific subset of types of output.
I agree that the "forcing valid json output" is super cool.
But it's unrelated to the problem of LLM hallucinations. A hallucination that's been validated as correct json is still a hallucination.
And if your problem space is simple enough that you can validate the output of an LLM well enough to prove it's free of hallucinations, then your problem space doesn't need an LLM to solve it.
1 reply →