Comment by Lerc
5 days ago
The LLM aspect of this, I think shows both a common weakness and an opportunity.
If you suspect something is a commonly held misconception, frequently asking a LLM about it is close to useless, because the abundance of text repeating the misconception (it is common after all) just makes the model repeat the falsehood. Asking the model to apply a more balanced view quite often triggers an iconoclastic antagonism which will just give you the opposite of whatever the commonly held opinion. I have had some success in asking for the divergence between academia and public opinion on particular issues.
Models are still very bad at determining truth from opinion, but these are exactly the times when we need the models to work the best.
There may be an opportunity if there are enough known examples of instances like this story for a dataset to be made where a model can be taught to identify the difference between honest opinion and bad faith actors, or more critically identify confidently asserted statements from those supported by reasoning.
Unfortunately I can see models that can identify such falsehoods being poorly received. When a model can say you are wrong and everybody around you says you are right, what are the chances of people actually considering the possibility that the model is, in fact, correct?
> Models are still very bad at determining truth from opinion, but these are exactly the times when we need the models to work the best.
Suppose models could determine truth and opinion. How would anyone go about training that into the model? Even in unsupervised, adversarial, whatever training scenarios you still need some kind of framework to discern between the two. Academic manuscripts? There are decades of publications built upon "discoveries" later retracted or shown to be falsified.
How would you build the truth/opinion vector(-s) with impartial objectivity?
> <...> a model can be taught to identify the difference between honest opinion and bad faith actors, or more critically identify confidently asserted statements from those supported by reasoning.
You have a confidently asserted statement {from an industry veteran, backed by nothing but credibility}, and a statement supported by reasoning{, which is in turn supported by... implicit assumptions, verging on wild guesses and exaggerations}. Do you see a reasonable way to somehow embed the context in curly braces in any part of the LLM pipeline without nullifying the need for the LLM itself?
That's why I stated this case as an opportunity. To be able to do this you would need to have a set of examples where sometimes the dominant narrative is incorrect. This article represents one of those cases. Identifying more would be hard work and objectivity would be difficult, but I think possible.
Addressing your final point, I think there is scope for doing that. Having a provenance aspect to embeddings would do that, I suspect existing LLMs infer this information quite well already but I think there might be a possibility to go a little further at inference time by instead of a straight text to token embedding to have a richer input processing model that takes text plus other known context data to produce an embedding holding that extra data. The input processing model would have to be trained to convert the text plus context into a vector containing that info in a form the model already understands.
I think this would be useful in a number of other areas as well, firstly being able to distinctly tag model generated output so it doesn't confuse itself. Tagging individual tokens from code to say how long this code has been in the project, if it comes from a version that lints correctly, compiles, is used in production etc. Not to mention tagging prompts from the user as prompts and filtering that same tagging out of all non-prompts so that prompt injection is much harder to do.
> To be able to do this you would need to have a set of examples where sometimes the dominant narrative is incorrect.
You are sidestepping the issue here. The issue is NOT lack of such database per se, the issue is lack of impartial oracle used to build such database.
> instead of a straight text to token embedding to have a richer input processing model that takes text plus other known context data to produce an embedding holding that extra data
The issue is lack of context informing us of the sincerity behind the claim, therefore proposal is to use LLMs to infer that. When presented with difficulties inferring that context from immediate message, your proposal is to build broader context and then feed it to LLM. To do what, spit it back packed in certain language tone? Do you see the tautology here?
Discussion around AIs/LLMs is quickly becoming discussion around Bitcoin at certain points in the hype cycle. You can't solve for external effects internally. An LLM cannot tell truth from opinion, bitcoin network cannot guarantee a transaction actually took place.
1 reply →
> Models are still very bad at determining truth from opinion
Models are not bad at it. Models are not even trying. As you point out, it is about what is the most common text. It has nothing to do with logic.
On that I disagree. LLMs are not simple Markov chains.
They may fail at a lot of logical tasks, but I don't think that is the same as exhibiting no logic.
Getting even slightly respectable performance on the ARC-AGI test set, I think shows that there is at least some logical processing going on. General intelligence is another issue entirely, but there's definitely more than nothing.