Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.
When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.
I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.
There is nothing stopping us from adding the ability to understand the context of information an LLM consumes, too, right? The same process humans use to assess data reliability could be taught to an LLM?
Maybe. With enough money and resources and time, maybe. But it would need to be very reliable to be at all useful, and that level of reliability is very hard to achieve with something so nuanced and complex.
You have an entire lifetime of experience and the wisdom of countless generations to inform you of the context with which you should process new information. You were specifically trained for years to be able to do this somewhat effectively. Doing the same for an LLM may or may not be possible. They contain much of that experience in some form, but are not well equipped to make use of it iteratively. It is incredibly expensive to attempt even a simple and ineffective version of it with our current methods.
If the LLM can achieve even 90% accuracy, it becomes more of a hindrance than a help. Do you make use of the function and trust it successfully understood the provenance of new information? If so, ten percent of the time you will be misled, which could be disastrous across a population. If you don't simply trust it, then do you painstakingly check the provenance of everything it tells you? Of course not, you already don't do that with current search engines and blogs and such. Yet falling somewhere in between would be so much worse than simply not having the LLM attempt to discern trustworthiness from providence in the first place.
But all of that is opinion and I may be incorrect. Perhaps someone will create some ingenious way of imparting wisdom and discernment to LLMs next year and this will all be trivial.
I agree and finding "official domains" is an interesting problem in and of itself. I wonder if PageRank like weights of input data would help, or some web of trust-like idea to prune blatant spam websites.
Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.
It has little to do with gullibility and more to do with the way they do things.
In principle AI Agents are superhuman at codeforces but tons of these vibe coded apps have ridiculously stupid performance problems.
The problem is a utilization of expertise thing, it can tell you with incredible detail about pros and cons and caveats to a problem if you ask about it. Truly superhumanly intelligent.
Calibration when it comes to actual decision making takes an incredible reduction in capability after that. Maybe it has something to do with the sycophancy or something but its like someone with exactly zero opinions (and therefore, taste).
My experience is that you can brainstorm with it on "Do A or B?" and it will produce a coinflip answer A, then if you continue and ask about a specific concern "what about this aspect of B?", it immediately 180s and flips back to B being way better than A.
Acting on this expertise when decision making points arise naturally for itself is where everything really nosedives. It's just gonna do whatever.
So when you are doing this "financial advice please" questioning, it is entirely hitting these weaknesses squarely in the face.
How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?
Same as always: Develop a healthy sense of skepticism, cultivate a stable and diverse network of sources whose credibility you understand, don’t trust what you read, and certainly don’t trust synthesis unless you know how it’s made?
Of course, and I'm really not trying to move the goal posts here, but I agree that that's what (most) humans are going to do. But we're rapidly moving in a direction of agent-to-agent flow and "they" seem to lack the degree of discernment the average human does. They certainly don't have the same incentive structure we do to detect fraud. They happily regurgitate and build off of lies because, to them, the lies are not lies, and they can't be. They're just a pile of geometrically close labels from the perspective of the model.
So “caveat emptor”? I get that’s always part of reality but we shouldn’t exactly steer into that skid IMO. It just shifts the burden to the rest of us and does nothing to address the root cause
IMO, it's similar to what's happening with code. People have been shipping sloppy code and fake news forever. It's just that the cost of doing these things has fallen off a cliff.
Sure but the agents obfuscate the signals you might use to judge the authority/trustworthiness of search results. They get disinformation results, wrap it into a reasonably sounding report, present it to you as trustworthy. There are so many ways for bad actors to game the system
> Isn't all of this true for a human-used internet as well?
Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.
But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.
The primary purpose of Google is to stop that. You have 100% trust in Google results because of the effort they have put into cleaning the results you see.
Billions have been spent on it.
Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).
This is completely and utterly wrong. Google may put some effort into catching malware - but even that is limited to the more obvious cases. But for almost everything else, they put between 0 effort and negative effort into trustworthiness of the sources (e.g. they clearly favor big businesses for certain product searches regardless of quality).
Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.
When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.
I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.
There is nothing stopping us from adding the ability to understand the context of information an LLM consumes, too, right? The same process humans use to assess data reliability could be taught to an LLM?
Maybe. With enough money and resources and time, maybe. But it would need to be very reliable to be at all useful, and that level of reliability is very hard to achieve with something so nuanced and complex.
You have an entire lifetime of experience and the wisdom of countless generations to inform you of the context with which you should process new information. You were specifically trained for years to be able to do this somewhat effectively. Doing the same for an LLM may or may not be possible. They contain much of that experience in some form, but are not well equipped to make use of it iteratively. It is incredibly expensive to attempt even a simple and ineffective version of it with our current methods.
If the LLM can achieve even 90% accuracy, it becomes more of a hindrance than a help. Do you make use of the function and trust it successfully understood the provenance of new information? If so, ten percent of the time you will be misled, which could be disastrous across a population. If you don't simply trust it, then do you painstakingly check the provenance of everything it tells you? Of course not, you already don't do that with current search engines and blogs and such. Yet falling somewhere in between would be so much worse than simply not having the LLM attempt to discern trustworthiness from providence in the first place.
But all of that is opinion and I may be incorrect. Perhaps someone will create some ingenious way of imparting wisdom and discernment to LLMs next year and this will all be trivial.
Only in the context of something like RAG - there, you can attach provenance information to data included in the prompt.
With training data, however, no such annotation is possible.
Nothing stopping us except perverse incentives. Why would a company admit its model isn't using 100% reliable data?
3 replies →
During runtime research? Maybe, but most of what an LLM says doesn't come from runtime research.
During training? Not a chance.
I agree and finding "official domains" is an interesting problem in and of itself. I wonder if PageRank like weights of input data would help, or some web of trust-like idea to prune blatant spam websites.
Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.
It has little to do with gullibility and more to do with the way they do things.
In principle AI Agents are superhuman at codeforces but tons of these vibe coded apps have ridiculously stupid performance problems.
The problem is a utilization of expertise thing, it can tell you with incredible detail about pros and cons and caveats to a problem if you ask about it. Truly superhumanly intelligent.
Calibration when it comes to actual decision making takes an incredible reduction in capability after that. Maybe it has something to do with the sycophancy or something but its like someone with exactly zero opinions (and therefore, taste).
My experience is that you can brainstorm with it on "Do A or B?" and it will produce a coinflip answer A, then if you continue and ask about a specific concern "what about this aspect of B?", it immediately 180s and flips back to B being way better than A.
Acting on this expertise when decision making points arise naturally for itself is where everything really nosedives. It's just gonna do whatever.
So when you are doing this "financial advice please" questioning, it is entirely hitting these weaknesses squarely in the face.
1 reply →
[flagged]
How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?
Same as always: Develop a healthy sense of skepticism, cultivate a stable and diverse network of sources whose credibility you understand, don’t trust what you read, and certainly don’t trust synthesis unless you know how it’s made?
Of course, and I'm really not trying to move the goal posts here, but I agree that that's what (most) humans are going to do. But we're rapidly moving in a direction of agent-to-agent flow and "they" seem to lack the degree of discernment the average human does. They certainly don't have the same incentive structure we do to detect fraud. They happily regurgitate and build off of lies because, to them, the lies are not lies, and they can't be. They're just a pile of geometrically close labels from the perspective of the model.
6 replies →
So “caveat emptor”? I get that’s always part of reality but we shouldn’t exactly steer into that skid IMO. It just shifts the burden to the rest of us and does nothing to address the root cause
13 replies →
IMO, it's similar to what's happening with code. People have been shipping sloppy code and fake news forever. It's just that the cost of doing these things has fallen off a cliff.
The difference is context. With human cheating, the intent is usually a bit easier to spot.
Sure but the agents obfuscate the signals you might use to judge the authority/trustworthiness of search results. They get disinformation results, wrap it into a reasonably sounding report, present it to you as trustworthy. There are so many ways for bad actors to game the system
> Isn't all of this true for a human-used internet as well?
Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.
But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.
The primary purpose of Google is to stop that. You have 100% trust in Google results because of the effort they have put into cleaning the results you see.
Billions have been spent on it.
Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).
The frequently incorrect or misleading Gemini answer at the top of search results is busy eroding that trust.
People will have varying levels of trust in Google, but 100% is clearly too much, given the quality of results.
Google search results are amazing and you know it. It’s 100.
(I am not including AI results in this).
This is completely and utterly wrong. Google may put some effort into catching malware - but even that is limited to the more obvious cases. But for almost everything else, they put between 0 effort and negative effort into trustworthiness of the sources (e.g. they clearly favor big businesses for certain product searches regardless of quality).