Comment by Amorymeltzer
16 hours ago
I'm reminded of a section of Cory Doctorow's latest non-fiction, The Reverse Centaur's Guide To Life After AI (it is of course an excellent read), where he discusses how the semantic web and having an internet set-up for agents and AI to use won't work:
>It also hasn't happened because there is a reward for cheating. Scammers will label their phone numbers as the customer service number for your cable operator or bank. Counterfeiters will label their batteries or phone cases as official accessories. It's hard enough to get people to correctly label their stuff—but fact-checking billions of labels created by millions of people, none of whom you know personally or whose identities you can verify? That's insurmountable.
There's more he says—the preceding paragraph is below—but that line "there is a reward for cheating" is something I keep thinking about. It's hard not to see in so many of the discussions of LLMs.
>For one thing, there are irreconcilable differences in the ways in which things should be described. A site run by transphobes isn't going to label every story about a trans woman as "a story about a woman," irrespective of what the standard says. Microsoft is never going to label Excel as equivalent to Apple's Numbers, Google Sheets, and LibreOffice's Calc. Hermès is never going to list a $40,000 handbag as being made of the same material and having the same capacity as a $30 leather handbag of the same size and shape.
Isn't all of this true for a human-used internet as well?
Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.
When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.
I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.
There is nothing stopping us from adding the ability to understand the context of information an LLM consumes, too, right? The same process humans use to assess data reliability could be taught to an LLM?
7 replies →
I agree and finding "official domains" is an interesting problem in and of itself. I wonder if PageRank like weights of input data would help, or some web of trust-like idea to prune blatant spam websites.
Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.
2 replies →
[flagged]
How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?
Same as always: Develop a healthy sense of skepticism, cultivate a stable and diverse network of sources whose credibility you understand, don’t trust what you read, and certainly don’t trust synthesis unless you know how it’s made?
21 replies →
IMO, it's similar to what's happening with code. People have been shipping sloppy code and fake news forever. It's just that the cost of doing these things has fallen off a cliff.
The difference is context. With human cheating, the intent is usually a bit easier to spot.
Sure but the agents obfuscate the signals you might use to judge the authority/trustworthiness of search results. They get disinformation results, wrap it into a reasonably sounding report, present it to you as trustworthy. There are so many ways for bad actors to game the system
> Isn't all of this true for a human-used internet as well?
Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.
But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.
The primary purpose of Google is to stop that. You have 100% trust in Google results because of the effort they have put into cleaning the results you see.
Billions have been spent on it.
Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).
The frequently incorrect or misleading Gemini answer at the top of search results is busy eroding that trust.
People will have varying levels of trust in Google, but 100% is clearly too much, given the quality of results.
1 reply →
This is completely and utterly wrong. Google may put some effort into catching malware - but even that is limited to the more obvious cases. But for almost everything else, they put between 0 effort and negative effort into trustworthiness of the sources (e.g. they clearly favor big businesses for certain product searches regardless of quality).
Blockchain making a come back
it seems like end stage of LLMs is an MCP economy in which Google and Apple will have a massive advantage in verified locations, businesses, identities and representation of services accurately.