Comment by mstaoru
15 hours ago
Well it's not only this, or protection from LLMs training on LLM output. LLMs training on human output is also problematic.
I was traveling to an obscure small town, doing some "research" with LLMs beforehand. Every and each one told me enthusiastically to go to "Foobar square" (name changed) for the "best street food in XYZ town", some added a lot of colorful details.
There was no Foobar square in XYZ town. There was no Foobar square anywhere in the world. There was a SINGLE old Reddit comment, with no upvotes, to a unpopular post in an unpopular subreddit, where someone clearly badly misspelled the name of the square, and said something like "for street food go to Foobar square". Nothing about "the best" even.
It's all a lie.
I think this was a common game on city/town subs. It happened here, there was a post asking for a good restaurant and someone just made up a name. It went viral and people started posting made-up menus for the place, reviews, and for a couple of months any time someone asked about a restaurant this fictional place would get mentioned.
It was all done as a joke to see if they could get Gemini or ChatGPT to start recommending it.
The Montréal subreddit has been doing this for ages before LLMs were a thing because every summer and fall there's endless threads from tourists and students asking the same questions that recommending a local gay bathhouse became the meme answer.
I've had gemini claiming code would compile and run while also outputting the same variable in the same sniplet with "fork" "frok" and "fokr" in the name. I'm not surprized it's trained on garbadge.
Your one example doesn't make all of LLMs a lie.
It's like someone reading a National Enquirer article about "Hillary Clinton being an alien from outer space" (a real headline topic from decades ago) and drawling the conclusion that all journalism is "a lie". The user has to understand media literacy and be at least a little skeptical of the claims that are made, then cross reference with another source.
Okay, but the anecdote states that every model repeated the pseudo-factoid about Foobar square, not just the 4 GB open source model equivalent of a tabloid.
Without disclosing what you were prompting for, it's impossible to evaluate your claim.
1 reply →