Comment by xpct
16 hours ago
If I recall correctly, there were some papers which suggested that LLMs favor LLM-generated passages over human written ones. I can consistently reproduce this by asking Claude which code snippet it prefers: the one it generated in a different chat, or one that I refactored for my own needs and find more useful. It always picks its own :) I've also experienced that both Claude and Codex routinely include generated websites when I ask them to search for something. It also doesn't help that the web search tools that OAI and Anthropic have are deeply limiting: can't exclude keywords or domains.
I noticed this when I tried to get LLMs to play text adventures. Early on in my experiments, I wanted to give them hints when they got stuck at a puzzle. I did this by stopping the loop, injecting thoughts into the LLMs own persistent scratchpad (as if it had thought of that itself), and then starting the loop up again.[1] That way, the LLM would read what it had intended to remember from the previous turn including the hints I injected, and use that for the next turn.
I discovered that LLM-generated tokens in the scratchpad were relatively stable, but injected thoughts were frequently ignored and often deleted from the scratchpad within a few turns – even when the injected thought was the literal answer to the puzzle it was stuck at!
A reader[2] then pointed me toward research similar to what you might recall: LLMs interpret text by maintaining activations for input tokens, so text that is not generated by the same LLM will seem "unlikely" to the LLM in a sense, and when given the alternative between likely and unlikely text, it's probably trained to judge the unlikely text as a weird "slip of the mind" and discredit it in favour of the more likely text. I speculate this is part of how they can be useful in the first place, despite their non-determinism.
[1]: https://entropicthoughts.com/getting-an-llm-to-play-text-adv...
[2]: https://entropicthoughts.com/getting-an-llm-to-play-text-adv...
Did you ever try to ask a chatbot to rephrase your hint in its own words? If it prefers LLM generated text, surely that would help.
The other day, I remember an article was posted to HN about something, but it came from a company that provides SEO services to companies by doing something like this:
1. For a given company, analyze their target audiences and the questions they are likely to ask LLMs about.
2. For each such question, ask it to each of the major LLMs, and compute the KL divergence between the pages they want to rank for the question vs. the LLM's response.
3. Rewrite the article to minimize said KL divergence.
In effect, they're performing an iterative optimization of some sort that moves the embedding space of their article closer to the question asked to the LLM, and any embedding model or generated responses are going to prefer said responses over others.
I believe we will keep seeing more of this stuff.
And it's all because of ads. The incentives in an ad-funded internet are just always going to lead to this sort of thing. The most important thing is getting the user to load your page, not actually satisfying their query.
Let's hope the LLM model continues to be paying for credits, because any that move to ad revenue will become useless for real work.
> Let's hope the LLM model continues to be paying for credits
LLM vendors make this hard because you can't trust them with your session data. Yesterday you were opted out of training, then suddenly today you're opted in.
It's an extension of the idea that they don't need to care about anybody's copyright. They don't care about preserving the security or privacy of customer data, because there is negligible incentive to do so.
For now, there's no substitute but as LLMs get commoditized trusting LLM SAAS vendors becomes an unacceptable business risk.
1 reply →
> And it's all because of ads.
Not really. A while ago there was a news piece stating that Israel was behind a series of fake think-tanks with very accessible websites which were created with the express purpose of feeding AI agents with alternative facts aligned with their foreign policy.
If anyone has the link at hand, please post it.
14 replies →
The sad thing is this is unavoidable, or will be for a long time
Look at cable tv - even after going premium, you eventually wound up paying for ads anyway
In that case, you could make the argument that you could still purchase premium channels like hbo to avoid ads, but the internet doesn’t work that way - you depend on all the content generated by those ad funded channels
You could argue that Netflix changed that, and that’s why I said won’t change for a long time. I don’t think anyone’s discovered the business model yet that will keep content free for consumers while still generating revenue for companies
Can we just ban advertisements and "marketing" at a political level please?
1 reply →
There is a term for the general practice of optimizing responses called GEO - Generative Engine Optimization [0]. A cousin of SEO and equally unsavory in how trust is being eroded through info shaping. Self-discovery by individuals is the victim.
0: https://en.wikipedia.org/wiki/Generative_engine_optimization
The searchable term is "Self-preference Bias" :)
My first encounter with any kind of study was the G-Eval paper [1]. They study whether their LLM judge prefers human or LLM-generated summaries (answer: it's the latter).
[1] Section 4 in https://aclanthology.org/2023.emnlp-main.153/
Take a random essay and add in a bunch of the phrases that LLMs love like “load-bearing,” “crucial,” structural,” and “woven,” and then submit the original and the edited version to an LLM and ask which is better. It will choose the second one virtually every time. They have ingrained biases that associate those words with good writing and arguments.
This is why using other LLMs as scorers for benchmarks and evaluations is such a bad idea, they'll have preferences you can't anticipate and won't understand immediately.
The idea that that LLM reliability or bias can be solved with more LLM is... infuriatingly persistent.
1 reply →
Sometimes I wonder if there’s just one guy somewhere who loved using the word load-bearing, all his papers got trained on, and now he can’t write anything without being assumed to be Claude.
The prose equivalent of Artgerm (a comic cover artist whose style looks to have heavily inspired a lot of AI art).
> If I recall correctly, there were some papers which suggested that LLMs favor LLM-generated passages over human written ones
That makes sense. What an LLM does is output what the model thinks is the best set of tokens in response to a given input, so when you ask it to judge the best response to that input it is going to conclude that the best one is the one that must closely matches what it would output, which is what it did output.
Of course you aren't giving exactly the same context+input, but close enough that any difference doesn't push the output it made far from what it is going to say is ideal.
Does an LLM have any idea of what "best" is?
I think it doesn't, and just predicts the range of most statistically likely next tokens based on its training data, and picks one of those.
In ML, "best" is what minimises the loss function, so in a certain (very basic) sense it does.
If by root urls you mean domains, openai at least supports this.
https://developers.openai.com/api/docs/guides/tools-web-sear...
That is what I meant! Couldn't remember the word 'domain' while I was writing out my comment. Thank you
> I can consistently reproduce this by asking Claude which code snippet it prefers: the one it generated in a different chat, or one that I refactored for my own needs and find more useful. It always picks its own :)
A better question to ask for each snippet is "Estimate the seniority and competence of the developer who wrote the following code, ignoring bugs that linters or LLMs can catch and focus only on structure, maintainability, logical layout and readability."
It almost always estimates the author of my code as above the author of it's own code.
You're asking basically to ignore bugs and correctness. Can it be a useful comparison?
A bit different, but one thing I’ve seen is models repackaging Reddit slop. Like, it will do a search, find a Reddit thread somewhat related where someone in a comment casually mentioned incorrect information that any human would have dismissed. The model takes that as granted, but expands on it and present it as a well established fact, presented in a very plausible fashion.
In general I don’t find models to be good at evaluating the quality of a source :(
The other day Gemini gave me the correct answer, but cited an outdated Reddit thread which basically said the opposite.
As if it needs results to do that. You can give it a description of what you're looking for and get back some nonsense about how it's a "viral" and "legendary" page, with plenty of entirely fake details because it didn't actually find the page you wanted.
Newspapers have been doing this for a long time, notably the Metro in London.
There’s also whole TikTok (etc) channels who take random posts from Ask Reddit and just AI narrate the question and the top n highly-voted answers while showing a screenshot of each comment.
What percentage reads Metro and what percentage uses LLMs
i actively assume it is worse since, for example, spez signed a 60 million dollar deal to give Google access to the firehose. so then if you have niche, highly engaged subreddits infested by AI bots creating posts, then commenting on posts, then being trained on that content... you have Ouroburos eating its own poop, and models have less then zero incentive to evaluate the quality of a source, especially if they are the source
I don't have an oai subscription to try, but I'd be interested to know if Codex picks Claude's code over a human's and vice versa.
>It always picks its own
Makes sense to me, in that its own output would align closer to its own training set
I was giving local models a try recently, I think it was Qwen 3.6 I was trying at the time. I gave it a codebase and just asked it to review it. Its main feedback was that the comments and documentation were excellently written, but they were all Opus 5 slop.
> always picks its own
If you hate AI writing enough, this turns AI filters into a kind of humiliation ritual. AI will derank normal business writing for human readers, and uprank inflated, verbose, tic-heavy slop. So you have to put the heavy slop out with your name on it. Really perverse moment.
disagree that there is one kind of ranking and one kind of engine analyzing that ranking; sort of de-facto true that one company does run the ad world; strongly agree that this is a nightmare possibility and directly dystopian
[dead]
OMG if this is true, do you realize what this means? The easiest way to do AI SEO is to generate all your content with AI, and we've seen what SEO does to the web...
The Internet is doomed. Time to start some human-only darknets.
SEO is what ruined the web AFAIC.
> Time to start some human-only darknets.
I know very little about darknets. How could you ensure that they are human-only?
The only way to generate a human-only net is to figure out how to remove all economical incentives from there. The “golden age of the internet” a lot of people, including myself, are nostalgic for had very very little economical incentives. I missed usenet, but I remember being active on small forums in the late 90s and early 2000s that were almost entirely passion projects. I recall a huge controversy in one forum when the guy who was maintaining it asked if it’s ok if he puts a banner ad at the top because someone was offering him $200 to advertise another forum.
Removing the economical incentives is very hard though. Even HN is gamed by many tech companies and projects. Reddit is obviously a lost cause. It’s a sad state of affairs, but I don’t think there is an alternative.
Lose anonymity and bring back key-signing parties. Maybe you can't guarantee that everything is human-generated, but at least you know the chain of trust that leads to the human that signed off.
Yes, I'm aware of the irony of creating a darknet that only works by removing anonymity.
1 reply →
The golden age of internet was during seo, what are you talking about?
Bots trained on trash data, then produce trash. Why are we surprised here?
> asking Claude which code snippet it prefers: the one it generated in a different chat, or one that I refactored [...] It always picks its own
...is not the same as claiming...
> LLMs favor LLM-generated passages over human written ones
Here, you're using the same LLM to both produce and judge the resulting work. If anything, I would expect an LLM to tend to prefer its own work given that the same training is producing and judging.
It's not intuitive to me for why preference for its own writing would emerge, and during what type of training or tuning.
Perhaps something like: learning to identify what source files it has worked on by the code style alone, because tasks may give human code (public repos, etc) and ask to make changes.
It’s optimizing for good writing. Therefore, it believes its outputs are good. Therefore, it believes inputs that look like its outputs are good.
1 reply →
It would need to be researched, but I wonder if it ends up being something that happens at the token level?
If that were true I would expect to see prose that more closely resembles the "caveman" messages found in the HuggingFace attack than the overly flowery nonsense we see in AI blogspam.
Well, the one it generated is based on how it thought the best way to solve the problem was.
I am sure most humans would pick code written in their style, too.
> I can consistently reproduce this by asking Claude which code snippet it prefers: the one it generated in a different chat, or one that I refactored for my own needs and find more useful.
Interesting. For me I've noticed it tends to do the opposite.