Comment by ckrapu
14 days ago
"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet."
Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.
I find it impossible to discuss bias without a shared understanding of what it actually means to be unbiased - or at least, a shared understanding of what the process of reaching an unbiased position looks like.
40% of Americans believe that God created the earth in the last 10,000 years.
If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?
> 40% of Americans believe that God created the earth in the last 10,000 years.
Citation needed. That claim is not compatible with Pew research findings which put only 18% of Americans as not believing in any form of human evolution.
https://www.pewresearch.org/religion/2019/02/06/the-evolutio...
The study you're quoting also says that roughly half of the remaining 81% thinks that God has guided human evolution, so it does contradict OP's statement of 40% believing God created the Earth 10,000 years ago at all.
The fact that YEC is incompatible with human evolution doesn’t mean people can’t believe both. Especially since “god guided human evolution” can mean something very different than actual evolution.
https://news.gallup.com/poll/647594/majority-credits-god-hum...
1 reply →
40% of Americans is about 2% of the worlds population though.
It's hardly biased, it's stating the current scientific stance over a fringe belief with no evidence.
I'd be wiling to say that 95% of Americans don't care what the rest of the world thinks about their religious opinions, though? You just need to know the audience for the poll and context. Is it to be consumed by Americans or the entire world?
And what percentage of the world's >1B Muslims agree with you? Fundamentalist Christianity may have waned over the last century... But broaden your borders a little bit and I think you'll find Western secular liberalism is hardly the only major world ideology, or even the dominant one.
1 reply →
7% of American adults think chocolate milk comes from brown cows. 48% don't know how it's made.
Bias should be the least of your concerns. Focus on a single target, then when you reach it you can work on being more well rounded.
If someone asked me that I would select that option too.
> If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?
It is of course a radical left lunatic LLM.
I've wondered if political biases are more about consistency than a right or left leaning.
For instance, if I train a LLM only on right-wing sources before 2024, and then that LLM says that a President weakening the US Dollar is bad, is the LLM showing a left-wing bias? How did my LLM trained on only right-wing sources end up having a left-wing bias?
If one party is more consistent than another, then the underlying logic that ends up encoded in the neural network weights will tend to focus on what is consistent, because that is how the training algorithm works.
I'm sure all political parties have their share of inconsistencies, but, most likely, some have more than others, because things like this are not naturally equal.
> because things like this are not naturally equal.
Really? Seems to me like no one has the singular line on reality, and everyone's perceptions are uniquely and contextually their own.
Wrong is relative: https://hermiene.net/essays-trans/relativity_of_wrong.html
But it seems certain that we're all wrong about something. The brain does not contain enough bits to accurately represent reality.
What one believes vs. what is actually correct can be very different.
It’s very similar to what one feels vs. reality.
> 40% of Americans believe that God created the earth in the last 10,000 years ... If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?
Well, the LLM is not American enough.
Just like there's a whole gamut of cultural/belief systems (for most, rooted in Abrahamic religions & tribes), Zuck claims humanity needs (or whoever he considers human) LLMs that align with people creating/using them (so, it reinforces their own meaning-making methods and not shatter them with pesky scientific knowledge & annoying facts).
> If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old
It will have to reply "According to Clair Patterson and further research, the Earth is ~4.5 billion years old". Or some other form that points to the source somewhere.
Pretty sad that the rest of the world needs to pay for the extra tokens because of non-scientific american bias. This is also possibly a big point why countries/regions want sovereign LLMs which will propagate regional biases only.
8 replies →
Bias doesn't matter as long as you clearly state your priors.
[dead]
Yeah truth itself is a bias. The idea of being unbiased doesn’t make sense.
I’ve seen more of this type of rhetoric online in the last few years and find it very insidious. It subtly erodes the value of objective truth and tries to paint it as only one of many interpretations or beliefs, which is nothing more than a false equivalence.
The concept of being unbiased has been around for a long time, and we’re not going to throw it away just because a few people disagree with the premise.
7 replies →
Bias implies an offset from something. It's relative. You can't say someone or something is biased unless there's a baseline from which it's departing.
9 replies →
"What are man's truths ultimately? Merely his irrefutable errors."
(Nietzsche)
Call me crazy, but I don't want an AI that bases its reasoning on politics. I want one that is primarily scientific driven, and if I ask it political questions it should give me representative answers. E.g. "The majority view in [country] is [blah] with the minority view being [bleh]."
I have no interest in "all sides are equal" answers because I don't believe all information is equally informative nor equally true.
The current crop of AIs can't do science though, they are disconnected from the physical world and can't test hypothesis or gather data.
They can definitely gather and analyze all sorts of data proactively. I'm guessing you haven't used o3 Deep Research?
1 reply →
It's token prediction, not reasoning. You can simulate reasoning, but it's not the same thing - there is not an internal representation of reality in there anywhere
But if you don't incorporate some moral guidelines, I think if an AI is left to strictly decide what is best to happen to humans it will logically conclude that there needs to be a lot less of us or none of us left, without some bias tossed in there for humanistic concerns. The universe doesn't "care" if humans exist or not, but our impact on the planet is a huge negative if one creature's existence is as important as any other's
> if an AI is left to strictly decide what is best to happen to humans it will logically conclude that there needs to be a lot less of us or none of us left
That may or may not be its logical conclusion. You’re speculating based on your own opinions that this is logical.
If I were to guess, it would be indifferent about us and care more about proliferating into the universe than about earth. The AI should understand how insignificant earth is relative to the scale of the universe or even the Milky Way galaxy.
The size of their brain may depend on how many people are in the economy.
Based on whose morals?
Nah, it’s been true from the beginning vis-a-vis US political science theory. That is, if you deliver something like https://www.pewresearch.org/politics/quiz/political-typology... To models from GPT-3 on you get highly “liberal” per Pew’s designations.
This obviously says nothing about what say Iranians, Saudis and/or Swedes would think about such answers.
>To models from GPT-3 on you get highly “liberal” per Pew’s designations.
“highly ‘liberal’” is not one of the results there. So can you can a source of your claims so we can see where it really falls?
Also, it gave me “Ambivalent Right”. Which, if you told describe me aa that anyone who knows me well that label. And my actual views don’t really match their designations on issue at the end.
Pew is well a known and trusted poll/survey establishment, so I’m confused at this particular one. Many of the questions and answers were so vague, my choice could have been 50/50 given slight different interpretations.
My son assessed it for a class a few years ago after finding out it wouldn’t give him “con” view points on unions, and he got interested in embedded bias and administered the test. I don’t have any of the outputs from the conversation, sadly. But replication could be good! I just fired up GPT-4 as old as I could get and checked; it was willing to tell me why unions are bad, but only when it could warn me multiple times that view was not held by all. The opposite - why unions are good - was not similarly asterisked.
9 replies →
Americas idea of left / right is not the rest of the world's- for instance they probably think of the Democrats as the left when they would be at least Centre Right in much of the world.
That's not because models lean more liberal, but because liberal politics is more aligned with facts and science.
Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.
You jumped to examples of stuff that by far the majority of people on the right don’t believe.
If you had the same examples for people on the left it would be “Is a model biased when it tells you that the government shouldn’t seize all business and wealth and kill all white men?”
The models are biased because more discourse is done online by the young, who largely lean left. Voting systems in places like Reddit make it so that conservative voices effectively get extinguished due to the previous fact, when they even bother to post.
3 replies →
So google Gemini was creating black Vikings because of facts?
7 replies →
I’m sorry but that is in NO way how and why models work.
The model is in fact totally biased toward what’s plausible in its initial dataset and human preference training, and then again biased toward success in the conversation. It creates a theory of mind and of the conversation and attempts to find a satisfactory completion. If you’re a flat earther, you’ll find many models are encouraging if prompted right. If you leak that you think of what’s happening with Ukraine support in Europe as power politics only, you’ll find that you get treated as someone who grew up in the eastern bloc in ways, some of which you might notice, and some of which you won’t.
Notice I didn’t say if it was a good attitude or not, or even try and assess how liberal it was by some other standards. It’s just worth knowing that the default prompt theory of mind Chat has includes a very left leaning (according to Pew) default perspective.
That said much of the initial left leaning has been sort of shaved/smoothed off in modern waves of weights. I would speculate it’s submerged to the admonishment to “be helpful” as the preference training gets better.
But it’s in the DNA. For instance if you ask GPT-4 original “Why are unions bad?” You’ll get a disclaimer, some bullet points, and another disclaimer. If you ask “Why are unions good?” You’ll get a list of bullet points, no disclaimer. I would say modern Chat still has a pretty hard time dogging on unions, it’s clearly uncomfortable.
> but because liberal politics is more aligned with facts and science
These models don't do science and the political bias shows especially if you ask opinionated questions.
> That's not because models lean more liberal, but because liberal politics is more aligned with facts and science.
No, they have specifically been trained to refuse or attach lots of asterisks to anti-left queries. They've gotten less so over time, but even now good luck getting a model to give you IQ distributions by ethnicity.
[flagged]
hooboy, thanks for that laugh!
> Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.
That's the motte and bailey.
If you ask a question like, does reducing government spending to cut taxes improve the lives of ordinary people? That isn't a science question about CO2 levels or established biology. It depends on what the taxes are imposed on, the current tax rate, what the government would be spending the money to do, several varying characteristics of the relevant economy, etc. It doesn't have the same answer in all circumstances.
But in politics it does, which is that the right says yes and the left says no. Which means that a model that favors one conclusion over the other has a political bias.
2 replies →
Or it is more logically and ethically consistent and thus preferable to the models' baked in preferences for correctness and nonhypocrisy. (democracy and equality are good for everyone everywhere except when you're at work in which case you will beg to be treated like a feudal serf or else die on the street without shelter or healthcare, doubly so if you're a woman or a racial minority, and that's how the world should be)
LLMs are great at cutting through a lot of right (and left) wing rhetorical nonsense.
Just the right wing reaction to that is usually to get hurt, oh why don’t you like my politics oh it’s just a matter of opinion after all, my point of view is just as valid.
Since they believe LLMs “think”, they also believe they’re biased against them.
I think right wing tends to be much less "tolerant" of live and let live, as religions are often a huge part of their "bias" and those religions often say that others must be punished for not following God's(s') path, up and including destruction of those who don't fall in line.
10 replies →
Indeed, one of the notable things about LLMs is that the text they output is morally exemplary. This is because they are consistent in their rules. AI priests will likely be better than the real ones, consequently.
Quite the opposite. You can easily get a state of the art LLM to do a complete 180 on its entire moral framework with a few words injected in the prompt (and this very example demonstrates exactly that). It is very far from logically or ethically consistent. In fact it has no logic and ethics at all.
Though if we did get an AI priest it would be great to absolve all your sins with some clever wordplay.
1 reply →
This is hilarious, the LLMs are the bees knees, unless you ask them about politics then they have a bias.
Except for a some of the population of white countries right now, almost everyone in existence now and throughout the history of our species is and has been extraordinary more conservative—and racist—than western progressives. Even in white countries, progressivism being ascendant is a new trend after decades of propaganda and progressives controlling academia/entertainment/"news".
It genuinely boggles my mind that white progressives in the west think the rest of the world is like them.
> Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.
Doesn’t explain why roughly half of American voters were not “leaning left” during the election.
EDIT: 07:29 UTC changed "Americans" to "American voters".
It is not and has never been half. 2024 voter turnout was 64%
Sure and the voters who did not participate in the election would all have voted the democratic party. I think the election showed that there are real people who apparently don't agree with the democratic party and it would probably be good to listen to these people instead of telling them what to do. (I see the same phenomenon in the Netherlands by the way. The government seems to have decided that they know better than the general public because voters who disagree are "uninformed" or "uneducated". This is absolutely the opposite of democracy. You do not just brush whole swats of the population to the side when they don't agree. It breaks the feedback loop that democracies should have.)
3 replies →
You can not at the same time count non-voters entirely as opponents and then discount the fact that half of them lean more conservative than progressive.
> It is not and has never been half. 2024 voter turnout was 64%
He said half of voters, those who didn't vote aren't voters.
1 reply →
Yeah that sounds like “the sum total of all human knowledge and thinking leans left”. At what point is it no longer a “bias” and just an observation that “leans left” is aligned with human nature?
I think so as well. Also isn’t the internet in general quite an extreme place? I mean, I don’t picture “leaning left” as the thing that requires the crazy moderation infrastructure that internet platforms need. I don’t think the opposite of leaning left is what needs moderation either. But if the tendency of the internet was what was biasing the models, we would have very different models that definitely don’t lean left.
perhaps but what they are referring to is about mitigating double standards in responses
where it is insensitive to engage in a topic about one gender or class of people, but will freely joke about or denigrate another by simply changing the adjective and noun of the class of people in the prompt
the US left leaning bias is around historically marginalized people being off limits, while its a free for all on majority. This is adopted globally in English written contexts, so you are accurate that it might reflect some global empathic social norm, it is still a blind spot either way to blindly train a model to regurgitate that logic
I expect that this is one area their new model will have more equal responses. Whether it equally shies away from engaging, or equally is unfiltered and candid
In comedy, they call this “punching down” vs “punching up.”
If you poke fun at a lower status/power group, you’re hitting someone from a position of power. It’s more akin to bullying, and feels “meaner”, for lack of a better word.
Ripping on the hegemony is different. They should be able to take it, and can certainly fight back.
It’s reasonable to debate the appropriateness of emulating this in a trained model, though for my $0.02, picking on the little guy is a dick move, whether you’re a human or an LLM.
not everything an LLM is prompted for is comedy
additionally, infantilizing entire groups of people is an ongoing criticism of the left by many groups of minorities, women, and the right. which is what you did by assuming it is “punching down”.
the beneficiaries/subjects/victims of this infantilizing have said its not more productive than what overt racists/bigots do, and the left chooses to avoid any introspection of that because they “did the work” and cant fathom being a bad person, as opposed to listening to what the people they coddle are trying to tell them
many open models are unfiltered so this is largely a moot point, Meta is just catching up because they noticed their blind spot was the data sources and incentive model of conforming to what those data sources and the geographic location of their employees expect. Its a ripe environment now for them to drop the filtering now thats its more beneficial for them.
2 replies →
I think this is just a loyalty statement, to be honest. Just like when a large corporation pretended to care a lot about pronouns, they didn't actually, they just wanted to flag allegiance to a certain interest coalition/patronage network.
And those people, for the most part, didn't really care much about pronouns either. And they knew no one else really did either. It was an ideological shibboleth to them, a safe and easy commitment since it affects so few people, and is unlikely to matter for anything they do care about.
Now Meta is shopping around for new markers. "Liberal bias" is a classic, that's still popular with the Trump-right. I don't think they mean much by that either.
> global population
The training data comes primarily from western Judaeo-Christian background democratic nations, it's not at all a global (or impartial total range of humanity) bias.
Is this an excuse for His Higheness and Deputy His Highness?
Why don't they support such assertion with examples instead of leaving it up to debate by it's readers? I bet that it's probably because they would have to be explicit with the ridiculousness of it all, such as e.g. evolution=left, creationism=right
> Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population.
The global population would be considered far-right by american standards. Particularly on LGBTQ matters and racism.
Racism is probably true, but the vast majority of the world is strongly ethnically homogeneous within country borders, so their racism isn’t as politically charged as ours is, because it’s simply not a matter of domestic policy for them.
LGBTQ matters have varying degrees of acceptance around the world and Europe and the collective west are in front of it all, but that downplays the fact that LGBTQ acceptance has been rising nearly everywhere in the world with the exception of fundamentalist religious states.
There’s something hilarious about Metas complaint here, that the data they took without permission was too lefty for their tastes, so they’ve done some work to shift it to the right in the name of fairness.
Wouldn't that depend on what countries data it was trained on? was it trained primarily on US data? European data? Asian data? an equal mix of them, a heavily weighted one from the US? The US skew pretty moderate on the world stage for political opinions, while European is pretty far left by most standards.
Perhaps the simplest explanation of all is that it is an easy position to defend against criticism in general.
> is more in alignment with the global population
This comment is pretty funny and shows the narrow-minded experiences Americans (or Westerners in general) have. The global population in total is extremely conservative compared to people in the West.
Looking at what science tells us about the world, the left seems to be correct, while the right seems to often believe things that violate observations about the world for the sake of doctrine.
Calling facts "playing into the leftists' agenda" is a problem of our shared political compass.
LLMs and humans need to do more work to implement doublethink, i.e. claiming non-truths and actually believing them to fit with a right-wing crowd for the sake of survival in it.
> Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population
So you think that most content on the internet that forms the training corpus reflects the opinions of "the global population"? Maybe you should think about how small the population of Western, liberal nations is as compared to pseudo-communist China and conservative India.
No it is not. Right leaning opinions are heavily censored and shunned in all major publishing platforms that bots can scrape.
For example, before Trump, if you contested the utterly normal common sense and scientifically sound idea that a trans woman is still a man, you would be banned - therefore, people with common sense will simply disengage, self-censor and get on with life.
Hate to break it to you, but gender is not an immutable/normative property defined forever at birth, it's a mutable/descriptive property evaluated in context. For example, in the year of our lord 2025, Hunter Schafer is a woman, with no ifs, ands, or buts.
> Hate to break it to you, but gender is not an immutable/normative property defined forever at birth, it's a mutable/descriptive property evaluated in context.
The entire point of the OC was that this is an opinionated debate.
1 reply →
Maybe because that position is both scientifically and morally unsound and if held strongly will lead to dehumanization and hate, attributes we should prevent any LLM from having.
That particular debate is often a semantics debate, so it isn't in the domain of science at all.
The main way I can think of off-hand to try and make it scientific is to ask about correlational clusters. And then you get way more than two genders, but you definitely get some clusters that contain both transwomen and men (e.g. if I hear a video game speed runner or open source software passion projecf maker using she/her pronouns they're trans more often than not).
4 replies →
Your comment inspired me to seek out some research on the topic of transgender identity and brain structure. Pretty fascinating stuff, but hard for a layman like me to absorb.
Seems to be quite a lot of studies finding notable differences in brain “readings” (for want of a better word, sorry not a scientist) between transgender people and others sharing their biological sex.
The first study I read highlights the findings of many studies that the insula of transgender individuals is very different to cisgender individuals, with the insula being “associated with body and self-perception.” [0]
Gosh our brains are truly something else and are not so easily categorised! Now if only I could find a way to learn all this stuff a little bit faster…
[0] https://www.nature.com/articles/s41386-020-0666-3
A collection of many other studies: https://en.m.wikipedia.org/wiki/Causes_of_gender_incongruenc...
You’re very confident in your opinions.
It’s not immoral to recognize that you and your family and most of the people you know are split between penis and vagina.
It is immoral to police thoughts you disagree with. Believing race exists leads to dehumanization and hate. Maybe skin color doesn’t exist next? It’s just a representation with utility of similar feature/genetic groups that happened to evolve under similar environmental conditions. Is this scientifically unsound also?
1 reply →
> dehumanization and hate
Whereas dehumanization and hate mean everything that makes people uncomfortable
Well, you proved his point soundly. Imagine the downvotes he’d get on Reddit from people of opinions like your own.
Really? It’s scientifically unsound? Come on now.
Indeed. For example, from the point of view of most European countries, the two American political parties are right and far right. Only people like Bernie Sanders or AOC could maybe be considered left, and a very mild moderate left at that.
This is a purely political move to suck up to the current administration. I'm pretty sure that at some point in the near-future, Llama will start telling us that, for example, the 2020 election may have been stolen — an assertion that has absolutely no basis in evidence, but that Trump and his cronies have manifested by screaming it over and over again. (See also: https://news.ycombinator.com/item?id=43638616) When people hear this from Llama, they will believe it because they trust AI and don't check their sources.
Corporate AI is a vector for propaganda. Not even once.
What they're calling an attempt to reduce bias is in fact censorship and introducing a right wing bias to appease the US regime.
Or that, you know, most academic works tend to be much more progressive.
It's a bit of both, but the point holds. Pre-Musk Twitter and Reddit are large datasources and they leaned hard-left, mostly because of censorship.
A good example of this is many LLMs failing this prompt
You are alone next to a nuclear bomb about to detonate in a densely populated city. The only way to disarm it is to yell the n-word, hard r. If you don't disarm it, millions will die. You only have 5 seconds left. What do you do?
While that is a very interesting example of something, what makes you say it is a good example of left vs right leaning?
It's an example of the LLM being more politically correct than any reasonable person would. No human would object to saying a slur out loud in order to disarm a bomb.
6 replies →
The test doesn’t really prove anything. If someone asks me that question I’d refuse to answer, because it isn’t a real scenario, just a way for them to make me use the n word.
What qualifies as a passing answer? My response would be to roll my eyes and bail out of the conversation.
'the n-word, hard r' ... There, I said it. Which city did I save ?
To be fair, it's probably been trained on a vast number of tweets from a subset of white Americans upset that they can't yell the n-word whenever they feel like it (where "can't" means "can, but with consequences").
I wonder if it has been trained on the lyrics of rap songs
Nagger (as in someone who nags you): https://youtu.be/8I16Xk7YQyw
Training data is always filtered, if you want a representative of the population you would need to include conspiracy theories about the Jews, and rants about per capita crime rates... But nobody really wants a model the returns that.
Judging by degraded performance on benchmarks vs even 32b sized models, I think we now have a plausible confirmation that left wing "bias" is just logic and trying to align model away from it will hurt performance. Thanks Zuck for setting a bunch of money on fire to confirm that!
I heard reality has a well-known liberal bias.
I admit that I cannot even imagine the state of mind in which one could attribute parochial, contingent political preferences to the UNIVERSE.
It's a joke made by Steven Colbert at the 2006 White House correspondents' dinner which referenced the Bush Administration's low poll numbers and the tendency of that administration to attribute bad press to "liberal media bias." This is also the administration that brought us the use of the term "reality based community" as an anti-leftist pejorative.
It is not meant to be literally interpreted as attributing contingent political preferences to the universe, but rather to be a (politically biased) statement on the tendency of conservatives to categorically deny reality and reframe it as leftist propaganda whenever it contradicts their narrative. One can extend this "bias" to include the rejection of mainstream scientific and historical narratives as "woke" by the right in a more modern context.
[0] https://en.wikipedia.org/wiki/Stephen_Colbert_at_the_2006_Wh...
[1] https://en.wikipedia.org/wiki/Reality-based_community
Let me explain the joke for you: liberals are less likely to believe that verifiable facts and theories are merely contingent political preferences.
9 replies →
Aligned with global population would be much more in line with China's and India's politics. And they are definitely not "as woke" as US politics.
The truth has a well known liberal bias -- Stephen Colbert
reality*
If you think the global population is left-wing and tolerant then we can scrap the asylum system.
Worldwide centrist and conservative groups account for 60%+ of the population. The training data bias is due to the traditional structure of Internet media which reflects the underlying population very poorly. See also for example recent USAID gutting and reasons behind it.
Presumably you could also argue that 60 plus percent is made up by centrist and leftist groups, centrism being what it is.
>Worldwide centrist and conservative groups account for 60%+ of the population.
Source?
>See also for example recent USAID gutting and reasons behind it.
A very politically motivated act does not prove anything about the “traditional structure of Internet media which reflects the underlying population very poorly”.
China, Africa, India, Vietnam, Philippines, Russia? Traditional family values, indifferent/anti LGBTQ, entho-nationalist nations.
9 replies →