I had a related, but orthogonal question about multilingual LLMs.
When I ask smaller models a question in English, the model does well. When I ask the same model a question in Turkish, the answer is mediocre. When I ask the model to translate my question into English, get the answer, and translate the answer back to Turkish, the model again does well.
For example, I tried the above with Llama 3.3 70B, and asked it to plan me a 3-day trip to Istanbul. When I asked Llama to do the translations between English <> Turkish, the answer was notably better.
Someone apparently did observe ChatGPT (I think it was ChatGPT) switch to Chinese for some parts of it's reasoning/calculations and then back to English for the final answer. That's somehow even weirder than the LLM giving different answers depending on the input.
I've seen this happen as well with o3-mini, but I'm honestly not sure what triggered it. I use it all the time but have only had it switch to Chinese during reasoning maybe twice.
I saw Claude 3.7 write a comment in my code in Russian followed by, likely from a previous modification, the English text “Russian coding” for no reason.
> the LLM giving different answers depending on the input.
LLMs are actually designed to have some randomness in their responses.
To make the answer reproducible, set the temperature to O (eliminating randomness) and provide a static seed (ensuring consistent results) in the LLM's configuration.
See my other comment. The answer is transfer learning: leveraging massive amounts of data in one language like Python, a few bridges to another language like Ruby, and obtain a “native” result in the other language.
But in this case the LLM is not exposed to explicit translation pairs between these two languages and rather by seeing enough examples in similar contexts, LLMs transfer some of their learnings in Python to Ruby (for better or worse results)
Fascinating phenomenon. It's like a new Sapir–Whorf hypothesis. Do language models act differently in different languages due to those languages or the training materials?
This is one of those subtle clues that the LLM does not actually 'know' anything. It is providing you the best consensus answer to your prompt using the data upon which the weights rest, is that data was input primarily as english then you are going to get better results asking in english. It is still Searle's Chinese Room except you need to first go to the 'Language X -> English' room and then deliver its output to the general query room before delivering the next result to the 'English -> Language X' room.
Both, but primarily due to the lack of training materials. 10 or so million native speakers of my language will never be able to generate the same amount of training material as over a billion English speakers do.
There is a steep drop in quality in any non-English language, but in general less native speakers = worse results. They tend to have a certain "voice" which is extremely easy to spot and the accuracy of results goes out the window (way worse than in English).
For most low-resource languages, support in LLMs is trained through translation pairs between english and the other languages, because translation data is easier to come across than say, conversations about coding, history, physics, basically the kind of data that is usually used for instruct training.
This kind of training data typically looks like ChatGPT style conversations where all the prompts are all templated like “Translate the following text from X to Y: [text]” and the LLM’s expected answer is the translated text.
LLMs can generalize through transfer learning (to a certain extent) from these translation pairs to some understanding (strong) and even answering (weak) in the target language. It also means that the LLM’s actual sweet spot is in translation itself since that’s what was trained in, not just a generalization.
Indeed. I've thought from the beginning that LLMs should focus specifically on ONE language for this exact reason (i.e. mediocre/bad duplication of data in multiple languages). All other languages than English essentially "syphon" off capacity/layers/weights that could otherwise have held more genuine data/knowledge. Other languages should not come into the picture afaics - dedicated translation LLMs/existing-solutions can handle this aspect just fine and there's just no salient reason to fold partial-multi-language-capacity in through fuzzy/unorganised training.
I'd mentally put this in the same box as "chain of thought", where models perform better when explicitly describing the reasoning steps. The only difference in your case being that the model is undertrained in non-English data, so it's "next token prediction" of non-English prompts is less robust, and thus explicitly converting to English and then back makes it better.
This is probably the case for the "deep reasoning" models as well. If you for example try DeepSeek R1, it will likely reason in either English or Chinese (where it presumably is well trained) even if the prompt is in other languages.
Don't speak French, but interesting that it's not quite felt like an insufferable American tourist not in the group chat, in your language. LLMs all belong in that spectrum in my primary language.
Given the fact that LLMs like most neural networks work by passing their input through layers, wouldn't this be expected? There's no going back to an earlier layer and if the first layers are in some sense needed for "translating" [0] to English, any other functionality in those layers cannot be used.
[0] I am simplifying here, but it would make sense for an LLM to learn this, even though the intermediate representation is not exactly English, given the fact that much of the internet in English and the empirical fact that they are good at translating.
Some studies are trying to ensure that the model reasons through abstractions instead of linguistic representations. (Of course the phenomenon of reasoning in substantially different quality depending on input language signals a fault - reasoning is beyond "spoken" language.)
Was pretty good with Latvian (better than other models this size as well as variants of Llama or Qwen that I could run) and I assume probably with other EU languages as well.
I've just tried it in one of the supported languages, and it seems to respond far better than any model under 24B that I've tried before. With its licensing, it sounds much more exciting to me than the OP.
More diversity in the LLM space is always good. In my experience though, speaking as a native speaker of one of the less-used European languages, Mistral's models already use it pretty well.
I live in a country with 3 national languages and I happen to use all of them + English + another one where most of our clients are based. Mistral is the only model atm which doesn’t make a mess of it all. It’s not perfect, but it doesn’t force me to “pretranslate” things.
As a native of another small European language, no state of the art model comes anywhere close to not being laughably bad, so more work in this space is definitely welcomed as far as I'm concerned.
Really? In my experience, Le Chat eventually devolves into spanglish when trying to speak Spanish, so I would have expected worse from Mistral for minority languages.
Meltemi is ok, but it's "old" and not that good by today's standards.
If you need a good Greek local LLM try https://huggingface.co/ilsp/Llama-Krikri-8B-Instruct.
Yes, I know it's based on LLama and not a foundation model, but it is still a LOT better than Meltemi.
I mean, Mistral AI is a Paris-based company, and theirs was considered on par or better than other open weight models such as llama3.1 and qwen2.5, and mistral-24b is currently beating oh-so-great gemma3-27b depending on tasks.
Also, Stable Diffusion was originally (and still is I believe) developed in Munich.
It's true though that raising capital and finding investors works wayyy better in the US (kindof needless to say on HN) and so was getting top talent - at least in the past. Don't get me started on energy prices ;) but I don't believe those contribute significantly in the end anyway.
You don't think American companies raising hundreds of millions to ten billion for training models contributed to their model performance or market positions?
I think a pile of money and talent is largely the cause of where they're at.
But this is an image-like benchmark. Has anyone looked at the article about the EU-ARC, what is the difference? Why can't you measure it on a regular one?
I glanced through it, didn't find it right away, but judging by their tokenizer, they are learning from scratch. In general, I don't like this approach for the task at hand. For large languages, there are already good models that they don't want to compare with. And for low-resource languages, it is very important to take more languages from this language group, which are not necessarily part of the EU
Why would they want more languages from outside of the EU when they've clearly stated they only target the 24 official languages of the European Union?
For example: Slovene language. You simply don't have enough data on it. But if you add all the data that is available on related languages, you will get a higher quality. LLM fails with this property for low-resource languages.
They compared with Llama 3.1 and found that to be better on average for their tasks like European MMLU. And Llama 3.1 is the worst in the batch with Qwen 2.5 and Gemma 3 being significantly better.
I had a related, but orthogonal question about multilingual LLMs.
When I ask smaller models a question in English, the model does well. When I ask the same model a question in Turkish, the answer is mediocre. When I ask the model to translate my question into English, get the answer, and translate the answer back to Turkish, the model again does well.
For example, I tried the above with Llama 3.3 70B, and asked it to plan me a 3-day trip to Istanbul. When I asked Llama to do the translations between English <> Turkish, the answer was notably better.
Anyone else observed a similar behavior?
Someone apparently did observe ChatGPT (I think it was ChatGPT) switch to Chinese for some parts of it's reasoning/calculations and then back to English for the final answer. That's somehow even weirder than the LLM giving different answers depending on the input.
Reminds me of this funny video: https://www.youtube.com/watch?v=NY3yWXWjYjA ("You know something has gone wrong when he switches to Chinese")
I've seen this happen as well with o3-mini, but I'm honestly not sure what triggered it. I use it all the time but have only had it switch to Chinese during reasoning maybe twice.
5 replies →
I saw Claude 3.7 write a comment in my code in Russian followed by, likely from a previous modification, the English text “Russian coding” for no reason.
> the LLM giving different answers depending on the input.
LLMs are actually designed to have some randomness in their responses.
To make the answer reproducible, set the temperature to O (eliminating randomness) and provide a static seed (ensuring consistent results) in the LLM's configuration.
3 replies →
In had it doing the reasoning in Turkish and English despite the question being in German.
i’ve seen that with deepseek
I suspect this also happens in programming languages. Subjectively I get the feeling that LLMs prefer to write in Python or JS.
Would be interesting to see whether they actually score better in leetcode questions when using python.
See my other comment. The answer is transfer learning: leveraging massive amounts of data in one language like Python, a few bridges to another language like Ruby, and obtain a “native” result in the other language.
But in this case the LLM is not exposed to explicit translation pairs between these two languages and rather by seeing enough examples in similar contexts, LLMs transfer some of their learnings in Python to Ruby (for better or worse results)
Based on my very very limited understanding of how LLMs work, surely they don't "prefer" anything, and just use what they have been trained on?
Presumably there is a lot more public info about, and code in Javascript and Python, hence this "preference"
Maybe the LLM preferring English is because of a similar phenomenon - it has been trained on mostly western, English speaking internet?
4 replies →
Fascinating phenomenon. It's like a new Sapir–Whorf hypothesis. Do language models act differently in different languages due to those languages or the training materials?
This is one of those subtle clues that the LLM does not actually 'know' anything. It is providing you the best consensus answer to your prompt using the data upon which the weights rest, is that data was input primarily as english then you are going to get better results asking in english. It is still Searle's Chinese Room except you need to first go to the 'Language X -> English' room and then deliver its output to the general query room before delivering the next result to the 'English -> Language X' room.
8 replies →
They absolutely do. They know more in English than in Spanish, I've seen that on all models, since the beginning.
5 replies →
Both, but primarily due to the lack of training materials. 10 or so million native speakers of my language will never be able to generate the same amount of training material as over a billion English speakers do.
There is a steep drop in quality in any non-English language, but in general less native speakers = worse results. They tend to have a certain "voice" which is extremely easy to spot and the accuracy of results goes out the window (way worse than in English).
1 reply →
For most low-resource languages, support in LLMs is trained through translation pairs between english and the other languages, because translation data is easier to come across than say, conversations about coding, history, physics, basically the kind of data that is usually used for instruct training.
This kind of training data typically looks like ChatGPT style conversations where all the prompts are all templated like “Translate the following text from X to Y: [text]” and the LLM’s expected answer is the translated text.
LLMs can generalize through transfer learning (to a certain extent) from these translation pairs to some understanding (strong) and even answering (weak) in the target language. It also means that the LLM’s actual sweet spot is in translation itself since that’s what was trained in, not just a generalization.
Indeed. I've thought from the beginning that LLMs should focus specifically on ONE language for this exact reason (i.e. mediocre/bad duplication of data in multiple languages). All other languages than English essentially "syphon" off capacity/layers/weights that could otherwise have held more genuine data/knowledge. Other languages should not come into the picture afaics - dedicated translation LLMs/existing-solutions can handle this aspect just fine and there's just no salient reason to fold partial-multi-language-capacity in through fuzzy/unorganised training.
I'd mentally put this in the same box as "chain of thought", where models perform better when explicitly describing the reasoning steps. The only difference in your case being that the model is undertrained in non-English data, so it's "next token prediction" of non-English prompts is less robust, and thus explicitly converting to English and then back makes it better.
This is probably the case for the "deep reasoning" models as well. If you for example try DeepSeek R1, it will likely reason in either English or Chinese (where it presumably is well trained) even if the prompt is in other languages.
ChatGPT is very informal and talks like a millennial when I ask questions in French. I hate it.
Is there a phenomenon where middle-aged people are very informal or slang-y in France? Usually the kids are the ones creating new lingo in English.
In ChatGPT settings, you can set your preferences, e.g. choose between tu/vous, and ask it to be more formal.
This should fix your issue, right?
Out of curiosity, does vous/tu change its behaviour?
Don't speak French, but interesting that it's not quite felt like an insufferable American tourist not in the group chat, in your language. LLMs all belong in that spectrum in my primary language.
sorry u hate a whole generation
3 replies →
I sometimes dream that they would internally reason in Ithkuil and gain amazing precision.
Given the fact that LLMs like most neural networks work by passing their input through layers, wouldn't this be expected? There's no going back to an earlier layer and if the first layers are in some sense needed for "translating" [0] to English, any other functionality in those layers cannot be used.
[0] I am simplifying here, but it would make sense for an LLM to learn this, even though the intermediate representation is not exactly English, given the fact that much of the internet in English and the empirical fact that they are good at translating.
Some studies are trying to ensure that the model reasons through abstractions instead of linguistic representations. (Of course the phenomenon of reasoning in substantially different quality depending on input language signals a fault - reasoning is beyond "spoken" language.)
In the past hours a related, seemingly important article appeared - see https://www.quantamagazine.org/to-make-language-models-work-...
This important paper from Anthropic includes evidence that part (but only part) of reasoning is cross-lingual:
https://www.anthropic.com/research/tracing-thoughts-language...
I have observed this and this is what I would expect to have happened thinking from first principles.
Maybe someone should edit the title to mention this is from 2024: [Submitted on 30 Sep 2024 (v1), last revised 15 Oct 2024 (this version, v2)]
Added. Thanks!
I also quite liked the EuroLLM project: https://huggingface.co/blog/eurollm-team/eurollm-9b
Was pretty good with Latvian (better than other models this size as well as variants of Llama or Qwen that I could run) and I assume probably with other EU languages as well.
I've just tried it in one of the supported languages, and it seems to respond far better than any model under 24B that I've tried before. With its licensing, it sounds much more exciting to me than the OP.
More diversity in the LLM space is always good. In my experience though, speaking as a native speaker of one of the less-used European languages, Mistral's models already use it pretty well.
I live in a country with 3 national languages and I happen to use all of them + English + another one where most of our clients are based. Mistral is the only model atm which doesn’t make a mess of it all. It’s not perfect, but it doesn’t force me to “pretranslate” things.
As a native of another small European language, no state of the art model comes anywhere close to not being laughably bad, so more work in this space is definitely welcomed as far as I'm concerned.
Really? In my experience, Le Chat eventually devolves into spanglish when trying to speak Spanish, so I would have expected worse from Mistral for minority languages.
On this topic, don’t miss the quite useful benchmark:
https://euroeval.com
ah, yes... Europe, the continent with 10 countries
one of them with 50k population
Could you elaborate on what you wish to convey with this comment?
2 replies →
I wonder how this compares to RWKV-V5 7B
https://blog.rwkv.com/p/eagle-7b-soaring-past-transformers
There is also a Greek LLM from 2024.
Meltemi: A large foundation Language Model for the Greek language
https://huggingface.co/ilsp/Meltemi-7B-v1.5
Meltemi is ok, but it's "old" and not that good by today's standards. If you need a good Greek local LLM try https://huggingface.co/ilsp/Llama-Krikri-8B-Instruct. Yes, I know it's based on LLama and not a foundation model, but it is still a LOT better than Meltemi.
I mean, Mistral AI is a Paris-based company, and theirs was considered on par or better than other open weight models such as llama3.1 and qwen2.5, and mistral-24b is currently beating oh-so-great gemma3-27b depending on tasks.
Also, Stable Diffusion was originally (and still is I believe) developed in Munich.
It's true though that raising capital and finding investors works wayyy better in the US (kindof needless to say on HN) and so was getting top talent - at least in the past. Don't get me started on energy prices ;) but I don't believe those contribute significantly in the end anyway.
You don't think American companies raising hundreds of millions to ten billion for training models contributed to their model performance or market positions?
I think a pile of money and talent is largely the cause of where they're at.
>European versions of ARC
But this is an image-like benchmark. Has anyone looked at the article about the EU-ARC, what is the difference? Why can't you measure it on a regular one?
I glanced through it, didn't find it right away, but judging by their tokenizer, they are learning from scratch. In general, I don't like this approach for the task at hand. For large languages, there are already good models that they don't want to compare with. And for low-resource languages, it is very important to take more languages from this language group, which are not necessarily part of the EU
You might be confusing ARC-AGI and EU-ARC which is a language benchmark [1]
[1] https://arxiv.org/pdf/2410.08928
Why would they want more languages from outside of the EU when they've clearly stated they only target the 24 official languages of the European Union?
For example: Slovene language. You simply don't have enough data on it. But if you add all the data that is available on related languages, you will get a higher quality. LLM fails with this property for low-resource languages.
4 replies →
Can someone explain this? They just reduce the English text during pretraining to balance it out? Shouldn't that harm every other benchmark though?
Upset that my mind went, "TEKKEN 7 LLM." Imagine Heihachi Mishima vibe-coding for you.
TIL there are european versions of ARC, HellaSwag, MMLU, and TruthfulQA.
A paper on languages that begins with a grammatical error in the first sentence does not inspire confidence:
> LLMs represents a disruptive technology
Hey, at least it's not generated by chatgpt :D
Funny how LLMs now write cleaner than humans in most cases.
I imagine there was a similar tipping point in the Industrial Revolution where machines started marking "better" manufactured items than artisans.
4 replies →
Given that it’s about non-English languages it is forgivable
They compared with Llama 3.1 and found that to be better on average for their tasks like European MMLU. And Llama 3.1 is the worst in the batch with Qwen 2.5 and Gemma 3 being significantly better.