Comment by trouve_search

My theory with no insider information: it’s a little of all of the above, but mostly money. To some extent, you can dig yourself out of a data hole with RL and a lot of compute. And you can buy a lot of compute and some data with a lot of money. Big labs have been operating in this regime for a while and it’s one of the drivers behind their costs beyond just scaling the weights and doing the actual training. Mistral just doesn’t have access to this level of compute or the money to try and muscle their way in.

> I am wondering what is keeping them back, though: Money? Compute? Skills? Training data?

Not ruthless enough and no backing by a corrupt govt administration that has no morals but focuses on self-enrichment instead.

Might sound drastic but I think that's actually closer to the truth thn everbody likes to admit.

> My fear is that you are really only getting really good models by training on very dubious data (outputs from the frontier models etc) and that Mistral is too European and too enterprisey to take those risks.

Exactly.

Should it, though?

I think an European company, taking Chinese models, perhaps doing its own post-training on them and training the Chinese-ness out, with a great chat service, enterprise API and coding agent, could be pretty valuable in itself.

> I am wondering what is keeping them back, though: Money? Compute? Skills? Training data?

Considering all their talk about new DCs and compute, and a few offhand comments, it sounded to me that compute is a big limitation.

> what is keeping them back, though: Money? Compute? Skills? Training data?

All of the above and more. Everything holding Mistral back is the same thing that has held Europe back from competing in the entire digital revolution. See this 1991 article lamenting the loss of any viable European PC manufacturer: https://www.nytimes.com/1991/04/22/business/europe-stumbles-...

Mistral being in Europe is disadvantaged with:

1. Money: less diverse private pension fund environment = less LPs to invest in VC funds = less VC dollars to invest in new ventures. European money is vacuumed out of the private sector into state pension funds and dumped into low yielding government bonds. This starves the private sector of capital while inflating the % of GDP driven by government spending every year (government pension funds buying government bonds in circular fashion enable runaway deficit spending...just like circular AI infrastructure spending).

2. Talent & compute: due to #1, Silicon Valley can outbid Europe for the best talent and hardware. Watch an OpenAI launch video and listen to all the European accents.

3. Local market fragmentation: Europe is a collection of countries that pretend to work together while not even having a unified capital market. The average EU citizen can barely communicate with their neighbor in a common language beyond the level of a toddler (english fluency is massively overstated by Americans who only experience tourist capitals).

4. Regulatory disadvantages: In everything from company regs, employee regs, unions, privacy regs, data portability regs, etc.

It's not "culture" or Europeans being "lazy" as most people would claim. There's currently thousands of young french people working 80 hour weeks creating dumb consulting powerpoints or legacy investment banking deal memos as we speak. Ambitious people exist everywhere in equal proportion, they're just working on the wrong things.

Europe can't compete in the digital revolution the same way they could compete in the industrial revolution due to various system design choices. Culture is simply the aesthetically observed byproducts of system design.

[dead]

OpenAI used to make Codex-specific models, but they stopped. What I've gathered from interviews and similar is that training two models isn't worth the (small) lift from having a coding-specific model. You're pre-training on everything anyway, and coding RL is reasonably useful for general-purpose models too.

DeepSeek is both cheaper and better than Mistral.

Also, new Medium 3.5 is far more expensive than previous Mistral models, and much more expensive than e.g. Deepseek

I think machines like the DGX Spark are about to become a lot more common/popular. It’s big enough to run sparse 150-250B MoEs with enough throughout for a single user. Deepseek v4 Flash is #1 (in terms of usage) on OpenRouter because it’s good enough to be useful. You can run it on a Spark (though it runs better across 2, which is getting up there in cost)

I do think there's a chance open weight models have a bit of a moment with the costs of frontier models growing on business balance sheets. It's unfortunate from my "privacy loving" PoV that it's mostly Chinese models filling the gap. ( the top models on openrouter for instance ).

I have used Mistral models out of pure ideology for web agents and the like which aren't doing a lot of heavy lifting.

I thought distillation meant small models don't have to compete with the big models and can always eventually achieve close parity, but it's just a matter of time to do the distillation? (i.e. how much lag do you want to live with) Am I oversimplifying?

It's all relative. For local use I'd classify it by hardware (VRAM size) using FP8 or Q6 quantization:

1. tiny <2-3B -- easily runnable on lower-spec hardware

2. small 4-8B -- runnable on 8GB GPUs

3. medium 9-12B -- runnable on 12GB GPUs

4. large 13-24B -- runnable on 16GB (for the lower end models) and 24GB GPUs

5. very large 25-32GB -- runnable on 32GB GPUs

6. huge >32GB -- not easily runnable on consumer GPUs without compromising performance (offloading layers to the CPU/RAM), quality (heavy quantization, esp. at <= Q4), or price (investing in multi-GPU setups and/or server-grade hardware).

You could possibly split huge down further, as 70GB models (e.g. llama 3) are easier to get working than >120GB models and 1TB models are completely intractable.

channeling Rocky (extraterrestrial) there I see :)