Comment by botw44

The ai labs would be very dumb to get rid of subscriptions. First, I don’t even think the subscriptions are losing money, I suspect they’re around break even, maybe small loses. More importantly, the subscriptions are how they lock in users and convince companies to pay api rates. Without user loyalty that they cultivate with subscriptions businesses will just use the cheapest model on open router or maybe local models.

Do you have more info about this? I can't tell if you're being misled by the unfortunate "Mixture of Experts" terminology (which don't work the way you're describing), or alluding to something different.

Or, maybe I'm wrong, but my understanding is: MoE is just an architecture to keep the activated weights smaller per token. The experts get routed basically token-by-token, and the "experts" themselves don't have a semantic domain so the "expert" word was maybe a poor choice.

Same, I had Deepseek search for, download and transfer (to my Linux emulation machine) the best Dreamcast games yesterday.

GPT refused to do so (citing that it's illegal even though I own the games). Deepseek did a wonderful job for 7 cents.

At work I use Opus because, why not? But I could easily switch to a less capable model if needed.

I have a question that perhaps you or someone else here has an answer for: I enjoy using Opus via Google Antigravity (usually agy) for perhaps 90 minutes a week. For Google’s subsidized $20/month plan they seem to give out a reasonably generous amount of Claude tokens. How does this compare with Anthropic’s $20/month plan using Claude Code?

BTW, I also use DeepSeek v4 Flash very frequently: fast and so cheap it is almost free.

What's the speed on DeepSeek Flash? And what provider?

Fable can delegate tasks to Opus or Sonnet, so it has some agentic properties and I believe it does them in parallel.

The parallelism is where this starts to fall apart on a local PC. Like I can run some Qwen quants, but I can’t run a decent Qwen model while also running another model smart enough to actually implement it. I’d have to do them in series, and given how long Fable seems to take even with parallelism, I’d probably be waiting days for an answer.

Wrong, mostly.

Model capability is a function of model size. Raising the bar raises model performance in every domain.

An "idiot savant" model that's overtrained for a specific domain would beat a generalist model of the same size. But scale the generalist up enough, and it'll trounce the specialist. Removing poetry data from a model training mix doesn't give you much - it might even cost you some performance - and "idiot savant" approach of overtraining for a domain has a hard ceiling.

So far, it seems like there's some equivalent of "g factor" in LLMs - a broad "intelligence" value that performance across many diverse domains correlates with. And, as a rule, larger models have more of it.

> Being behind just a little bit can have exponential consequences.

That seems to be the argument of Dario, Sam et. al., but I'm not ready to believe it. Time will tell, but this can be a marathon and Anthropic and OpenAI is in getting ready to sprint the last lap of the first mile.

What you're describing is (pre-)training. Distillation requires richer labels, the probability distribution over tokens (it would be logits rather than probabilities but that's not important). From a chat transcript you can only understand the argmax/most likely token of that distribution (and only if the API allows you to set the temperature to 0). It's not impossible for an API to give you that but they won't if they don't want you distilling their models.

The intuition is that distillation exploits not only the "right" answer but the relationship between answers (what's the second most right answer? the third? etc).

Among other things, because you simply can't get those "massive amounts" of text from a SOTA model at reasonable cost. And complex reasoning cannot possibly be trained in a pure one-shot fashion, real post-training takes massive resources. The whole story doesn't add up.

Not for proprietary models, all you get is a terse summary.

what Dario wants is to retain any influence whatsover on how the research progresses before the inevitable nationalization of the frontier. he gets to keep the N-2 tech and maybe influence the N-1 tech, but the only influence on the frontier he has is today; whatever he imprints in the pipeline the government takes over.

IOW I don't think he thinks in the same categories as most folks here.

But he already got it, no? Claude Fable can only be made available to US citizens, which implies that every user who wants to use Claude Fable must provide proof of citizenship in some way, basically KYC.

Regulatory capture is the OpenAI and Anthropic end goal, for certain.

But I also think they exist in a sort of un-designed corporate narcissism, which is a common trait in bubble economies — I am not judging them particularly severely.

Netscape under Clark and Andreessen and Sun under McNealy both fell into corporate narcissism: the belief that only they really mattered, that they were chosen, and that the world needed to rearrange itself to just let them shine. They arguably let themselves get played by Oracle (a corporate psychopath) and others as a result.

OpenAI's position is profoundly corporate-narcissistic: all we need is all the money in the economy and not to have to do anything upsetting like think about turning a profit for the next four years. Like rich kids. It would be nice if you believed we were so important that we should get an enormous stipend for just being us.

Anthropic's position is: we think we're so unique and ominous that government needs to make us both essential and terrifying. We have to exist otherwise worse people will.

Both narcissistic positions.