The reduction is in cached inputs. I've commented about this before but many labs, except Deepseek and Xaomi now, absolutely scam you for cached reads.
You are basically paying out the nose for a few seconds of VRAM residence if you are giving significant money for cache reads.
The very nature of autoregressive language modeling is that every single output token produced "reads" the cache.
So in principle the price floor for a cache hit is the flat cost of 1 output token.
Now in reality it has to be more than that because you are occupying VRAM with the cache that forces out other users. But it can still be really cheap.
Industrial Chinese electricity costs is similar to that of Texas, It's 8-9cents a kWh. The only benefit is industrial China decides to put millions of solar panels down, so "peak" sunlight hours can drop electricity costs significantly since their rates are highly dynamic.
I think this is probably correct based on the way state investment into the Chinese EV market has been working - fund a whole bunch of them and let them fight it out to be one of the few brands that will have the longevity. It's pretty brutal with the cars.
yep, from what i hear, the govt makes sure there is intense local competition in the market so it produces a few really good companies that survive... its kind ironic considering what is going on with mono/oligopolies over here...
The reduction is in cached inputs. I've commented about this before but many labs, except Deepseek and Xaomi now, absolutely scam you for cached reads.
You are basically paying out the nose for a few seconds of VRAM residence if you are giving significant money for cache reads.
The very nature of autoregressive language modeling is that every single output token produced "reads" the cache.
So in principle the price floor for a cache hit is the flat cost of 1 output token.
Now in reality it has to be more than that because you are occupying VRAM with the cache that forces out other users. But it can still be really cheap.
No one is producing one output token though.
And using up gpus for that cache is a pretty big opportunity cost. I highly doubt it's done in vram. That would be insane for the one hour caches.
So its memory + the time it takes to unload/load into vram + the extra cost per output token
Is it a scam? Idk
- Cheap electricity - Cheap, domestically produced GPUs - Efficiency research by many phDs. (many AI companies used Deepseek's research though)
Add to that home made inference chips and dirt cheap RAM from CXMT
Industrial Chinese electricity costs is similar to that of Texas, It's 8-9cents a kWh. The only benefit is industrial China decides to put millions of solar panels down, so "peak" sunlight hours can drop electricity costs significantly since their rates are highly dynamic.
I've read on X that deepseek api can stay alive for hours vs 5 minutes tops for other providers. they do it with ram and ssd, not only vram.
The rest of the best of the business is paying for it
State backed loss leaders.
I think this is probably correct based on the way state investment into the Chinese EV market has been working - fund a whole bunch of them and let them fight it out to be one of the few brands that will have the longevity. It's pretty brutal with the cars.
yep, from what i hear, the govt makes sure there is intense local competition in the market so it produces a few really good companies that survive... its kind ironic considering what is going on with mono/oligopolies over here...
Is that worse than VC-backed loss leaders? :)
Yes. VCs can’t compete with the second largest economy on the planet.