This. I don't know why OpenAI, Anthropic and others have these high valuations. We have a temporary demand shock on hardware (GPUs, RAM, Storage) but the market dynamics will create more supply in the next years. Once everybody and their mother can get their hands on hardware, we will see tons of token-per-dollar inference services competing against each others. None of these providers will do any training, just hosting open-weights model. Why would anyone give OpenAI and Anthropic their subscription money then.
And we haven't even scratched the surface on highliy-specialized distillations. All those generall-purpose open-weight model will see distilled variations for their respective industry niches.
MoE models can be largely offloaded to system RAM - recent versions of llama.cpp automatically distribute parts of the model, with all the dense parts going into VRAM and just the routed experts residing in system RAM.
I'm currently running Qwen3 Coder Next, which is a 80B A3B model, at 15tps on a 8GB 7700S and 64GB RAM.
It's an idiot (no Thinking), so it has to go over its work several times to get it right, but it's the largest idiot I can feasibly run at the moment.
Also by accident I'm undercutting the only provider of this model on OpenRouter by 40%, doing $0.48/mTok.
Of course at my token rate it would take an entire day to even reach a million tokens, but it's the principle that matters.
And if the market teaches us one thing, it is that GPU prices will drop over time. The current absurd prices is attracting competitors, and the more competitors, the lower the price.
I would not be surprised if, in 5 years, there's a neat, low-power GPU in high end laptops, alternatively, a cheap, say 500 USD or so, GPU in a box, that you can use for your AI-center at home or in the office.
This. I don't know why OpenAI, Anthropic and others have these high valuations. We have a temporary demand shock on hardware (GPUs, RAM, Storage) but the market dynamics will create more supply in the next years. Once everybody and their mother can get their hands on hardware, we will see tons of token-per-dollar inference services competing against each others. None of these providers will do any training, just hosting open-weights model. Why would anyone give OpenAI and Anthropic their subscription money then.
And we haven't even scratched the surface on highliy-specialized distillations. All those generall-purpose open-weight model will see distilled variations for their respective industry niches.
Well, with the recent news about OpenAI being unable to find investors, I feel like it's already happening
Don’t trust the news, especially now that AI is seen as part of US national security. We’re in a war, both financially and technologically.
No, it would be hyped up even more for a pump & dump :)
Many companies are figuring this out.
You still need to run them on GPUs
MoE models can be largely offloaded to system RAM - recent versions of llama.cpp automatically distribute parts of the model, with all the dense parts going into VRAM and just the routed experts residing in system RAM.
I'm currently running Qwen3 Coder Next, which is a 80B A3B model, at 15tps on a 8GB 7700S and 64GB RAM.
It's an idiot (no Thinking), so it has to go over its work several times to get it right, but it's the largest idiot I can feasibly run at the moment.
Also by accident I'm undercutting the only provider of this model on OpenRouter by 40%, doing $0.48/mTok.
Of course at my token rate it would take an entire day to even reach a million tokens, but it's the principle that matters.
AI companies aren't valued on the GPUs they own.
I think he meant NVIDIA is the winner in the end.
And GPUs anyone can buy.
And if the market teaches us one thing, it is that GPU prices will drop over time. The current absurd prices is attracting competitors, and the more competitors, the lower the price.
I would not be surprised if, in 5 years, there's a neat, low-power GPU in high end laptops, alternatively, a cheap, say 500 USD or so, GPU in a box, that you can use for your AI-center at home or in the office.