Comment by augusto-moura
2 days ago
IMO AI companies are selling the infrastructure and easier access to the models, beyond competing to get the smarter model. We can run Qwen locally, but at what cost? A good enough GPU for running 27B is more than 4,000 USD where I live (Brazil). Even if I were to spend 100s of hours in credits I would never reach a point where the 4000 USD investment pays off. Unless I'm doing it for the sport, building my own LLM rig is not financially viable.
This is without saying that company clients of Anthropic or OpenAI will not have the upfront money to get a RTX 3090 for each and every employee. Much better to sign a deal with the LLM providers and get cheaper token price with a sales person.
> RTX 3090 for each and every employee
That’s quite inefficient. Get 2x RTX 6000 Pro and run DeepSeek v4 Flash for the whole company. This may or may not be cost effective compared to cheap third party providers, but it lets you own your own data, is not actually that expensive, and reports suggest that you can get 2k tok/s or so from a setup like this (at respectable batch size, but that’s exactly what you want in this situation).
I’m not sure I believe that the unquantized performance is as good as people are claiming, but even light quantization would make it fit easily.
What would the time to first token be on a setup like this?
Depending on your compliance needs this may be the only way.