Comment by latchkey

7 days ago

Not for what they are using it for. It is $1m+/chip and they can fit 1 of them in a rack. Rack space in DC's is a premium asset. The density isn't there. AI models need tons of memory (this product annoucement is case in point) and they don't have it, nor do they have a way to get it since they are last in line at the fabs.

Their only chance is an aquihire, but nvidia just spent $20b on groq instead. Dead man walking.

24 comments

latchkey

Reply
spwa4  7 days ago

Oh don't worry. Ever since the power issue started developing rack space is no longer at a premium. Or at least, it's no longer the limiting factor. Power is.

  • latchkey  7 days ago

    The dirty secret is that there is plenty of power. But, it isn't all in one place and it is often stranded in DC's that can't do the density needed for AI compute.

    Training models needs everything in one DC, inference doesn't.

p1esk  7 days ago

The real question is what’s their perf/dollar vs nvidia?

  • zozbot234  7 days ago

    I guess it depends what you mean by "perf". If you optimize everything for the absolutely lowest latency given your power budget, your throughput is going to suck - and vice versa. Throughput is ultimately what matters when everything about AI is so clearly power-constrained, latency is a distraction. So TPU-like custom chips are likely the better choice.

    • fragmede  7 days ago

      > Throughput is ultimately what matters

      I disagree. Yes it does matter, but because the popular interface is via chat, streaming the results of inference feels better to the squishy messy gross human operating the chat, even if it ends up taking longer. You can give all the benchmark results you want, humans aren't robots. They aren't data driven, they have feelings, and they're going to go with what feels better. That isn't true for all uses, but time to first byte is ridiculously important for human-computer interaction.

      1 reply →

  • energy123  6 days ago

    That's coupling two different usecases.

    Many coding usecases care about tokens/second, not tokens/dollar.

  • latchkey  7 days ago

    Exactly. They won't ever tell you. It is never published.

    Let's not forget that the CEO is an SEC felon who got caught trying to pull a fast one.