Comment by Tuna-Fish
1 day ago
Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perceived value to the user, that might make them less willing to pay for it.
Also, inference costs are bound to go way down with more optimized architectures. GPUs are fundamentally not great at inference. No platform where the weights are streamed from a large pool of memory is. If the models ever quiet down, there will be massive step changes in cost/token, energy/token and tokens/second, as models are etched into silicon ala https://chatjimmy.ai/
A couple of years ago Altman was saying the price of AI compute is going to drop 90% year over year or something like that, so I don't think they're nervous about talking about lowering their costs. They probably just haven't been able to lower their costs.
You have to keep in mind that about 99% of their announcements are targeted towards investors (their most important revenue source..), so they're not going to be afraid to mention metrics that make the business look better.
Jevons paradox. Cheaper tokens does not mean we will spend less.
Cheaper tokens means the company's margins increase, which would be valuable for investors to hear
The main limit to my token spend right now is that I'm running out of hours in a day.
Ah yes, Sam “Not Consistently Candid” Altman
Oh, is that the guy that sold Loopt by claiming it had hundreds of thousands of users and it turned out to have 500 DAU after his exit?
1 reply →
he lied. he's good at that.
> Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perceived value to the user, that might make them less willing to pay for it.
Wouldn't they be bragging about it to investors? It feels like something that would matter a lot to them, and at least OpenAI kinda feels desperate to find them.
There's also the small question about whether a drop in inference cost would actually change anything about profitability, when training seems to get exponentially more expensive.
Why would any company brag about their margins ? Yet they do, to attract investors.
The key AI labs are not public companies, they are at liberty to brag about their margins to potential investors in private.
And investors will leak such claims quickly enough that this reasoning cannot plausibly hide big secrets.
9 replies →
this is changing soon
14 replies →
Growing companies don't brag about their margins, they brag about their growth and revenue. Margin talk is for when you're a mature company squeezing out every bit of profitability you can - if anything it would be a negative sign to be worrying about your margins when you're supposed to still be growing and innovating.
[dead]
I mean, did anyone expect them to not have margins? Why keep it secret?
Because companies that want to go public need to look profitable or potentially profitable. And before they go public they have to release real, actual, legally demonstrable numbers for their costs and revenue anyway.
Because the most important thing for any pure play AI company right now is to prove they are a viable company. And sure they have proved they can make billions, but also that they can lose billions more. They are going to need even more money and to prove to the next round of investors at an even higher valuation that they are a viable business they need to show not that they can generate revenue, but that they can one day turn a healthy profit. And that is the trillion dollar question.
I doubt having to replace every single chip in your data center every time you release a new model will bring down costs.
Went to that URL asked one question - "how is this different from other AI" and it took 598/6144 tokens, not sure what that means.
Not super clear from the site itself, but this LLM is running on specialized silicon implementing just it. So has super low energy use and blazing speed.
See https://taalas.com/products/
Edit: updated link
Incredible increase over Nvidia! Need to read more.. Thanks!
Because they can think more than one quarter into the future? Why on earth would someone adopt something into their core workflow that was fantastically unprofitable? Uncertainty and business don’t mix. Most people aren’t hype-eating bacteria that only care about maximizing their next paycheck.
Regardless of profitability there will always be multiple good LLM vendors as well as open-source alternatives (slightly worse but still pretty good). If one vendor fails then it's easy to switch your core workflow to a competitor.
One reason is that all the code you write with this goes in your private git. If using AI no longer is possible because of cost, you can still profit a lot from what you did with it before.
For consultants? Sure. What percentage of contractors are consultants? And is that better than going with something in your stack that’s sustainable even if it’s not totally optimal? I’d wager most would say no.
If inference costs drop 90% or whatever, that would be a massive write-off of hardware even before they gave any returns for it?! Given Chinese and others are snapping at the heels and would also benefit from such reduction in cost.
> Why on earth would AI labs be bragging about how little the product they sell actually costs them to make?
Investor confidence. They have a bit of a need for cash (also an interesting part of the profitability discussion of course).
> Also, inference costs are bound to go way down with more optimized architectures
I agree. Jimmy is incredible, I wonder what non-toy use cases they have. Surely they’ll come out with updated chips soon.
That said, I was apparently a bit over-excited for Groq and Cerebras. I thought they’d quickly dethrone Nvidia for inference, but not so far. Even the GPT spark trial isn’t seeming to go far.