Comment by Animats

4 hours ago

There are real issues on the money front. The big AI companies have a financial model that assumes a huge increase in demand in the next year or two. Otherwise the bubble pops.

"Anthropic, OpenAI and every other AI company deliberately obfuscated these costs because they knew that the second a user actually had to pay for the fuckups of an AI model they’d scream like they were being stung to death by bees."

So some of the growth was purchased by underpricing, subsidizing the customers with venture capital. Uber did that, and eventually got out of it by raising prices and squeezing the drivers.

The "fuckup" problem is real. LLM-type AI exacts huge costs because it is terrible at reporting "I don't know". When it doesn't know, it generates noise and polishes it. If a "confidence too low for output" signal could be extracted, this whole technology would be a lot more useful. You could use small, inexpensive models on small problems, and only use big models when the small models failed. Most customer service bots fit that model. Needing ever-larger models to fix the noise problem is not cost-effective.

2 comments

Animats

kgeist 23 minutes ago

>LLM-type AI exacts huge costs because it is terrible at reporting "I don't know". When it doesn't know, it generates noise and polishes it.

>If a "confidence too low for output" signal could be extracted, this whole technology would be a lot more useful

Anthropic's interpretability research explored this topic a bit in 2025. Apparently, the signal is extractable:

  It turns out that, in Claude, refusal to answer is the default behavior: we find a circuit that is "on" by default and that causes the model to state that it has insufficient information to answer any given question. However, when the model is asked about something it knows well—say, the basketball player Michael Jordan—a competing feature representing "known entities" activates and inhibits this default circuit (see also this recent paper for related findings). This allows Claude to answer the question when it knows the answer. In contrast, when asked about an unknown entity ("Michael Batkin"), it declines to answer.

  By intervening in the model and activating the "known answer" features (or inhibiting the "unknown name" or "can’t answer" features), we’re able to cause the model to hallucinate (quite consistently!) that Michael Batkin plays chess.

  Sometimes, this sort of “misfire” of the “known answer” circuit happens naturally, without us intervening, resulting in a hallucination. In our paper, we show that such misfires can occur when Claude recognizes a name but doesn't know anything else about that person. In cases like this, the “known entity” feature might still activate, and then suppress the default "don't know" feature—in this case incorrectly. Once the model has decided that it needs to answer the question, it proceeds to confabulate: to generate a plausible—but unfortunately untrue—response.

https://www.anthropic.com/research/tracing-thoughts-language...

josephlord 3 hours ago

Isn’t the thing that an LLM never knows (at all)? It just guesses words based on the context and previous words and often gets lucky.

It isn’t thinking or knowing and then expressing the resulting understanding but just spitting out contextual words and hoping it reaches a conclusion or ending of some sort.