Comment by liorn
2 months ago
I asked it about Tailwind CSS (since I had problems with Claude not aware of Tailwind 4):
> Which version of tailwind css do you know?
> I have knowledge of Tailwind CSS up to version 3.4, which was the latest stable version as of my knowledge cutoff in January 2025.
> Which version of tailwind css do you know?
LLMs can not reliably tell whether they know or don't know something. If they did, we would not have to deal with hallucinations.
They can if they've been post trained on what they know and don't know. The LLM can first been given questions to test its knowledge and if the model returns a wrong answer, it can be given a new training example with an "I don't know" response.
Oh that's a great idea, just do that for every question the LLM doesn't know the answer to!
That's.. how many questions? Maybe if one model generates all possible questions then
We should use the correct term: to not have to deal with bullshit.
I think “confabulation” is the best term.
“Hallucination” is seeing/saying something that a sober person clearly knows is not supposed to be there, e.g. “The Vice President under Nixon was Oscar the Grouch.”
Harry Frankfurt defines “bullshitting” as lying to persuade without regard to the truth. (A certain current US president does this profusely and masterfully.)
“Confabulation” is filling the unknown parts of a statement or story with bits that sound as-if they could be true, i.e. they make sense within the context, but are not actually true. People with dementia (e.g. a certain previous US president) will do this unintentionally. Whereas the bullshitter generally knows their bullshit to be false and is intentionally deceiving out of self-interest, confabulation (like hallucination) can simply be the consequence of impaired mental capacity.
2 replies →
Interesting. It's claiming different knowledge cutoff dates depending on the question asked.
"Who is president?" gives a "April 2024" date.
Question for HN: how are content timestamps encoded during training?
Claude 4's system prompt was published and contains:
"Claude’s reliable knowledge cutoff date - the date past which it cannot answer questions reliably - is the end of January 2025. It answers all questions the way a highly informed individual in January 2025 would if they were talking to someone from {{currentDateTime}}, "
https://docs.anthropic.com/en/release-notes/system-prompts#m...
1 reply →
they arent.
a model learns words or tokens more pedantically but has no sense of time nor cant track dates
9 replies →
I did the same recently with copilot and it of course lied and said it knew about v4. Hard to trust any of them.
Did you try giving it the relevant parts of the tailwind 4 documentation in the prompt context?