Comment by bandrami
17 days ago
One weird thing I've found is that it's incredibly difficult to get an LLM to generate an invalid syllogism. They can generate false premises all day, and they will usually call a valid syllogism with a false major or minor premise invalid. But you have to basically quote an invalid syllogism to get them to repeat it; they won't form one on their own.
First try with claude: https://claude.ai/share/fabaf585-3732-4264-9ff3-03e4182c82a4
Very cool. Claude failed hard on this a few months ago. Gemma and phi have gotten better at it in recent versions, too, though qwen is still confidently getting it wrong.
Things are changing so fast that "few months" will invalidate most quality watermarks. It's good to re-evaluate frequently.
Are you only talking about open models?
Only time encountering the word syllogism was a Norm Macdonald joke.
Disappointingly, syllogism seems to have 3 definitions which mean slightly different things: https://www.thefreedictionary.com/syllogism
I guess the commonality is that a syllogism typically contains deductive reasoning (i.e. from the general to the specific)
Syllogism:
Universal claim: all cats are animals
Particular claim: Max is a cat
Singular claim: Max is an animal.
Do you own a dog house?
i know this comment is not really HN-worthy, but i find your username very wntertaining and funny
1 reply →