Comment by seizethecheese
3 hours ago
Kimi doesn't do well on my "ask a trivia question that other AIs get wrong" test.
The question it came up with, "which U.S. state is closest to Africa?" is a pretty standard trivia question without any reason to believe other AIs would get confused. https://pellmell.ai/s/dccdeca69f929f79bc89317035610049
Even GPT-OSS-120b gets this right: https://pellmell.ai/s/1a43dfc7a3baa214aa0fa1b95d2c536a
These types of tests are kind of moot as agentic harnesses are taking over.
IMHO an Ai is the llm plus it's harness.
A good harness would allow the llm to investigate on a map.
Just like the llm can use a python script to figure out how many r's there are in strawberry.
These tests are simply not that predictable of performance of the llm.
The test here is not how close the state is to Africa, the test is coming up with a question that is hard for other AIs to answer.
Are you giving it your API for these other AIs to evaluate their responses? This 'test' seems perverse.
I don't understand the question.
The other AIs don't see the question until they are asked to react.