Comment by Workaccount2
1 year ago
Would be a shame, because chess is an excellent metric for testing logical thought and internal modeling. An LLM that can pick up and unique chess game half way through and play it ideally to completion is clearly doing more than "predicting the next token based on the previous one".
> chess is an excellent metric for testing logical thought and internal modeling
Is it, though? Apparently nobody else cared to use it to benchmark LLMs until this article.
People had noticed this exact same discrepancy between 3.5-turbo-instruct and 4 a year ago: https://x.com/GrantSlatton/status/1703913578036904431