Comment by GodelNumbering
13 hours ago
A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on
A better benchmark might be asking the LLMs to write GO ai and then comparing that-- the issue is that there will be a HUGE difference in performance that depends purely on this game being in the LLM's training... but training a general LLM to directly play these games would be a waste of capacity and shouldn't be encouraged for benchmaxxing sake.
Programming an engine OTOH is a skill that is more general and they should all have.
Might be useful to have the target of the engine be some specific virtual machine that gets a strict cycle budget-- e.g. execution runs so many cycles, and result is read out of a specific memory address at the end (or when it terminates early).
This is a great idea. It could learn from its mistakes, repeat things that work, abstract complex situations. I would even want humans monitoring the project.
Excuse me? Is this thing supposed to be a borderline SGI or not? We already know LLMs are good at spitting out code.
Playing these games autoregressively isn't even the right way to use the LLM for this task (unless it was trained to do so...). It's somewhat like having a creative writing bechmark but requiring that all the input/output be base64 encoded. It can do it-- but no guarantees on the results!
And it's also just bencmaxxing bait: you can get a huge improvement on the task by RLing on it, but make no improvement on anything else. Doing so would just waste model capacity.
If you could tell that every LLM was equally not being exposed to the task then you could justify it as a test of abstract reasoning, but you can't. So it ends up on how much go transcripts ended up in the training, which is ... not a very interesting metric.
imho doing it on 9x9s kind of misses the whole point, anyone who has gotten to single digit kyu vs humans will understand when I say its effectively tic tac toe with extra steps
It's a very fun tic tac toe!
(I have not reached single digit kyu)