Comment by mashlol
1 day ago
I'm not an expert, but as I understand it there are existing solvers for poker/holdem? Perhaps one of the players could be a traditional solver to see how the LLMs fare against those?
1 day ago
I'm not an expert, but as I understand it there are existing solvers for poker/holdem? Perhaps one of the players could be a traditional solver to see how the LLMs fare against those?
While others have commented about solvers, I'd also like to bring up AI poker bots such as Pluribus (https://en.wikipedia.org/wiki/Pluribus_(poker_bot)).
This also wouldn't even be a close contest, I think Pluribus demonstrated a solid win rate against professional players in a test.
As I was developing this project, a main thought came to mind as to the comparison between cost and performance between a "purpose" built AI such as Pluribus versus a general LLM model. I think Pluribus training costs ~$144 in cloud computing credits.
Should be noted that this bot is heads up only? I believe a form of heads up poker is effectively solved as well-- limit hold'em heads up
the LLMs would get crushed
To expand on this - an LLM will try to play (and reason) like a person would, while a solver simply crunches the possibility space for the mathematically optimal move.
It’s similar to how an LLM can sometimes play chess on a reasonably high (but not world-class) level, while Stockfish (the chess solver) can easily crush even the best human player in the world.
How does a poker solver select bet size? Doesn't this depend on posteriors on the opponent's 'policy' + hand estimation?
5 replies →
How would an LLM play like a human would? I kind of doubt that there is enough recounting of poker hands or transcription of filmed poker games in the training data to imbue a human-like decision pattern.
2 replies →
You are of course correct but to be pedantic:
Stockfish isn't really a solver it's a neural net based engine
Unlike Chess, in poker you don’t have perfect information, so there’s no real way to optimize it.
1 reply →
The solvers don't typically work in real time, I don't think. They take a while to crunch a hand.
"Solvers" normally means algorithms which aim to produce some mathematically optimal (given certain assumptions) behaviour.
There are other poker playing programs [0] - what we called AI before large language models were a thing - which achieve superhuman performance in real time in this format. They would crush the LLMs here. I don't know what's publicly available though.
[0] e.g. https://en.wikipedia.org/wiki/Pluribus_(poker_bot)
Solvers, in a poker context, are a category of programs. They run a simulation after you enter the known information.
Like piosolver, as an example.
The best poker-playing AI is not beatable by anyone, so yes, it would crush the LLMs.