Comment by sinuhe69
2 years ago
World model might be a too big word here. When we talk of a world model (in the context of AI motels), we refer to its understanding of the world, at least in the context we trained it. But what I see is just a visualization of the output in a fashion similar to a chess board. A stronger evidence would be a for example a map of the next move, which will show whether it truly understood the game’s rules. If it show probability larger than zero on illegal board fields, it will show us why it sometimes makes illegal moves. And obviously, it didn’t fully understand the rules of the game.
> probability larger than zero
Strictly speaking, it should be a mistake to assign a probability equal to zero to any moves, even for illegal board moves, but especially for an AI that learns by example and self-play. It never gets taught the rules, it only gets shown the games -- there's no reason that it should conclude that the probability of a rook moving diagonally is exactly zero just because it's never seen it happen in the data, and gets penalized in training every time it tries it.
But even for a human, assigning probability of exactly zero is too strong. It would forbid any possibility that you misunderstand any rules, or forgot any special cases. It's a good idea to always maintain at least a small amount of epistemic humility that you might be mistaken about the rules, so that sufficiently overwhelmingly strong evidence could convince you that a move you thought was illegal turns out to be legal.
The rules of chess are small and well known. For example, rooks can't go diagonal no matter the situation. There's no need for epistemic humility.
Every so often, I encounter someone saying that about some topic while also being wrong.
Also, it took me actually writing a chess game to learn about en passant capturing, the 50 moves without capturing or pawn move forced draw, and the 3 state repetition forced draw.
3 replies →
Just for funsies:
Say a white rook is on h7 and a white pawn is on g7.
Rook gets taken, then the pawn moves to g8 and promotes to a rook.
The rook kind of moved diagonally.
"Ah, when the two pieces are in this position, if you land on my rook, I have the option to remove my pawn from the board and then move my rook diagonally in front of where my pawn used to be."
Functionally, kind of the same? Idk.
There's got to be a probability cut-off, though. LLMs don't infinitely connect every token with every other token, some aren't connected at all, even if some association is taught, right?
The weights have finite precision which means they represent value-ranges / have error bars. So even if the weight is exactly 0 it does not represent complete confidence in it never occurring.
3 replies →
No, it is not a visualization of the output, it is a visualization of the information about pawn position contained in the model’s internal state.