Comment by William_BB
18 hours ago
Ever heard of the infinite monkey theorem?
This is basically what LLMs do on really hard tasks. Prompt it a million times on a really hard problem and it might output the correct answer once.
18 hours ago
Ever heard of the infinite monkey theorem?
This is basically what LLMs do on really hard tasks. Prompt it a million times on a really hard problem and it might output the correct answer once.
The infinite monkey theorem assumes random distribution of symbols*.
Given the tokenizers have a vocabulary in the 10k-100k range, "a million attempts" will generally still only get the first token of the answer correct.
Even really rubbish models, e.g. talkie, the "what if we only use pre-1930s data to train a model?"** model, had to be almost all the way to the right answer to reach the really low HumanEval pass@100 score of ~0.04 (I'm only eyeballing the relevant chart).
* Actual monkeys not being like this is, while amusing, irrelevant
** https://talkie-lm.com/introducing-talkie
>Ever heard of the infinite monkey theorem?
Even if every atom in the universe were a supercomputer generating a trillion trillion random characters every second since the Big Bang, the chance of producing Hamlet would still be essentially zero.