Comment by MrDrMcCoy
2 days ago
I broadly knew that about temperature, but lack the background in machine learning/statistics to differentiate top-n-sigma from top-k/top-p.
2 days ago
I broadly knew that about temperature, but lack the background in machine learning/statistics to differentiate top-n-sigma from top-k/top-p.
Top-K: example setting 20. Select only from the 20 most likely tokens.
Top-P: example setting 0.9. Select tokens whose probably accumulates to this number. So say you have tokens with 0.7 then 0.2 then 0.1, the last will not be selected because the first two tokens already accumulated to >=0.9.
Min-P: example setting 0.05. Don't select tokens less probable than this value. So a token with 0.1 would be considered, a token with 0.01 would not.
The purpose of all of these is to exclude very unlikely next tokens.
Min-p is specifically "Don't select tokens less probable than a multiple of the top token's probability" with min_p of 0.1 multiplied by an example top probability of 0.3 being 0.03 as the truncation at that time step.
Source: One of the min_p authors
Thanks, today I learned!
So do I, but we live in the future: https://chatgpt.com/share/6a7fc3d2-39f4-83e8-a7c6-825ddfb5e7...
You might find this article relevant: https://news.ycombinator.com/item?id=49151933
I've run the inference to get the answers I linked to. If someone else does the same thing, that involves extra energy. If I read your conversation instead of generating my own, then that's one less tree that has to be chopped down.
Until we go advanced geothermal or we crack fusion, energy is dirty. Read my inference or link me to yours so we don't boil the planet.