Comment by hedora
4 days ago
The model gives a probability distribution as an answer; you can check to see if two arbitrarily chosen probability distributions are similar or not.
4 days ago
The model gives a probability distribution as an answer; you can check to see if two arbitrarily chosen probability distributions are similar or not.
I mean predicting the result of a fair coin toss is not a classification task those models are trained to do. It can be framed as a decision making task, but for a fair coin any probability distribution of heads/tails will do equally well.
BTW, for a task of predicting a 51/49 biased coin, the correct answer is 100/0.
Not if you use a proper scoring rule.
See: https://en.wikipedia.org/wiki/Scoring_rule
I'd say selecting a scoring rule that depends on the probability distribution (the Brier score, for example) changes the task from predicting the results of coin flips to predicting the distribution. Maybe it will work with a corresponding prompt.
1 reply →