Comment by red75prime
4 days ago
I mean predicting the result of a fair coin toss is not a classification task those models are trained to do. It can be framed as a decision making task, but for a fair coin any probability distribution of heads/tails will do equally well.
BTW, for a task of predicting a 51/49 biased coin, the correct answer is 100/0.
Not if you use a proper scoring rule.
See: https://en.wikipedia.org/wiki/Scoring_rule
I'd say selecting a scoring rule that depends on the probability distribution (the Brier score, for example) changes the task from predicting the results of coin flips to predicting the distribution. Maybe it will work with a corresponding prompt.
You can use a local proper scoring rule.