Comment by RugnirViking
4 days ago
Have you tried asking jev to predict the result of a con flip? My one and only prompt to jev so far has been that. After I 90% heads as a result, I figured I'd wait a bit before checking again to see if this tech has become useful
No matter what answer is given, the prediction success rate is 50%. There’s no difference between the model giving a head answer 90% of the time and 50% of the time.
You'll get 50% prediction accuracy for a fair coin no matter what result a model gives. Try testing adversarial games. Mixed strategies make actual sense in them.
The model gives a probability distribution as an answer; you can check to see if two arbitrarily chosen probability distributions are similar or not.
I mean predicting the result of a fair coin toss is not a classification task those models are trained to do. It can be framed as a decision making task, but for a fair coin any probability distribution of heads/tails will do equally well.
BTW, for a task of predicting a 51/49 biased coin, the correct answer is 100/0.
3 replies →
I'm dying to know if you got heads or tails when you flipped the coin.
That's like not using an LLM for coding assistance because it can't count r's in strawberry.
Try it yourself to prove you’re useful. Predict my coin flip.
I can't, because I don't think you have any coins on you. :)
Okay: { heads: 0.5, tails: 0.5 }
50% heads, 50% tails. What would you have given for predicting my coin flip?
Sorry, I underspecified. Your prediction must be one of either HEADS or TAILS. No other prediction is acceptable.
1 reply →
A probabilistic system that cannot reliably reproduce probabilities - somewhat ironic, but absolutely not its purpose.
It’s not really ironic since the goal of the system is to reduce variance. Or I guess ironic in the same way as it is that the Buckingham Palace guards just stand there when they could leave at any time.
Try asking it to write a program to predict coin flips. That's what it is trained to do.
Jev literally cannot write programs.
Well, you can always ask it to predict a next bit in the code. Or even a letter. Maybe combine multiple letter into one and call it a token. We'll give a name to a procedure that gets tokens - decode. To improve speed, we can predict multiple tokens speculatively. To improve quality we'll give it extra space to store Chain of Thought. Ah, wait, we just invented LLMs
1 reply →
My bad, maybe I read it wrong or the comment was edited, but I was referring to LLMs in general. You won't get a fair coin flip from the model either way.
Found the human in the comments section.
this is not what *decision models* do.