Comment by datadrivenangel
4 days ago
With an open model, you could probably reverse engineer the token probabilities and get that probability estimate.
Something like: "Is {sentence_a} a plausible answer to {sentence_b}? Respond only with a single yes/no token" and then look at the probabilities of those.
If the model is not open turn up the temperature a bit (if the API allows that) and ask the above question multiple times. The less sure the model is the more the answer will vary.