← Back to context

Comment by jampekka

9 hours ago

AA-Omniscience Index gives +100 for correct, 0 for "I don't know" and -100 for incorrect.

For your scenario the confident confident strategy will give average of -90. Saying I dont't know to all will give 0.

A lot of models have negative AA-Omniscience Index.

They also do have AA-Omniscience Accuracy and AA-Omniscience Hallucination Rate that handle "I don't knows" differently.

https://artificialanalysis.ai/evaluations/omniscience

0 comments

jampekka

Reply