Comment by rebekkamikkoa
7 hours ago
Hi Antoine!
Interesting point about backend variance. Do you think serving layer should become part of standard LLM eval reporting?
7 hours ago
Hi Antoine!
Interesting point about backend variance. Do you think serving layer should become part of standard LLM eval reporting?
Hi! Yes, I definitely think so. I've seen variance across all model families I looked at. The magnitude changes, but the presence of variance is a constant.