Comment by larodi

Batching. Yes.

And one thing it can help you locally is when you rate certain content and want to make sure it didn’t hallucinate. So you toss 3 or 5 times or… batch_size times .)

Curious that batch if has been there from day one, but it takes a while for people to see/grasp/grok it.