Comment by stingraycharles

Yup, same experience, it’s because the attention basically has exponential complexity. So at large context windows, they need to compress the attention (eg group multiple tokens together), when then leads to loss in accuracy.

It’s almost always better to keep your context windows small.