Comment by Alifatisk

I remember when Google announced Geminis theoretical limit of 10M tokens context window, I was impressed. But it seems like that theoretical limit stayed as theoretical and they just pushed up to 2M. Which is still impressive.

Today, it seems Meta has crushed that wall with truly 10M tokens, wow.

I was also curious to how well Llama would be able to utilize the whole context window, it kinda pointless to have a large window if you can't recall most, if not all of it. The needle in the haystack test showed this is not the case, I wonder how they achieved this.