Comment by krackers

I think eventually you'll run into prompt length limitations. I think (but am not even close to qualified in this field) that the attention mechanism is quadratic in complexity so either there's a hardcoded limit or you'll just run out of resources. I think the next major advancement would be to find a way to maintain internal/hidden state that's updated over time like an RNN does instead of having to append to a prompt