Comment by gunalx

And as I see it that is impossible with current autoregressive global attention models because each part depends on the one before. Maybe if we can cache the local attention in blocks and only recompute the global parts.