Comment by janalsncm
7 hours ago
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
7 hours ago
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
[flagged]