Comment by thw20

Good work! This is very interesting. Here's a related work that construct low-rank approximation for attention: https://arxiv.org/abs/2505.12942.

Maybe the idea of Query calibration matrix Rxx is of interest to the author!