Comment by phkahler
5 days ago
If it can be made orthogonal, can you go a step further and diagonalize it? The storage and performance improvement from that would be huge.
5 days ago
If it can be made orthogonal, can you go a step further and diagonalize it? The storage and performance improvement from that would be huge.
You can take the output of the matrix LSTM, which is going to be matrix for each token, and compute the SVD. To get better storage, we want U and V to be the same for all tokens, so that we can operate on the diagonal S matrix. But LSTM is likely highly nonlinear, U and V will be vastly different for different tokens.
I don’t know AI, but, weight matrices aren’t square in general, right? My first guess for something like this would be to take the SVD instead, since you can always do that, but I’m sure that’s been tried already.
But orthogonal matrices are square.
Ah, good point. I assumed they were coming up with some orthogonal basis vectors for a (potentially) non-square matrix. Edit: actually I’m not sure, this Newton-Schulz process seems to work for non-square matrices as well. Generally I see “orthogonalization” refer to the process of coming up with those orthogonal basis vectors but it could be a domain-specific lingo thing.
I wouldn't say that making the matrix diagonal in some basis is some further step.
If we have an singular value decomposition, M=USV^*, the columns of U are linearly independent they are a basis for the space M maps things into, and the columns of V are linearly independent then it's a basis for the space it maps things from, and [M]_{BB'} = S.