Comment by rhdunn
5 hours ago
So... distribute a LoRA (or equivalent) that modifies the base weights with the abliteration vectors. That makes sense as it would be possible to try different abliterations and keep the storage space down.
5 hours ago
So... distribute a LoRA (or equivalent) that modifies the base weights with the abliteration vectors. That makes sense as it would be possible to try different abliterations and keep the storage space down.
Yeah, orthogonalisation is a special case of a rank-1 LoRA.
I don’t think this is true.
Could you please explain why? It's an additive update to the weights, adding an outer product of a vector with its transpose, which must have rank 1. What am I missing?