Comment by hdevalence

> Unfortunately the ed25519 curve is probably not really the best choice there for a primitive to optimize due to the cofactor being an extraordinary nuisance for other applications outside of plain signatures and key agreement.

Luckily you can use an optimized ed25519 to implement ristretto255, which solves this problem :)