Comment by akoboldfrying

Agreed. One possible objection might be that they need fast random access to weights, but I skimmed parts of the paper and it looks like they process 128 entries at a time, which to me sounds like it should be amenable to better compression: short enough that better compression results could still be efficiently cached in faster local RAM, long enough that better compression would save useful amounts of memory per block.