Comment by zormino
8 hours ago
Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
8 hours ago
Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
Only if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing).
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.