Comment by SparkyMcUnicorn
10 months ago
You can use 0.6B for speculative decoding on the larger models. It'll speed up 32B, but slows down 30B-A3B dramatically.
10 months ago
You can use 0.6B for speculative decoding on the larger models. It'll speed up 32B, but slows down 30B-A3B dramatically.
No comments yet
Contribute on Hacker News ↗