Comment by jumploops
5 hours ago
> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses.
As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.
Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.
Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].
Everything old is new again (:
[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...
[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf
[2] https://www.microsoft.com/en-us/research/publication/distrib...
> guessing which tokens a model might generate
A transformer that generates tokens based on pre-training? We could call it a GPT for short.
Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?
Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.
> optimizations applied at different levels of the stack
That's because it's just "guess and check" not some deep universal insight.
It's closer to "we can make this highly parallel for not that much cost, but we struggle to use that concurrency. So what if we just guess what the next step is going to be? If we are right we get a big speedup, if we are wrong we just throw that work away". Which I would classify as a notable insight. Doing work that you are 50% certain is useless is not the most obvious thing
Simple optimizations are often the best :)