Comment by onlyrealcuzzo
8 hours ago
This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
8 hours ago
This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B Qwen3 model running on a single B200. Obviously these numbers are more significant with smaller models and on faster GPUs. Credit to fastokens [0] for the benchmark.
These are preliminary numbers, so I will need to do some more testing before including this in the README.
[0] https://github.com/crusoecloud/fastokens
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!
Latency isn't performance? Maybe you mean "not throughput-critical"?
2 replies →
Same here, as we're sitting in the middle between requests and what budget constraints are allowed given a particular token allowance there can be 10 ~ 100 milliseconds improvement in the UX (TTFT) given such massive tokenization speed up.
‘I run an AI platform’ I have so many genuine questions I don’t even know where to start.
Pick one. I would like to hear it.
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)
Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.
Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context
Always good to make it 0.001%
Time to first token, especially for smaller models, can be sharply reduced.
Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
Tokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
3 replies →
[dead]