Comment by ricardobeat
4 days ago
Now, can you do it in <200ms for 45 questions at once, have 0% malformed output, and any kind of meaningful benchmark? We’ll wait!
4 days ago
Now, can you do it in <200ms for 45 questions at once, have 0% malformed output, and any kind of meaningful benchmark? We’ll wait!
127ms latency on 26B model, here you go: https://gambler-relay-us-west1.leo-fish.ts.net/demo
Running on old home hardware, Jev is probably running on a very powerful cluster.
How it's done: https://news.ycombinator.com/item?id=49813610
Nothing has malformed output if you coerce it's output into a statically defined set of options
> <200ms for 45 questions at once
Considering your own question length: ~120 characters x 45 divided by 4.1 ~= 1317 tokens.
So question processing at 5.5k PP(around the actual PP speed of GPT5.6 Sol) it would take around ~0.24 seconds + the context processing.
Computing the output should be around ~20ms (at 50 tok/s), computing 45 tokens in parallel.
> have 0% malformed output
Pretty trivial; only the allowed output is selectable :)
So, I keep repeating myself: Jev was a low-hanging fruit all along; no one cared, and probably no one will in a few weeks?
Yeah but a lot of developers who didn't even know that this was a possibility now do, and will probably find use cases for it.
You can probably even share context between questions by cleverly manipulating the attention mask.
Nice idea! Didn't think about that; a single linear memory allocation could do the trick