Comment by orbital-decay
1 day ago
It's a non-instruction-tuned classifier model trained on a confidence-aware RL variety that generates its own schema and follows it, with a confidence score output. Think BERT on crack, smart enough to be used as a decision maker (conceptually). They call it "not an LLM" because it's non-generative but of course it's a language model in the same way all non-instruction-tuned classifiers are.
I was a bit skeptical when read the initial pr on it, yesterday ran a test involving ~250M tokens, something we measure went from ~60% to >80% success (with almost no tuning) and at less than 50% cost the low-end LLM was running at, looking at it more seriously now ... the servers are US-only currently I understand and ZDR is by request
Hmm is there a market for provisioning a similar system with ZDR being easy?
This is a great evolution in the right direction compared to LLM + pydantic and temperature 0.
Isn't that the same transformer at the end of the day? It must be faster only because it generates a single token output, just one evaluation of the model. It takes the same input context and has the same O(n^2) attention blocks. It probably takes options as appended to the input and returns a probability over them instead of the whole dictionary. It's post-trained to do that specific job. If so what's the big deal?
They say it's "parallelized". Whatever that means in reality, their demos are pretty good, their prices are extremely low compared to alternatives, and it responds in ~100ms which is pretty fast for what they do. Whether it holds for longer inputs, edge cases, etc. remains to be seen, but I can imagine the use cases for that, for example you can use it directly in the sampling layer of a normal generative model, or just as a generic decision maker/controller. They can (and will, in their words) do this for images too. I don't know if it's a big deal, but it's kind of a fresh perspective.
Unless I misunderstood what they wrote, I read parallelized in the diffusion sense, akin to GemmaDiffusion and Inception Labs models. Incidentally, Mercury 2.5 is truly groundbreaking, giving it a try is highly recommended.
1 reply →