Comment by time0ut
12 hours ago
Grammars do risk pushing models off distribution in a way that impacts their output quality in a way Jev allegedly does not suffer from. Additionally, Jev's ability to answer questions independently is also exciting. Using an LLM to answer multiple questions in one generation has the property of earlier answers influencing later ones. TBD how many of TypeSafe's claims stand up, but my testing so far is promising. I hope they author some papers on their methods as well, but that might destroy their moat.
If you really know what grammers did, grammer is a filter to mask out option llm provided but you don't like.
It does not change potential distribution in any means. It DROPS part of answer model returned directly.
The text generation model go wild because model relies on previous section it answered to continue later section. And because now it contain item model have no idea, it is completely screwed.
In the case you only require model to answer one of a,b,c,d and don't care about later segment at all. It don't really matter.
What I mean is that, in general, constrained decoding can push model output off into less probable regimes. This is well studied; see for example https://arxiv.org/pdf/2606.21619. The mask may only retain very improbable logits. In pathological cases, the constrained output may be little better than noise filtered through the constraint. When using existing structured output APIs, it may not be possible to even know.
You don't even bother text after the [a] at first place in this case
Your question is something like
anwser only a,b,c,d for following question a. b. c. d....
the model output possibility of next character a: 0.8 b: 0.7 c: 0.3 f: 0.2 d: 0.1
If the list contains option you did not provide. The model is confused anyway, it don't matter if you use grammer to filter out the bad option or not, the answer is screwed already.
4 replies →