Comment by prathje
18 hours ago
Nice! I would love to use it for images as well. Then again is using Grammar-Based Decoding with a json response not the same? Is Jev just that with nice caching? Because then I have been using that already…
18 hours ago
Nice! I would love to use it for images as well. Then again is using Grammar-Based Decoding with a json response not the same? Is Jev just that with nice caching? Because then I have been using that already…
A functional difference is the probabilities associated with each option. True, you could use the constrained next-token distribution at the appropriate generation step, but those would not be calibrated by any means, which Jev's claim to be. An empirical analysis would be interesting.
Almost the same. The neat trick here is to ask the model to reply with just 1 letter, which is one token, rather than a long string of json. Its quicker. And just the fact that Jev made a pretty decent API for structuring your questions. And they do the massive parallelization.
Yes, that is my question too. Someone knowledgeable can comment