Comment by mritchie712
1 day ago
in short: it's faster, cheaper, smart structured output.
each "question" is answered in parallel instead of a sequential (like an LLM). so if you have an input like:
{"is_it_hotdog": noul, "is_it_apple", noul}
it answers is_it_hotdog and is_it_apple in parallel and gives a probability.
Can't I just parallelize my LLM calls myself for each question?
You can. It will be expensive, slow, and less reliable than a specialized model.
Anything you can do in Jev can be done with an LLM at much greater cost and latency.
Agree, except the probabilities for outcomes in the structured output. I don't think you can get those for most frontier LLMs (logprobas). You can get it for open source models but not frontier LLMs.
2 replies →