← Back to context

Comment by alex7o

3 hours ago

Guys I have a real q, what is the difference between an instruct based re-ranker and laya/jev I just don't see it.

Edit: One is that jev/laya are tuned to have better probabilities, but a reranker can be fine tuned to do that as well. And jev/laya use RLCD?

6 comments

alex7o

Reply
Swizec  3 hours ago

> difference between an instruct based re-ranker and laya/jev I just don't see it

Main difference is that laya/jev/et-al give you a zero-shot classifier that requires no training. You can prompt engineer your way to a quick fairly reliable cheap enough decision engine that you can use to iterate quickly (by prompt engineering).

Right now a lot of people are doing this with LLMs and it's too slow and expensive.

Imo the right iterative approach to productionizing these systems is something like:

    1. Build it with an LLM. Iterate on the prompt
    2. Start building a real-world dataset
    3. When the prompt works, turn it into a clear rubric for Jev or similar
    4. Keep iterating until desired accuracy achieved
    5. Use the real-world evals you've built to train a custom classifier fine-tuned to your needs

You now have a system that has produced useful results in production from the very beginning and by the end it's a reliable super cheap classifier that can make thousands of decisions per second.

  • janalsncm  1 hour ago

    I don’t think that’s it. I sincerely doubt most developers are doing side by side comparisons of calibration quality.

    OpenAI has a section on their embeddings model api page for zero shot classification. Of course you can choose an open weights embedding too if you’d like.

    https://developers.openai.com/cookbook/examples/zero-shot_cl...

    I think Jev wins on marketing and convenience. Most SWEs don’t want to talk about embeddings, cosine similarity, or precision/recall tradeoffs. They want something which plausibly works and is easy to use.

    • Swizec  26 minutes ago

      Yes it turns all that work of building a classifier into an api call. This is hugely valuable for prototyping and while you iterate on what the product should even do.

kakugawa  1 hour ago

Jev's value becomes more apparent when the task is a moving target. eg an auto-mode classifier.

avereveard  3 hours ago

Calibrated probability across multi task with zero shot I guess. A reranker is single task and tuning it make it even more narrow. And I guess some piping to make multiclass efficient since you cannot mask logprob for independent questions in the same output space without throwing calibration away.