← Back to context

Comment by airstrafer

19 hours ago

Yeah, I'd expect model performance to be super spiky on SWE work, at least they admit it with the name of the model. It's distilled from an already-distilled model.

Maybe still worth it if their "64% cheaper" figure holds.

8 comments

airstrafer

Reply
Bolwin  19 hours ago

I don't think you know what distill means

  • airstrafer  18 hours ago

    I guess I don't. Does post-training from another (larger) model not fall under the umbrella of distillation? I'd imagine it leads to the same spiky-ness issues...?

    • FergusArgyll  17 hours ago

      Distilling you don't have the actual model weights of the teacher. All you have are the teachers answers to a lot of questions. You then teach your own smaller model to answer more similarly to the big teacher model.

      Fine tuning you have the actual model weights of the original model, you then train that model to answer in a different (or better) way.

      2 replies →

Tsarp  19 hours ago

With the Devin subscription even at the 20$ plan, they offered unlimited SWE 1.7 usage. Wondering if they do the same for SWE 2.