Comment by nico
4 days ago
Very cool, thank you for putting it together and sharing the data
Would you mind adding Jeffy to your benchmark?
https://github.com/nicobrenner/jeffy
https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
4 days ago
Very cool, thank you for putting it together and sharing the data
Would you mind adding Jeffy to your benchmark?
https://github.com/nicobrenner/jeffy
https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
> You could train a "trolley head" but that would just measure whoever labelled the training data!
Thank you for giving it a go. Will have to put some time into how to got about it
You are correct though, and it’s an interesting property of training classifiers, that you are essentially encoding the preference of the data labeler(s)
Any relation to https://github.com/firelex/jeff ?
Hasn’t seen it, authors not related
Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
running it now!
Awesome! Super curious about how it fails, and then figuring out how to over fit, err I mean improve it, haha