Comment by Aurornis
11 hours ago
You can’t build a frontier model with one single thing. This is an incremental improvement but it doesn’t explain the entire success of the model. The training set is immensely important, regardless of how you feel about distillation.
Reminds me of this btw:
https://www.bbc.com/news/technology-12343597
Microsoft replied that Bing uses “many different signals” —- including cribbing from Google :-)
I remember 15 years ago or so, one of my first student job was to evaluate Bing results compared to the same query on Google. Didn't know then that I was a distillation attacker.