Comment by mannyv
16 hours ago
What's interesting and annoying is that the results only matter for these specific releases of the models, with the specific parameters used.
That isn't obvious, unless you've been doing ML for a while.
16 hours ago
What's interesting and annoying is that the results only matter for these specific releases of the models, with the specific parameters used.
That isn't obvious, unless you've been doing ML for a while.
Yep. It's actually kind of mind blowing how well "reasoning" works at de-biasing and pushing out of global minima, in hindsight, when you see how jagged things get again without.
These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task.
(This is a ~30s Ai-driven gif presentation of the results, the full grid shows up about 15s in): https://imgur.com/a/wGFZvGw
These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...