Comment by weitendorf

Yep. It's actually kind of mind blowing how well "reasoning" works at de-biasing and pushing out of global minima, in hindsight, when you see how jagged things get again without.

These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task.

(This is a ~30s Ai-driven gif presentation of the results, the full grid shows up about 15s in): https://imgur.com/a/wGFZvGw

These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...