To this day frontier models think that A and not B means A and B when the sentence gets pushed far enough back in their context window. The context length that model can reason over without obvious errors is much smaller than the advertised context. Between a 1/4th to a 1/20th what is advertised on the tin.
There is now a trillion-dollar industry bent to the task of convincing people these things can think. It’s gonna cause some damage.
I don't think they think. I still use them a lot despite that, because they are very powerful parameterised code generators.
…but they reason well enough given enough context (using their matmuls).
To this day frontier models think that A and not B means A and B when the sentence gets pushed far enough back in their context window. The context length that model can reason over without obvious errors is much smaller than the advertised context. Between a 1/4th to a 1/20th what is advertised on the tin.
Do you also happen to remember what you ate last thrusday?
1 reply →