Comment by pinkmuffinere
16 days ago
> Ordering bias. Early prompts listed market data newest→oldest. Even with explicit notes, several models still read it as oldest → newest, inferring the wrong state. Switching to oldest → newest fixed the immediate error and suggests a formatting prior in current LLMs.
This kind of error just feels comical to me, and really makes it hard for me to believe that AGI is anywhere near. LLM's struggle to understand the order of datasets, when explicitly told. This is like showing a coin trick to a child, except perhaps even simpler.
I've noticed similar issues to this with rendering related code. Most models had a strong preference for z-up over z-down (I think, it might have been the other way around), and correcting them only fixed it for maybe the next response then the model would go back to using the wrong coordinates and getting confused by the outcome.
No amount of added context or instructions seems to fix these kind of issues in a way that doesn't still feel pretty hobbled. The only way to get the full power out of the model is to conform your problem to the expectations that seem to be baked in - i.e. just change your rendering coordinate system to be z-up.