Comment by meroes

Isn’t that how math works in some respects? In that, there’s only a hierarchy of consistency (no absolute consistency) for most of math. And we just keep building and building. We tried the absolute consistency route and found it too limiting.

Maybe that this doesn’t work for LLMs is a sign they aren’t on the path to AGI…

Personally I found LLMs horrendous at this kind of stuff. I’m basically a RLHF peon by trade and if I’m ever needing a quick way to fool a model, I go to simple logical problems, where it can’t lean on external structures, only itself. I don’t mean logical syntax but logical reasoning. I can’t share recent stuff but a just a few months ago the models I work with failed to reason removing 12 cards from a regular deck couldn’t remove an entire suit. That kind of stuff. Why would I want to make my prompt longer and more detailed to provide it extra structure (which is logically superfluous) to ensure it gets the right answer. Im sure a wordy prompt could get it to the right answer. I’m interested in its ability to “reason”, not prompt engineering.

Given that math is devoid of external structure, I wonder if there something to this (it’s at least interesting to speculate)