Comment by Zababa

I don't think this problem is related to the fact that they don't have a world model, or because they don't form a mental model of how everything fits together, or a fundamental limitation of LLMs. These claims are often meaningless, and the boring answer is usually something like "software architecture is harder to verify than code/maths so RLing on it is harder, and it's harder writing good evals/benchmarks for it".