Comment by jrmg

To develop this approach you need to think through the reasoning of what you want to achieve. I don't think the reasoning in LLMs is nonexistent, but it is certainly somewhat limited. This is disguised by their vast knowledge. When they successfully achieve a result by relying on knowledge you get an impression of more reasoning than their is.

Everyone seems now familiar with hallucinations. When a model's knowledge is lacking and it is fine tuned to give an answer. A simplistic calculation says that if an accurate answer gets you 100%, then an answer gets you 50% and being accurate gets you 50%. Hallucinations are trying to get partial credit for bullshit. Teaching a model that a wrong answer is worse than no answer is the obvious solution, turning that lesson into training methods is harder.

That's a bit of a digression but I think it helps explain the difference to why I think a model would find writing an article like this.

Models have difficulty in understanding what is important. The degree to which they do achieve this is amazing, but it is still trained on data that heavily biases their conclusions to the mainstream thinking. In that respect I'm not even sure if it is a fundamental lack in what they could do. It seems to be that they are implicitly made to think of problems as "it's one of those, I'll do what people do when faced with one of those"

There are even hints in fiction that this is what we were going to do. There is a fairly common sci-fi trope of an AI giving a thorough and reasoned analysis of a problem only to be cut off by a human wanting the simple and obvious answer. If not done carefully RLHF becomes the embodiment of this trope in action.

This gives a result that makes the most people immediately happy, without regard for what is best long term, or indeed what is actually needed. Asimov explored the notion of robots lying so as to not hurt feelings. Much of the point of the robot books was to express the notion that what we want AI to be is more complicated than it appears at first glance.

I'm confident that they can. This isn't a new idea. Something like this would be a walk in the park for Opus 4.5 in the right harness.

Of course it likely still needs a skilled pair of eyes and a steady hand to keep it on track or keep things performant, but it's an iterative process. I've already built my own ASCII rendering engines in the past, and have recently built one with a coding model, and there was no friction.