Comment by otakucode

This is a very interesting read, but it causes me some concern. When I went to college, I majored in Computer Science, but I minored in Philosophy. A substantial part of philosophy deals with how arguments are explained and proven, and mathematics is a fairly direct outgrowth of that. Reading mathematics and computer science papers or proofs is quite different from reading more typical prose. They are often very rigid and extremely precise. A single symbol or word can change the integrity of the argument from unshakably true to entirely false. It requires much more care and effort to gain what feels like a true understanding of what they communicate - and that is an important and necessary part of their character.

There are many ways to communicate ideas, and most people in most situations rely not upon proofs derived through deductive logic. They rely instead upon persuasion, intuition, and building associations between ideas. Those tactics are in many ways more natural (as in, fairly direct consequences of brain structure rooted in biology) but with known flaws. LLMs are pure persuasion, and operate entirely in that realm. They can be cajoled into repeating logically unsound ideas in much the same ways humans can. And, they can do the same back to us.

My concern is that having one model generate a paper, and then using another model to explain and answer questions about the paper, could run the risk, especially in arenas where the number of individuals with deep deductive knowledge of the subject is small, of optimizing for persuading those individuals rather than proving real results to them. Since it's reasonable to presume the model has been trained on other explanations which have convinced the expert, and on convincing people generally (which, on balance, uses more flawed intuitive explanations to persuade rather than prove), it should be very good at least at creating an environment where the expert would need to practice extreme skepticism and fully utilize critical thinking to attack every element of every claim. That can be very challenging.