Comment by croniev
6 hours ago
What confuses me about this analogy is that it seems to imply this is a matter of a knowledge-base and logical inferences. But the whole reason why LLMs are so successful is that the logical approach was abandoned in favor of a statistical one (and huge amounts of data, compute, and weights). The argument is precisely, that what is not in the training does not find its way into the probability distribution. To claim that LLMs reason is a very far stretch that I don't feel comfortable to let pass.
Output can appear logical without the model actually applying formal logic. If its training has associated tools with wood, and sharp edges with splitting, then a prompt containing those ideas can make “use a sharp-edged tool to split wood” a likely continuation even if that exact combination never appeared in training.
By asking follow-up questions and establishing an in-context vocabulary, users can steer output toward a more complete description. For example, “Let’s call that tool and ‘axe’. How might I hold it?”; the model’s learned associations with holding sharp knives carefully can make “by the handle, with the blade facing away from you” a likely response.
I’m confident you could arrive at a near perfect description of an axe this way without any references or depictions of an axe in the training data. I’d imagine the same applies to many cybsecurity/cyber offense concepts (but probably not all).