Comment by frumiousirc

You are ignoring that prompting is fractal all the way down to the scale where the human writes everything by hand.

If I say, "start a C++ package skeleton" I'm getting CMake, not waf, make, bjam, or whatever. If I state the build system, then I'm still getting some package layout drawn from the training distribution. If I specify that, I'm still getting snake_case or CamelCase or whatever naming convention, drawn from the training distribution. If I specify that, etc etc.

It is distribution sampling all the way down.

So, yes, literally LLMs ARE reaching for "it".