Comment by dmantis

People regularly write here that nobody really knows how LLM weights are working really, we only know how to achieve that and run inference. Isn't that true?

I see it this way in layman sense:

- you feed the model with data;

- you run the query;

- you activate a network and it gives you most probable token, refeeds it back and generates next and so on;

- nobody knows exactly why and how this next token comes and can tell strictly which input data gives which output data without running the exact query with 0 temperature.

So yes, it's not an exact copy, but you also can't say that a particular repo is not reflected anyhow. And I mean that the weights are reflected by the particular logic, not a syntax.

Otherwise why all the hassle if we can just feed the language reference doc to 100mb model and it should be able to produce everything just fine if only the syntax connections are needed and everything else is freshly derivated and not based on the actual implementations.