Comment by xpe

> No, I'm just tired of constantly seeing a torrent of misinformation from people who don't know much about how these models actually work nor have done any significant work on their internals, yet try to write about them with authority.

I get that. So what can we do?

One option is when criticizing, write as clearly as possible. Err on the side of overexplaining. From my point of view, it took a back-and-forth for your criticism to become clear.

I'll give an example when more charity and synthesis is welcome:

>> Fine-tuning advanced LLMs isn’t knowledge injection — it’s destructive overwriting. [...] When you fine-tune, you risk erasing valuable existing patterns, leading to unexpected and problematic downstream effects. [...] Instead, use modular methods like [...] adapters.

> This is just incorrect.

"This" is rather unclear. There are many claims in the quote -- which are you saying are incorrect? Possibilities include:

1. "Fine-tuning advanced LLMs isn’t knowledge injection — it’s destructive overwriting."

Sometimes, yes. More often than not? Maybe. Categorically? I'm not sure. [1]

2. "When you fine-tune, you risk erasing valuable existing patterns, leading to unexpected and problematic downstream effects."

Yes, this can happen. Mitigations can reduce the chances.

3. "Instead, use modular methods like [...] adapters."

Your elision dropped some important context. Here's the full quote:

> Instead, use modular methods like retrieval-augmented generation, adapters, or prompt-engineering — these techniques inject new information without damaging the underlying model’s carefully built ecosystem.

This logic is sound, almost out of tautology: the original model is unchanged.

To get more specific: if one's bolted-on LoRA module destroyed some knowledge, one can take that into account and compensate. Perhaps use different LoRA modules for different subtasks then delegate with a mixture of experts? (I haven't experimented with this particular architecture, so maybe it isn't a great example -- but even if it falls flat, this example doesn't undermine the general shape of my argument.)

In summary, after going sentence by sentence, I see one sentence that is dubious, but I don't think it is the same one you would point to.

[1] I don't know if this is considered a "settled" matter. Even if was considered "settled" in ML research, that wouldn't meet my bar -- I have a relativity low opinion of ML research in general (the writing quality, the reproducibility, the experimental setups, the quality of the thinking!, the care put into understanding previous work)