Comment by fennecbutt

I was hoping model architecture would migrate towards something like this, perhaps it still might. Sparse models seem like they're getting more popular.

I'd always thought we'd eventually hotload loras or MoE experts.

It would certainly be useful on the robotics/VLA side of things as well; more limited mobile hardware, download and load/unload new skills as needed.

Tbf I also don't really care what facts my models have baked in (for llms at least). I care most that the model understands general logic and then general knowledge of some level is secondary. Reason being is that everything is RAG'd in anyway.

Models spitting out well established facts is cute but I don't really ever want to rely on say "electronics knowledge" that exists in a tenuous and vague form in the model weights.

Humans write books (and datasheets) for a reason. Books are RAG.