Comment by bigyabai
6 hours ago
You're kinda describing the MoE architecture; you can offload expert layers and stream them as-needed if the experts are small enough and the SSD is fast enough.
Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.
The issue is that the MoE knowledge is LLM "knowledge" and it still has a cost so overall it is lower quality/cost.
What he's envisioning is a dense 1B model that looks at the Python specification and your prompt and goes:
> Ah, I get it now! This is like Darmok and Jalad at Tanagra!
Or at least:
> Android UI development in Python? It's UNIX, I know this!
To an extent we do work like this sometimes but in general we rely on internalized knowledge so I don't know to what extent it is a viable strategy.