Comment by spIrr
9 hours ago
> limited by the number of activated parameters
not sure I got it?
Separately, the frontier labs are kinda pushing us into that behaviour by releasing models with ever-larger context windows.
9 hours ago
> limited by the number of activated parameters
not sure I got it?
Separately, the frontier labs are kinda pushing us into that behaviour by releasing models with ever-larger context windows.
To run at decent speed, all models try hard to use only most likely relevant part of the context and most likely relevant weights (MoE) to predict the next token. Doing the math in full is unfeasible.