Comment by curious_cat_163

I think condensing intelligence into smaller models is the way to go.

Also, “smaller” can mean many different things. The cost is not in storing the weights on disk.

It is in the power required to do the inference with the “active parameters”.

There is increasingly more evidence that those two can be decoupled and more power to those who are pushing on that lever!