← Back to context Comment by orsorna 7 hours ago Is it that silly? You could run multiple 27B models in parallel. 2 comments orsorna Reply peri-cl 7 hours ago You actually don't need more RAM to batch multiple inference tasks of the same model.(Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated). orsorna 4 hours ago You definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.
peri-cl 7 hours ago You actually don't need more RAM to batch multiple inference tasks of the same model.(Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated). orsorna 4 hours ago You definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.
orsorna 4 hours ago You definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.
You actually don't need more RAM to batch multiple inference tasks of the same model.
(Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
You definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.