Comment by pixl97

Yea, if you ever run your own models on a GPU there are a whole ton of different dials you can adjust that drastically affect compute use, memory use, and output token quality, and number of tokens held in memory.

If anyone reading has a GPU it's worthwhile just messing with a smaller model for a bit to watch how the settings affect output.