Comment by irishcoffee

Can I ask your setup? I’ve been tasked with taking a beefy machine (vague on details on purpose, the specs are legit) and trying to figure out how to run on-prem inference using open models. Gated in that I cannot use non-US models. I don’t think I have a handle on how to set it up for multiuser use without nuking TPS or context length. The current power users set and forget tasks overnight and expect results in the morning. Which… I have mixed feelings about, but that is the bar set by paying all the money for GitHub copilot enterprise.

Any tips?