Comment by nejch
10 hours ago
My team runs open models for devs at our company, mostly on H200s, and I'd also say yes, if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
The good thing is the models themselves are good enough to usually find the root cause if you give them read access to your deployment, logs and upstream issues/PRs to analyze.
If you do A/B deploys and E2E test them with popular harnesses (we do opencode/codex/claude), you'll catch most things. It'd be interesting to hear what the more nimble inference/neo-cloud providers do when they deploy models within days of them being released, as I know it definitely needs some patching.
But I think things have improved since the days when even chat templates/tool parsers were problematic, and their new flat model approach might help as well. I suspect some of the issues came from models inheriting config and parsers.
> if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
Well, except this isn't exactly a bleeding edge architecture. DeepSeek-V4-Flash was released four months ago. I think it's not unreasonable to expect it to work well and be somewhat stable on mainstream hardware and not have critical bugs like e.g. garbage output?
Can I ask your setup? I’ve been tasked with taking a beefy machine (vague on details on purpose, the specs are legit) and trying to figure out how to run on-prem inference using open models. Gated in that I cannot use non-US models. I don’t think I have a handle on how to set it up for multiuser use without nuking TPS or context length. The current power users set and forget tasks overnight and expect results in the morning. Which… I have mixed feelings about, but that is the bar set by paying all the money for GitHub copilot enterprise.
Any tips?