Comment by ownagefool
19 hours ago
I wrote a scheduler for VMs a long time ago. k8s is basically just the same thing but for containers.
It really confuses me how someone can argue for cloud providers over a decent open solution without realising their argument is simply they don't want to be managing the thing.
And that's fine, most teams shouldn't be neck deep in managing a platform. But that doesn't make the solution bad.
K8s isn't even hard! My team of three manages everything on K8s and we spend ~0 minutes per week on it. Write a script to generate some YAML files, stick it in a CI pipeline, and it's basically fire-and-forget.
You're going to want most of what K8s has anyway: blue-green deployments, some way to specify how many replicas you want, health checks, etc.
The initial setup cost is annoying if you've never done it before, but in terms of maintenance it's very very easy.
Are you using a service like GKE, or running your own cluster? I've set up clusters "from scratch" in dev environments to get familiar with it.
We do both: managed Kubernetes when it's available (AWS, Nebius, others), but for some hardware vendors they just give us raw machines and we self-host K3s on their nodes. We're an open-source LLM inference company so we're basically always scrambling for GPUs wherever we can get them, which means we need to be fairly scrappy with what we support while still having a semi-sane interface for ourselves internally. Kubernetes makes that pretty easy: onboarding a new vendor takes ~minutes, and then everything Just Works and we can interact with the pool of compute the same way we do every other pool since the K8s API is standard, with all of our built-in prod monitoring tools immediately set up and running.
That being said I love exe.dev and have been a happy customer since launch. It's a different use case but they do an amazing job at it. Very, very easy personal cloud dev box. But K8s is very very good too, just for production workloads rather than personal ones!